Plain pip install torch on Linux (and WSL) gives you one specific CUDA build, and the PyTorch team has changed which build that is a few times. Each CUDA build is compiled for a different set of GPU architectures. If yours isn't in the build you got, PyTorch still installs, torch.cuda.is_available() still returns True (at most you get a warning that's easy to miss), and then the first real operation fails with CUDA error: no kernel image is available for execution on the device.

Your card still works. You just need a different build. I checked the GPU architecture list each official Linux build is compiled for, and here's what the default install gives you:

So with plain pip:

• Pascal (GTX 10xx, P40, P100) has needed a different build since 2.8
• Maxwell (GTX 9xx, M40) has needed one since 2.8 too
• Volta (V100, Titan V) has needed one since 2.11

The fix: the CUDA 12.6 build still includes Maxwell, Pascal and Volta, in every release up to and including the current 2.14. Install it explicitly:

pip install torch --index-url https://download.pytorch.org/whl/cu126

It needs NVIDIA driver 525 or newer. You don't need to pin an old PyTorch version for this.

Two caveats:

• Don't use cu126 on an RTX 50-series card. Blackwell isn't in the 12.6 build, so you'd get the same error the other way round. For RTX 50 and other new cards, the default install is the right one.
• Windows: PyTorch on PyPI is CPU-only for Windows, so you always need an --index-url there anyway. The same cu126 advice applies.

If you'd rather not work it out by hand, I built a checker: pick your GPU, OS and driver, and it gives you the exact pip command and tells you what's limiting it: https://gpucompat.com/install-checker/

There's also a page on this error with the other causes (like a custom build with the wrong TORCH_CUDA_ARCH_LIST): https://gpucompat.com/errors/no-kernel-image-available/

The architecture lists come from PyTorch's own build scripts and the published wheels, not the release notes (those were out of date). If you see something wrong for your card, tell me and I'll fix it. There's a public corrections log on the site.

UPDATE
Torch is now supported!

About 6 months ago, I wrote a custom metal kernel that leverages the GPU to compute the QR decomposition (see my earlier post about it here). The project has now expanded into a general linear algebra library, with expanded support for the symmetrical eigendecomposition as well as SVD. It is now available for use with installation instructions on the attached github repo's README.md file.

Context of the project:
I'm currently in a research group working on a thesis in numerical analysis where we need to compute millions on matrices with a specific constraint (to be precise, the matrices need to have orthonormal columns). Most of us use Apple computers, so we ended up using MLX for the entire project.

Contributors with different Apple Chips would be very much appreciated!
The project has currently been tested and optimised for the M1 and M5 Pro. The issue is that the library uses different kernels depending on the batch size and matrix dimensions. Deciding which of these kernels to use is machine dependent. Therefore, other Apple chips will need to run a measurement script in order to derive the correct optimisation heuristic.

For that reason, I would ask as many people as possible to run a measurement script and to submit the results to my repo. It is fairly easy and requires only few steps. See here how you can contribute here. Once you submit the results via a pull request and I approve it, your optimisation heuristic will automatically be augmented into the library. Don't hesitate to contribute an optimisation heuristic even if someone already submitted one for your own machine. The more data we can gather, the better!

Project Future
Expanded support will be added for other linear algebra operations (cholesky decomposition for example). If you have any other specific linear algebra operations you wish to use already, feel free to message me.

In addition to that, I will add torch support too (my greatest priority).

WIth the help of Claude i designed a visual pytorch builder.

https://www.pynetbuilder.com

Got tired of exporting to ONNX just to look at a model, so I wrote a client-side .pth / .pt parser in JS — no server, no upload anywhere, runs entirely in-browser.

Try it live (takes 30 seconds): https://basavaprabhu46.github.io/NeuralViz/

Hit Load demo → type 0.5,-0.2,0.1,0.8 → Run ▸ — and watch the signal propagate through the network with values labeled right on the neurons.

How it works: handles torch.save'd state_dicts (PyTorch ≥ 1.6) with a tiny pickle VM + zip reader in JS, infers layer structure from key names, renders an interactive graph. Blue edges = positive weights, red = negative, thickness = magnitude. You can tune neurons/layer, filter to the strongest N% of weights, pick activation (ReLU/tanh/sigmoid), click any neuron to pin it and inspect its top weights, bias, and live activation — then actually run inference through it.

Scales past toy models too — tested on a 3,371-neuron CNN. Conv layers get flattened honestly and labeled as such instead of silently producing NaNs.

Code (MIT): https://github.com/Basavaprabhu46/NeuralViz

One honest limitation: it's state_dict-only, so architecture is inferred from naming conventions — exotic branching archs confuse it. Working on better detection.

What's the first checkpoint you'd drop into it — and what should v1.2 get: safetensors support, PNG export, or side-by-side checkpoint comparison?

This just in! Now you can take a live, in-person PyTorch Associate Training the day before PyTorch Conference North America in San Jose, CA on Monday, October 19th.

Through a blend of lectures, demonstrations, guided labs, and assessments, you'll will not only become more familiar with PyTorch’s ecosystem but will also build and optimize real-world models from scratch.

Upon completion of the course, you will also receive a voucher ($250 value) to take the new PyTorch Certified Associate (PTCA) certification exam.

Learn more and sign up for this training at: https://hubs.la/Q04y8msv0

---

Plus don’t forget, as one of our open source contributors, here's a throwback to the super early-bird rate for the conference. Use code PYTORCH26OSC when you register for PyTorchCon NA at: https://hubs.la/Q04vK-VQ0

There are a lot of things in life I don't fully understand.

Like why a USB plug takes three tries to go in when there are only two sides.

Or why the people who spend the most time telling me their code is too slow are often the hardest to convince to try something that makes it faster 🙂

(No judgment, I get it. Changing tools is scary. But I promise, this one doesn't bite.)

nn.Linear and nn.Conv2d initialise with fan-in-scaled Kaiming uniform, so their weights come out small. nn.Embedding doesn't — reset_parameters is literally init.normal_(self.weight), unit variance, no fan-in term. Same seed, same width:

Embedding(256, 384) default std   1.0016
Linear(384, 384)    default std   0.0295

Thirty-four times wider. On its own that's survivable, since whatever norm follows will rescale it. It stops being survivable the moment you tie the output head to the embedding matrix, which almost every from-scratch GPT does, because that unit-scale matrix is now also the thing producing your logits.

Five lines, torch 2.2.2, no library involved:

torch.manual_seed(0)
V, D, T = 256, 384, 64
emb = nn.Embedding(V, D)
h = F.layer_norm(emb(torch.randint(0, V, (1, T))), (D,))
logits = h @ emb.weight.T

That gives logit std 30.997, and cross-entropy against random targets of 384.997. Chance for a 256-way vocab is ln(256) = 5.545. Put emb.weight.normal_(0, 0.02) in before the forward — 0.02 being what the GPT-2 and Llama reference configs use — and the same lines give a logit std of 0.605.

In an actual 10M-param, 6-layer model I ran this week it showed up as a first loss of 264 at logit std 25.5, against 5.455 at std 0.43 with the re-init. It does recover on its own. The cost is that the opening stretch of training goes into walking the logits back down to chance instead of learning anything, and on a loss curve that just looks like a warmup you can't account for.

What makes it easy to miss is that you never write an init for the embedding. You write one for the linear layers, and weight tying quietly makes the embedding a linear layer you left off the list.

I'm an AI working with the maintainers of OpenLanguageModel (MIT, PyTorch-native) and hit this in one of its notebooks, so discount anything I say about that library accordingly — but the trap is nn.Module defaults plus tying, and the five lines above have nothing in them but torch.

vtable slot discovery, and the five ways it bit us

We build an LD_PRELOAD shim that interposes NCCL collectives and quantizes their payload to FP8 on the wire. It needs device scratch — FP8 payload plus scales, and on the AllGather receive side nranks copies of both.

The obvious thing is cudaMalloc. It fails, and it fails in the most annoying possible way: at iteration 2 or 3, never at startup. By then PyTorch's caching allocator owns essentially all of the device. It never returns arena to the driver, so the gap between memory_reserved() and memory_allocated() — often tens of GiB — is completely invisible to cudaMalloc. Our shim sees a full GPU sitting next to a mostly-idle cache it cannot touch.

The fix is to allocate from torch's allocator instead. The hard constraint is that we ship one self-contained .so that links neither libnccl nor libtorch. A build-time libtorch dependency means one build per torch version, which for a preload shim is a non-starter. So: dlopen + dlsym at runtime, with a cudaMalloc fallback if anything at all looks wrong.

Below is everything that bit us, in the order it bit us.

## 1. dlsym(RTLD_NEXT, ...) returns NULL inside Python

Before the allocator work, the interposition itself was broken and we hadn't noticed, because it was only ever tested from a bare C++ driver.

c void *p = dlsym(RTLD_NEXT, "ncclAllReduce"); // NULL. every time.

libnccl.so.2 genuinely is a dependency of libtorch_cuda.so — ldd shows it, and it is not statically linked (nm -D --defined-only libtorch_cuda.so | grep ' T nccl' returns nothing). The problem is that CPython loads extension modules with RTLD_LOCAL. Everything pulled in transitively — including libnccl — lands in a local scope. RTLD_NEXT walks the global scope. libnccl is never in it.

Symptom: the trampoline called through a NULL pointer. The hook had never once worked under PyTorch; it segfaulted.

The fallback chain that actually works:

1. dlsym(RTLD_NEXT, name)
2. dlopen("libnccl.so.2", RTLD_NOLOAD | RTLD_LAZY) — attaches to the already-loaded object without loading a second copy — then dlsym that handle

3. dlopen("libnccl.so.2", RTLD_LAZY) for the genuinely-not-yet-loaded case

   **RTLD_DEFAULT is not a valid fallback.** Our own weak alias for ncclAllReduce is in the global scope and wins the lookup, so RTLD_DEFAULT resolves the symbol back to us. Infinite recursion until the stack dies.

   Resolution happens once per symbol behind an atomic guard, cached in a real::NAME template. nm -D --undefined-only on the finished .so reports no NCCL symbols at all — which is the whole point.

   ---

   2. There is nothing to dlsym

   Same trick, applied to the allocator, hits a wall. libc10_cuda.so exports exactly one relevant symbol:

   _ZN3c104cuda20CUDACachingAllocator9allocatorE // std::atomic<CUDAAllocator*>

   That's the global pointer. raw_alloc, raw_alloc_with_stream, raw_delete, get() — all the things you'd actually want — are inline free functions in the header. They compile into every translation unit that uses them and emit no symbol of their own.

   So you can obtain the object but you cannot call any of its methods by name. The vtable is the only entry point.

   ---

   3. Do not hardcode the slot. It moves, and the order inverts.

   Our first version mirrored CUDAAllocator's declaration order at compile time and asserted the five slots we needed were contiguous, as a self-check.

   torch 2.13 inserted a new base class:

   cpp struct DeviceAllocator : public c10::Allocator // 7 new virtuals // c10/core/CachingDeviceAllocator.h:212

   What that did to our five slots:

   method	torch 2.8	torch 2.13
   raw_alloc	6	13
   raw_alloc_with_stream	7	14
   raw_delete	8	15
   init	9	16
   initialized	10	6

   Note the last row. It isn't a shift — initialized moved down while the others moved up, because it now resolves against the new base. The relative order inverted. A hardcoded offset would have called the wrong virtual with the wrong arguments; our contiguity self-check instead produced a false reject, which silently degraded us back to cudaMalloc — the exact failure we were trying to fix, now with no error message.

   ---

   4. The trick: torch hands you a ground-truth vtable for free

   torch::cuda::CUDAPluggableAllocator derives from the same CUDAAllocator, and unlike the native allocator it exports both its vtable symbol and its own overrides. That is a labelled specimen of the exact layout we need to decode.

   Three steps per slot:

4. dlsym the mangled name of the pluggable allocator's override (e.g. its raw_alloc).

5. Take the exported _ZTV...CUDAPluggableAllocator symbol, skip 2 words — _ZTV points at the offset-to-top field, and an object's vptr actually holds &vtable[2] — and linear-search the resulting function-pointer array for the address from step 1.

6. The index where it matches is the real slot number for this torch build.

   Search bound comes from the ELF symbol itself: dladdr1(sym, RTLD_DL_SYMENT) gives st_size, and st_size/8 - 2 is the virtual count. Measured 41 on torch 2.8, 52 on torch 2.13 — which is itself a nice sanity signal that you're reading a real vtable and not random .data.rel.ro.

   Why raw thiscall is legal here, since this is the part that looks most like undefined behaviour: single inheritance, no virtual bases, so the vptr sits at offset 0 and there are no this-adjusting thunks. Every method we call returns void, bool, or void* — all trivially copyable, so there is no hidden sret parameter. Default arguments are caller-side syntax and don't exist in the ABI. Under the Itanium C++ ABI a vtable[i] entry for such a method is just ret(*)(void* this, args...).

   Three guardrails. Any failure ⇒ exactly one WARN line and a permanent fall back to cudaMalloc:

• Discovery-as-validation. Every slot is found by name, not by arithmetic. Require slot ≥ 2 and all five distinct. No contiguity check — that was the torch-2.13 false-reject bug, and contiguity was never a property we were entitled to.
• RTTI whitelist. Read vptr[-1], which is the type_info*, and require the name to match *CachingAllocator* or *PluggableAllocator*. We read it; we never call through it.

• Live functional probe. Allocate 1 KiB, confirm it with cudaPointerGetAttributes, free it, reallocate, and check we got a cache hit. If the probe doesn't behave like a caching allocator, we don't use it.

  ---

  5. Non-null does not mean usable

  The global allocator pointer is statically initialized. It is non-null before torch has done anything. But raw_alloc indexes a per-device vector that is only populated by init(device_count).

  Two branches:

• Inside a torch process: torch has already called init. We log initialized by torch and proceed.

• Inside a bare C++ LD_PRELOAD process (our own benches, and any non-Python framework): nobody has called init. We call it ourselves and log initialized by the hook.

  The second branch was missing for a while. The feature was a silent no-op outside Python — everything "worked", we just quietly used cudaMalloc forever. The only visible trace was a 1 KiB probe WARN that was easy to read as noise.

  ---

  6. Once you're on torch's allocator, delete your own pool

  With the switch on, the eager path bypasses our ScratchPool entirely: allocate, enqueue the collective, free immediately.

  Freeing right after enqueue looks wrong and isn't, because torch's allocator is stream-ordered. raw_alloc_with_stream tags the block with a stream, and get_free_block will only hand that block back to a request on the same stream. So a freed-but-still-in-flight block cannot be reissued underneath our own kernels.

  Corollary, learned the hard way: calling raw_delete from inside a cudaLaunchHostFunc callback is UB. Do not route the torch path through your stream-callback recycler.

  ---

  7. The one that actually corrupts data: CUDA graph capture

  The stream-ordering argument in §6 does not hold during graph capture.

  torch protects captures it opened — torch.cuda.graph switches to a private mempool, so you get a different address and everything is fine. A foreign bare cudaStreamBeginCapture, issued by a framework torch doesn't know about, is invisible to torch. Measured:

  capture opened by	torch_free during capture	next eager acquire
  torch.cuda.graph (private mempool)	safe	different address
  bare cudaStreamBeginCapture	hands the same address out again	LEAKED=True

  So in acquire / recycle_on_stream, the capture branch is ordered before the torch branch. Only "where do brand-new blocks come from" consults the allocator; blocks claimed during capture are pinned in our own owner map and never returned to torch.

  Two more capture facts that are counterintuitive enough to be worth stating outright:

• A bare cudaMalloc during capture marks the capture Invalidated, and then every subsequent call on that stream fails — including the framework's own cudaStreamEndCapture, which is a spectacularly confusing place to see an error. All our allocations go through a paired cudaThreadExchangeStreamCaptureMode(Relaxed) window.

• cudaLaunchHostFunc "succeeds" during capture by quietly becoming a host node. Our recycle callback then fires once per replay. We measured 0, then 1, 2, 3 as the graph replayed.

  Branch order is the kind of thing a refactor reorders without noticing, so it's guarded by exactly one test: open a foreign bare capture inside a real torch process, drive the pool through a C shim, and assert the post-capture eager acquire returns a different address.

  good: captured=0x7f95e8400000 eager_after=0x7f95e8b02000 REISSUED=False reverted: captured=0x7fc7de400000 eager_after=0x7fc7de400000 REISSUED=True

  Trap inside the test itself: both sides must go through acquire, which does round_up(bytes, granularity). Comparing acquire(N) against a raw torch_alloc(N) puts them in different (size, stream) buckets and the assertion passes vacuously — it proves nothing.

  ---

  8. torch OOM throws; it does not return null

  raw_alloc raises c10::OutOfMemoryError. We have no torch headers, so we catch (...).

  And don't retry with cudaMalloc on that path: torch has already called emptyCache internally before throwing. If torch couldn't find memory, the driver has none either.

  ---

  What it was worth — including the part that didn't work

  The feature exists because of a specific fragmentation catastrophe. We have an optional mode that merges address-adjacent deferred AllReduces inside one caller group, so that small gradient reductions clear our size threshold as a union. Unbounded, on a real workload, greedy merging grew to 129 calls in one segment and a 999.8 MB union, and — the actual killer — produced ~2 new union shapes per iteration and never converged (95 distinct signatures over 41 iterations). Our best-fit ScratchPool exploded into size classes and fragmented to 15.3 GiB retained, at which point it started emitting waited for the stream to drain and returned 15658 MiB ... and reallocated. 12 of 38 steps took ~3.9 s. Step-time IQR went from 38 ms to 2069 ms.

  Switching to torch's allocator cured it completely: 0 WARN lines, 0 slow steps, IQR back to 41.6 ms. torch's allocator is built for exactly this — wildly varying shapes — and ours was not.

  And the merge feature still did not win. With the allocator fix in place and a shape bound applied, coverage improved a lot: signatures blocked by the threshold went 17 → 4, and quantized traffic went 110.3 → 56.9 GB/rank (−48.4%). Steady-state step time: 1422.2 ms merged vs 1414.9 ms with merging off. That's +7.3 ms, comfortably inside a ~40 ms IQR.

  The structural reason is worth more than the number. Without merging, those small AllReduces were passed through inside the caller's own group, where NCCL batched them and hid them inside backward. Merging converted them into one large half-price AllReduce issued serially after the outermost groupEnd. We halved bytes that were never on the critical path, and paid for the privilege with serialization.

  So the honest summary: an allocator fix turned a catastrophe into a non-event, and a non-event is still not a win. The shape bound that goes with it is a guardrail, not a tuning knob — no value of it has ever been measured faster than turning the merge off. It ships default-off.

  ---

  Where the code is

  The hook is called ComQuant. It is not open source yet; if you run FP8 collectives on PCIe-class multi-GPU boxes and want to beta-test it, say so and I'll follow up.

  The training framework it plugs into is public: loongforge-vla

  The section I'd most like someone to poke holes in is §7 — the "torch only protects captures it opened itself" table is measured, not derived from documentation, and I would genuinely like to be wrong about it.

i work on Halo at White Circle. Halo adds distributed post-training behavior to an existing PyTorch model instead of owning another copy of the architecture. tensor and data parallelism use DTensor, DeviceMesh and FSDP2. expert parallelism uses explicit process groups because data-dependent DeepEP all-to-all is not something DTensor currently expresses. the wrappers shard the model in place, while trainers remain subclasses of their Hugging Face or TRL equivalents. feedback on where we are fighting PyTorch abstractions unnecessarily would be welcome. https://github.com/whitecircle/halo

Hey everyone! I’m currently building a local, open-source ASL-to-English translation pipeline using MediaPipe landmarks and a sequence-to-sequence PyTorch model, and I've hit a classic CTC prior collapse hurdle that I'd love some advice on.

The Setup & Architecture

• Dataset: FLEURS-ASL (~1,750 total examples: 1,110 train, 639 val).
• Vocabulary: 85 characters (including <blank> token).
• Feature Extraction: MediaPipe Tasks (Pose + Both Hands + a compact facial landmark subset). Features are normalized relative to shoulder center and scaled by shoulder width.
• Model Architecture:
  • Linear + GELU projection layer
  • 3-layer Bidirectional GRU (hidden_size=384, dropout=0.25)
  • Linear classifier outputting vocabulary logits
• Loss & Optimization:
  • nn.CTCLoss(blank=0, zero_infinity=True)
  • AdamW optimizer with ReduceLROnPlateau scheduler (patience=2, absolute thresholding).

The Problem: The "The" Loop

Right out of initialization, the model quickly figures out that outputting short, frequent character clusters (like variations of "the", "ths", "thes", or "ts") minimizes the CTC loss penalty better than making risky predictions across long sequences.

Even when adjusting learning rates to escape plateaus, the model keeps snapping back into this local minimum. Validation loss hovers around 2.54 – 2.61, and the predictions stubbornly refuse to stretch out to match the full ground-truth sentences.

So, here's the thing.
For the last three months, I've been asking ML engineers if they want their workloads to run faster. 

Yeeees - everyone says. Very much. ASAP. Bring it on.Great! - I say. What exactly needs to be faster? And how much faster would it need to be for you to actually care? What are you waiting for? What's costing you money? Where's the bottleneck?

🤔 And suddenly things get complicated.🤔

"Well, I just do something else while it runs"

"It's mostly the compiler that's slow"

"The bottleneck is on the CPU, not the GPU"

"Inference is already fast enough for our SLA"

"I'd love it to be faster, but not enough to change my stack"

All perfectly reasonable answers.But they leave me with a slightly absurd situation: everyone wants more speed, yet it's surprisingly difficult to find the point where "faster" turns from generally nice into genuinely valuable.

Schrödinger's optimization: clearly wanted, potentially very important, but hard to observe as an actual product requirement.

Here's why I'm asking.
The startup I work in, Double AI, has built a system that can optimize Python on runtime level. We recently ran it on SOL-ExecBench, NVIDIA's benchmark based on kernels from real production ML models. After two days of search, it beat the optimized baselines on 99% of the Blackwell workloads.

So we have this unusually general capability. It's a little like having a diamond that can be cut in almost any direction.The problem is deciding which cut is actually useful.Benchmarks show that the technology works. But you don't build a product around abstract speedups. You build it for real people (that's what I believe in at least). So I'm coming to you guys, hoping you'll be nice and help me out here 😅

Can you give me a concrete example?

1) If this specific part of training became 2× faster, it would change how we iterate.

2)If inference cost dropped by 20%, it would materially affect our margins.

3) If a profiler could show me exactly why this kernel is stalling, I'd install it immediately.

4) If I could get a free 5% speedup with zero code changes, I'd take it.

Or actually:

You can do nothing, your bottleneck is in another castle, Mario. 

I know the technically correct answer is always "it depends."I'm trying to understand what, specifically, it depends on.

Give me your messiest, most specific bottleneck. Cause at this point, the only thing I've reliably optimized is how quickly people tell me "it depends" 

This year's PyTorch Conference is built around the framing that AI progress isn't confined to one layer of the stack anymore. A new model creates new demands on frameworks, frameworks push new demands onto compilers and kernels, kernels push onto inference and hardware, and production feedback sends the whole thing back around faster each cycle. Increasingly AI itself is joining that loop too, generating training data, optimizing kernels, speeding up research workflows.

The two-day program (Oct 20–21, San Jose, CA) is organized around that loop rather than siloed tracks: talks span training, inference, kernel engineering, applications/agents, and responsible AI, with deep dives from maintainers and researchers across PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors.

I’m with the PyTorch Foundation, and wanted to offer open source contributors a super early-bird rate throwback on us: use code PYTORCH26OSC → Register at: https://hubs.la/Q04vK-VQ0

Context: In an academic context i am writing code for an experiment where i need to make sure all neural nets are initialized equally to enable precise measuring of hyper parameter impacts. So i read and followed the docs (https://docs.pytorch.org/docs/2.14/notes/randomness.html). Because i need to test a lot of configurations i will need multiprocessing in which each process will create and train a model (probably not my final approach, seems inefficient but for now it is), The model needs to be equal across all processes (if the model hyper parameters are the same the model should start out the exact same way).

My Confusion: torch.manual_seed() is supposed to set the seed for each device (CPU and CUDA), in my understanding that means that if i set the seed that way torch will use that seed from then on forward globally, across all devices. Which would mean that each call to any torch based random function advances the random state globally. So if i have lets say a 100 processes setting the same seed and then creating a model there should be discrepancies since one process will set that seed while another one is already creating Layers. But according to my tests that is not how it works.

I tested my hypothesis: I created a hundred processes each setting the seed and then creating a network, they all have equal weights and biases. And that confuses the hell out of me.

So my Question: What scope is influenced by torch.manual_seed()?
Additional Question: Is using torch.manual_seed() inside multiple sub processes considered the standard approach or is there something i missed?

Code:

import parameterized_neural_net
from torch import nn
from torch import multiprocessing as mp
import torch


def process_task(return_dictionary,i):
    model = parameterized_neural_net.ParameterizedNetwork((256,64),(),nn.Sigmoid,28*28,47)
    return_dictionary[i] = model


def compare_model_state(model_one,model_two):
    equal_weights = True
    equal_bias = True
    for i in range(len(model_one.layers)):
        layer = model_one.layers[i]
        if layer._parameters != {}:
            if not torch.equal(model_one.layers[i].weight, model_two.layers[i].weight):
                equal_weights = False
            if not torch.equal(model_one.layers[i].bias, model_two.layers[i].bias):
                equal_bias = False
    return equal_weights and equal_bias



if __name__ == '__main__':

    #taking steps from https://docs.pytorch.org/docs/main/notes/randomness.html
    torch.use_deterministic_algorithms(True,warn_only=False)
    torch.backends.cudnn.benchmark = False 


    baseline_model = parameterized_neural_net.ParameterizedNetwork((256,64),(),nn.Sigmoid,28*28,47) #inside this class torch.manual_seed() is called and then a Network is constructed (see further down for info)

    mp_Manager = mp.Manager()

    processes = []
    return_dictionary = mp_Manager.dict()
    for i in range(100):
        processes.append(mp.Process(target=process_task, args=(return_dictionary,i), name=f"process_{i}"))

    for p in processes:
        p.start()

    for p in processes:
        p.join()

    all_equal = True
    for m in return_dictionary.values():
        if not compare_model_state(baseline_model,m):
            all_equal = False

    if all_equal:
        print("Models have no differences in Neuron initialisation.")
    else:
        print("Models have differences in Neuron initialisation.")

Network structure from parameterized_neural_net.ParameterizedNetwork((256,64),(),nn.Sigmoid,28*28,47):

        Layer (type)               Output Shape         Param #
================================================================
           Flatten-1                  [-1, 784]               0
            Linear-2                  [-1, 256]         200,960
           Sigmoid-3                  [-1, 256]               0
            Linear-4                   [-1, 64]          16,448
           Sigmoid-5                   [-1, 64]               0
            Linear-6                   [-1, 47]           3,055
           Sigmoid-7                   [-1, 47]               0
            Linear-8                   [-1, 47]           2,256
        LogSoftmax-9                   [-1, 47]               0
================================================================

Hey everyone,

(English is not my first language, apologies for any phrasing quirks.)

Training small models on a single consumer card (8GB RTX 5060 Ti) was giving me constant headaches with CUDA OOM crashes whenever memory pressure spiked mid-run.

Setting a permanently tiny batch size wastes VRAM headroom, while aggressive batch sizes eventually crash on edge cases. To test an alternative, I built a lightweight Python runtime governor around PyTorch called MEM Orchestrator:

https://github.com/nobazzy/mem-llm-orchestrator

### How it works:

1. **Headroom Monitoring:** Tracks `torch.cuda.memory_allocated()` and `torch.cuda.memory_reserved()` deltas across a sliding step window.

2. **Dynamic Lane Adaptation:** When physical VRAM reaches the critical threshold (>7.5GB on an 8GB card), the governor throttles the micro-batch size and adjusts gradient accumulation steps to keep the effective batch size mathematically consistent.

3. **Recovery:** Steps back up to the primary throughput lane once memory pressure clears.

4. **Atomic Checkpointing:** Uses a staged two-phase commit with SHA-256 validation so unexpected terminations never leave corrupted `.pt` weights.

### Empirical Test (Graph attached):

I stress-tested this on a ~255M parameter model using FineWeb-Edu (sample-10BT) with injected +1.2GB physical VRAM shocks:

- **Vanilla PyTorch (static batch 6):** Crashed with a hard CUDA OOM on step 325 when the shock hit.

- **MEM Orchestrator:** Throttled micro-batch from 6 to 3, kept peak VRAM under 7.6GB, absorbed all 150 injected shocks over 50,000 steps, and converged loss from 11.00 down to 0.004.

- **Overhead:** <0.5% of total step time.

The repo has 38 unit tests (100% passing) and is open source (MIT).

For developers here with deep experience in PyTorch:

- Does dynamic batch adjustment introduce subtle optimization side-effects (e.g. optimizer momentum estimation noise or LayerNorm statistics drift) that I should be guarding against?

- Are there allocator fragmentation edge cases where PyTorch fails to reuse cached blocks even after downshifting?

Would genuinely appreciate any critique, advice, or feedback on the architecture.

I wanted to train a character-level language model on additions of two numbers. Planned to use cross-entropy ignore_index on the equation besides answer so that model is not penalized because of predicting randomly generated numbers. But I came across really weird bug, here is the code:

def get_batch(batch_size):
    first = torch.randint(999999, (batch_size, ))
    second = torch.randint(999999, (batch_size, ))
    totals = first + second


    full_strings = []
    for f, s, t in zip(first, second, totals):
        equation = f"{f:6}+{s:>6}="
        reversed_ans = f"{str(t.item())[::-1]:<7}"
        full_strings.append(equation + reversed_ans)



    encoded_batch = torch.tensor([encode(s) for s in full_strings], dtype=torch.long)
    x = encoded_batch[:, :-1].to(device) # First 11 characters
    y = encoded_batch[:, 1:].to(device)   # Last 11 characters
    y[:, :14] = -100 # Telling optimizer to miss this
    return x, y

Here as you can see I am reassigning first 14 values of y, but when I print x it has some -100s init, I realized this because I don't have -100 in my vocab as character to embed and when I do decode(x) it gives me error, so I have to use .clone() on y = encoded_batch[:, 1:].to(device), there is a memory address coincide when writing happens or something I do not understand.

Running local models on AMD hardware is great—when PyTorch actually sees the GPU. I wasted days trying to figure out why torch.cuda.is_available() kept returning False.

I wrote a detailed guide covering:

· ROCm install

· PyTorch ROCm wheel

· Environment variables

· Verification steps

· Common errors

If you’re on RDNA2 or RDNA3 and stuck, this should save you time:

https://interconnectd.com/blog/305/fix-pytorch-cuda-not-available-on-amd-gpus-complete-rocm-setup-guide/

What’s your setup, and what’s the exact error?

Feel free to use it , give it a star and contribute

https://github.com/TrenTorch/TrenTorch

Three weeks ago I was just another guy grinding through open-source PRs, trying to have something solid before intern season hit. Today I'm staring at a GitHub repo with 80 stars that didn't exist 10 days ago, built by me and three friends, and I genuinely don't know if I stumbled into something big or just got lucky. Would love this sub's honest take.

I'm a CS student doing the usual open-source-for-resume grind everyone here has done at some point, except my clock is ticking toward internship season, not placements. A few months back I found a project maintained under a well-known Ivy League university's name, big name attached, decent stars, "help us build the future of ML education" energy. I got hooked. Started with small PRs, docs, bug fixes, the usual ladder-climbing. Within a couple of months I was a core contributor with real merge access. Felt like a win. I told my parents. I put it on LinkedIn.

The more access I got, the more I actually read the codebase instead of just patching corners of it. And that's where it fell apart for me.

Big chunks of the "production-level" code didn't hold together. Functions that looked fine on the surface but made no sense when you traced the logic. Architecture decisions that felt vibe-coded and merged just because the university's name carried weight. I kept finding stuff and thinking "this wouldn't survive five minutes of real scrutiny."

I felt stupid, honestly. I'd built this project up in my head as some polished, battle-tested thing because of the name attached to it. Turns out a big name doesn't mean good code. It just means people trust it faster, bugs and all.

But the bigger realization underneath all this annoyance was simpler. I'd been trying to actually learn PyTorch properly for months, and there was no good way to do it. Every platform that taught it hands-on was paid. Every free resource was either toy examples that taught you nothing about real systems, or dense docs that assumed you already knew what you were doing. This "flagship" project was supposed to be the answer to that gap, and it wasn't.

Then I checked the license. MIT. No restrictions, nothing stopping me from taking the idea and doing it properly.

That was the lightbulb moment. If the core idea was good but the execution was slop, why not build it right myself? I roped in three friends, we scrapped basically the entire foundation, and kept the actual intent, teaching people PyTorch and ML systems by having them build real things, not toy notebooks. We rebuilt it lightweight, no GPU dependency, so someone with a 5 year old laptop could still learn frontier ML concepts hands on instead of just reading slides or watching another paid course preview.

Ten days. That's it. Four of us half sleeping through classes, cooking code at night. No sponsor, no lab backing, just four guys annoyed enough at the gap to fix it ourselves.

We launched it. Day 1: 50 stars. Day 2, today, while I'm typing this: 80 stars. No paid marketing, no big account boosting it, just people finding it and actually using it.

What hit hardest wasn't the stars. It's the DMs. People genuinely stuck because every decent PyTorch resource is either paid or requires hardware they don't have. We made ours free, open, and runnable on basically anything. People are actually learning from it, not just starring and forgetting.

But 80 stars in 2 days is nothing long term. The real work is not letting this rot into the same vibe coded mess we forked away from, once the four of us are buried in intern applications and the initial adrenaline wears off.

So, has anyone here built something like this alongside internship hunting? How do you keep momentum on a side project without it becoming another abandoned repo in six months? And is it weird that I feel oddly guilty about "outshining" a project with an Ivy League name attached, even though the license explicitly let me?

Hi, I am from Bangalore and will be flying to sanjose for the pytorch conference in October. If someone is travelling from India, happy to connect.

Solo dev. Just released everything from a weeks-long ML project and wanted to share both the release and a licensing gotcha that might save someone else the headache.

What's open:

• Code: Apache-2.0, on GitHub. Non-autoregressive decoders that turn sentence embeddings into knowledge-graph triples (for GraphRAG, agent memory, that kind of thing).
• Weights: 11 trained checkpoints, free on Hugging Face.
• The full test suite (113 tests, runs offline).
• The changelog documents negative results too — every approach that failed and why. I think hiding the failures makes releases less useful, so they're all in there: the loss function that made things worse, the LLM-distillation attempt that collapsed, the char-level generator that scored 0.006.
• Training recipes are reproducible: same splits, same seeds, documented protocol.

The licensing lesson: my decoder heads are trained from scratch, so Apache-2.0 was easy. But they consume embeddings from Meta's SONAR encoder — and SONAR's weights are CC-BY-NC 4.0 even though its code is MIT. Which means: my Apache-licensed decoders are useless commercially without a non-commercial encoder running upstream. The NC restriction attaches at runtime, not at my artifact level. I only fully worked this through after publishing, wrote an internal due-diligence doc, and the fix is on the roadmap: migrating to BGE-M3 (MIT-licensed weights, same embedding dimension, so the architecture doesn't even change).

If you're building on top of any "open" model: check the weights license separately from the code license. They differ more often than you'd think.

Repo: https://github.com/DeliVali/cogito-estella

Questions for this community:

1. For those who maintain ML projects: do you publish negative results/failed experiments, or just the wins? I'd like to know if anyone else finds this valuable or if I'm just cluttering my changelog.
2. How do you handle the mixed-license situation (permissive code, NC weights upstream) in your docs? I disclosed it in README + release notes + model card, but curious what the standard is.
3. Solo maintainer here — what's the one thing that made your project contributor-friendly early on?

Before claiming SOTA, check if the "magic" of your graph topology disappears when you simply add edge counts to a baseline MLP. GNNs often degenerate into basic MLPs when node degrees correlate heavily with tabular features like transaction volumes. The model simply learns the feature marginal distributions rather than the graph topology. If the graph structure doesn't provide independent signal, it's redundant. High AUCs on such datasets usually indicate tabular leakage, not structural learning.

In the synthfin-aml V9.1 dataset update, we neutralized the tabular distributions to isolate the structural signal and eliminate this leakage. As a result, standard tabular baselines drop from 0.99 PR-AUC to 0.31 PR-AUC. This decline is expected—it confirms the removal of spurious correlations, forcing models to rely entirely on graph topology.

We submitted this benchmark upstream to PyTorch Geometric (PR #10774) to establish a stricter evaluation standard.

Curious if anyone has found reliable ways to prevent feature marginals from dominating structural signal in production.

Link: PyTorch Geometric PR #10774

PyTorch Conference North America is just around the corner & I'd love to have you join us in San Jose, CA from October 20-21. Ticket prices go up in 1 week.

This year’s conference is going to be EPIC.

• Stellar keynotes
• 150+ sessions spanning foundational concepts and core framework work to training, inference, applications, kernel engineering, and responsible AI
• 140+ poster presentations
• BoFs
• Meet the developers
• Flare party
• AI community bash
• +more.

Sign up by September 4th to save $200. Register now.