More Threads Made My RL Training Slower. The Real Bottleneck Was Somewhere Else.

I have an RL agent that has been training on and off for months on my own machine. For a long stretch it sat at roughly 400 steps per second, and I had a comfortable theory for why: the machine has 24 logical cores, PyTorch was only using a few of them, so surely more threads would help.

That theory was wrong, and testing it is what pointed me at the actual problem.

Threads and processes, in one minute

My misdiagnosis came from treating "more parallelism" as one thing, so a quick refresher on two words that get used as if they mean the same.

A process is a running program with its own private memory. Two processes can't see each other's data. If they need to share something, it has to be sent across explicitly, which means copying it. A thread lives inside a process. All the threads of one process share the same memory, so they can work on the same data without copying it, but they have to coordinate so they don't trip over each other.

Two practical consequences. Threads are cheap to start and cheap to share data between, but the coordination has a cost, and in standard Python only one thread can run Python code at a time (the global interpreter lock). PyTorch's own maths runs in native code that isn't held back by that, which is why a torch thread setting is worth having at all. Processes are heavier, since each has its own memory and its own interpreter, and data has to be passed between them, but they genuinely run at the same time.

Both showed up in this project. Torch threads split the maths of one gradient step across several threads inside the main process. Worker processes each run their own copy of the environment and send samples back. Threads split one job into pieces, and processes run separate jobs side by side. They fail in different ways, and that difference is most of this story.

The benchmark that contradicted me

Before changing anything, I timed one gradient step of the actual network at different thread counts:

Torch threads	Time per gradient step
1	0.60 ms
12	0.90 ms

Twelve threads was about 50% slower than one. The network is small, so the work per step is tiny, and with that little to compute the threads spend more time coordinating with each other than doing math. Giving a small network more threads is like putting twelve people on a task that takes one person thirty seconds: most of the time goes on handing things around. Threads only pay off when the single job is big enough to be worth splitting, and a 0.6 ms gradient step isn't.

A caveat I want to be upfront about: that benchmark ran while a training job was using the same CPU, so the absolute numbers are noisy. The direction was clear, but I can't claim I've cleanly proven how much of the slowdown is thread overhead versus contention. I also had holdout evaluations and a browser competing for the same cores the day before, and I can't separate those effects with the data I have.

So where was the time actually going?

Not in the learning step. In a setup like this, the loop is roughly: step the environment, collect a sample, occasionally do a gradient update. With a single environment, everything happens one thing at a time. The learner sits idle while the environment steps, and the environment sits idle while the learner updates. Adding threads to the learner didn't touch that at all.

What does help is running the environment stepping in separate worker processes, so several copies of the environment produce samples at the same time while the main process handles learning. I built that as a separate training script, so the existing run was left alone, and kept two things fixed so the comparison stays fair: the ratio of gradient updates to collected samples (still about one update per four samples), and the checkpoint schedule.

What the numbers looked like

Measured on the same machine, everything running at below-normal priority so my own apps still got first call on the CPU:

Setup	Steps per second
Single environment	~424
2 workers (short smoke test)	~642
4 workers	~900 to 1,050
12 workers	~1,409

That is 3.3x over the single-environment run. Estimated time for a 40M step run dropped from about 24 hours to about 8.

1 env2 workers4 workers12 workers

Twelve workers: about 1,409 steps per second. Three times the workers bought only roughly 40% more speed. That is what a bottleneck moving to the learner looks like.

The part I find more interesting is the shape of that curve. Going from 4 to 12 workers tripled the workers but only bought roughly 40% more speed. That is the signature of a bottleneck moving. The workers are no longer the limit. The main process, which does every gradient update, is. My read is that the learner is now what holds the run back, though I haven't profiled it to confirm. Processes have their own cost too: every sample has to be copied from a worker to the main process, so the hand-off could be part of the limit as well. I haven't measured that either.

What I'm not claiming

Speed is not the same as learning well. This run starts from scratch, its exploration rate only decays over the first 12M steps, and early results in that window are noisy by design. It has to be judged against the old run on a fixed holdout evaluation at equal step counts, not by throughput. A faster run that learns nothing useful is just a faster way to waste electricity.

So the honest summary: I found that my first instinct, more threads, was actively counterproductive for a network this small, and that overlapping environment stepping with learning was the change that mattered. Whether the faster run also learns better is the next post.