A coding agent rebuilt a neural network while it was still training

DoubleAI, an AI research company that also does GPU performance work, published results on 6 November showing a coding agent changing a neural network’s design while the network trains. In a normal run only the weights move. The architecture, the optimizer and the data mix are fixed before training starts.

In DoubleAI’s setup, the agent watches the model fail, works out why, and writes a short program after studying a few of the cases the model gets wrong. That program is inserted as a new layer, which the team calls a code layer. Training then resumes, and ordinary gradient descent decides how much the network should lean on the new layer. The layer starts switched off, so the model is unchanged at the moment of insertion, and the optimizer opens it gradually if it helps. According to the DoubleAI research post, the aim is to patch weaknesses that scale alone has not fixed, such as tasks that chain many steps.

The first test used nanochat, a small ChatGPT-style model of about 286 million parameters. The agent added three code layers at moments of its own choosing, and validation loss dropped after each one. The gap over ordinary training held until the run ended at step 5,000. To decide where to intervene, the agent invented a measurement the authors call the phantom gradient, which shows what change each layer in the network is asking for at each token.

A second test turned up an oddity. DoubleAI built a 124-million-parameter model that could only see the last 500 tokens, then gave it a lookup layer that fetches earlier text. When the layer was added halfway through training, the model recalled a variable defined far outside its window 72% of the time. When the layer was there from the start, recall was 55%, even though the loss curves looked almost identical. The authors suspect the model leans on the layer instead of building its own skills when the layer is there from step zero.

The largest gain came on CLRS-Text, which tests thirty classic algorithms such as sorting and string matching. An 86-million-parameter model trained the usual way plateaued at 55% accuracy, solving only about a third of the problem sizes it had not seen in training. With ten code layers added by the agent, overall accuracy reached 84%, and the share solved at unseen sizes rose to 79%. Training the ordinary model twice as long landed at a similar plateau, so the extra layers explain the gap rather than extra steps.

Then DoubleAI pointed the method at a large model. The agent was the model itself: DeepSeek-V4.1-Flash, which has 763 billion parameters and a mixture-of-experts design. It trained a probe on its own internal state that guesses whether a finished math answer is right, and a code layer discards answers the probe distrusts and tries again, up to three times. Scored on the 2026 AIME and the HMMT February 2026 math contests, with its thinking mode off, it rose from 64% to 71% correct. The model had predicted a gain of about eight points from older contests, and the hidden 2026 problems delivered seven. The probe catches four of every five wrong answers and wrongly flags one right answer in fifteen. In a smaller version of the experiment, Qwen3-8B wrote its own stopping and retry policy and went from 59% to 66%, while spending 2.8 times the thinking budget of a single attempt.

Every result comes from DoubleAI’s own experiments, and the authors call them early. They describe their tools as noisy: the phantom gradient points to the layer that needs help far better than it says what that layer should compute, and nobody yet has a reliable way to predict how a candidate layer will perform once trained. They kept the test sets hidden from the agent throughout. The post does not report any run on a model close to frontier scale being trained from scratch this way.

The 7-point gain on DeepSeek comes from rejecting answers and sampling again, which means each hard question can cost up to four attempts. Anyone comparing that number with a model that answers once should price in the extra compute first.

Reported by the DoubleAI research blog on 6 November 2026.