OpenAI chief scientist Jakub Pachocki has a message for anyone worried that Astra's recurrent architecture has turned the lab's frontier models into black boxes: it didn't.
The anxiety traces to The Information's report that Astra uses "recurrent depth" — meaning the same set of Transformer layers gets computed over and over. That lets some reasoning happen inside the model, rather than being written out as a visible chain-of-thought (CoT). The concern? Less of the reasoning happens in the open, making the models harder to monitor.
Pachocki is pushing back on that reading. He says the depth of the computational graph in OpenAI's current frontier models — Astra included — is at most twice that of GPT-4. That's not the multi-order-of-magnitude spike that some of the hand-wringing seemed to assume.
He does concede that CoT monitoring is fragile and heading in a harder-to-monitor direction. But he argues that's not because of the architecture shift. OpenAI has kept using CoT monitoring since its first generation of reasoning models, and strengthening that ability remains a core research goal.
Source: x.com/merettm/status/2095023204993490967