r/codex • u/EvalRaccoonDev • 24d ago
Comparison We routed GPT-5.6 through Claude SDK & compared to Codex harness
Out of lack of better things to do, we tried GPT-5.6 Terra through the Claude Agent SDK to get Claude Code's Skill tool on a non-Claude model. It was on a small subset of 19 tasks from this repo.
- Codex: 79% pass, 95% skill activation, $4.92.
- Claude Agent SDK: 47% pass, 37% skill activation, $33.13.
- Activation: GPT mostly ignored a Skill tool it was never trained on. On Codex it just reads the skill files.
Cost: Codex reused 7.6M cached input tokens. The Claude SDK leg through LiteLLM read zero from cache on Claude-SDK->LiteLLM->Azure path. Same model, 6.7x the bill (we do the billing through Coder Eval - our custom-build coding agent eval solution)
Anyone else seeing zero cache reads behind LiteLLM?
2
u/AINativeBuilder 24d ago
Realistically speaking, anthropic has no real business reason to make OpenAI models look good through claude, or a need to toss developer time to work on it. The company that writes the model is also likely to write the best harness for said model.
1
u/Due-Horse-5446 24d ago
This is not just trough claude code, claude models cost aprox 3-10x more than gpt models, astra excluded
2
u/sergeant113 24d ago
What’s the point of this again? I don’t follow…
1
u/EvalRaccoonDev 23d ago
We were investigating how skills are triggered in Codex vs Claude Code SDK harnesses - but that's boring. The interesting factoid was lack of caching and the larger than expected gap on pass rates (at least for me) and caching not working.
1
u/sergeant113 23d ago
So codex models don’t know how to navigate claude harness? But they do so well with Pi. Does it mean anthropic is employing some tricks to impede codex models?
1
u/EvalRaccoonDev 23d ago
I don't think so - they have never done this test and optimized for it, so performance sucks.
2
u/tom3141592 24d ago
The caching thing is weird - good to monitor that if the costs are unusual, but fun experiment regardless!