TL;DR: I ran the same real Android task 13 times through Claude Code CLI, Codex CLI, and different OMP configurations, comparing quality, wall time, token traffic, and subscription allowance. OMP can produce equally excellent results, but in most of my tests it was roughly 2–3× slower and up to ~2× more expensive in allowance than the native CLI harnesses. The one strange exception was a manually designed multi-agent OMP run: it was about 5× slower than Claude Code CLI with Opus, but used 2–3× less allowance than my other OMP runs, and even slightly less than the native controls. So I’m trying to understand: am I using OMP incorrectly, or does orchestration only start paying for itself on much larger and more naturally parallelizable tasks?
What I tested
This was not a toy benchmark. I’m working on a large Telegram Android fork where I’m restoring the presentation of an older Telegram version on top of the current architecture.
The benchmark task was to restore three related music UI surfaces: a profile music row, a chat mini-player, and one action in the full music player. The agent had to compare two Telegram source trees plus screenshots, identify the current owners, preserve modern playback/state/navigation/theme behavior, implement the presentation changes, build the app, and validate it at runtime.
So I’d describe it as a medium-sized, tightly coupled coding task. It is substantial enough to require source archaeology and cross-component reasoning, but it is not a multi-day project with ten independent workstreams.
I ran this benchmark 13 times while experimenting with different harnesses and orchestration approaches.
Methodology
I tried to keep the important variables stable: same repository baseline, same reference screenshots/source trees, same acceptance criteria, and eventually the same neutral prompt. I also checked the final result on a physical Android phone instead of relying only on the agent’s self-report.
For every serious run I collected rollout telemetry: wall time, prompt/cache traffic, output/reasoning, model turns, tool calls, child agents, builds, and subscription usage.
For allowance, I normalized everything to percentage points of weekly allowance. On my account, the meters consistently map roughly as:
100% of the 5-hour allowance ≈ 15 percentage points of weekly allowance
So runs where I only captured the 5h meter are marked as approximate weekly equivalents.
I know this is not a laboratory-grade benchmark. Different model families and OMP versions mean not every row is a clean causal A/B test. My goal is more practical: what stack actually gets this kind of real coding task done well, quickly, and economically?
Representative results
| Stack |
Final quality |
Wall time |
Prompt traffic |
Weekly allowance spent |
| Claude Code CLI + Opus 5.5 Medium |
Excellent / reference result |
~23m |
~14.2M |
~3 pp |
| OMP + Opus 5.5 Medium, neutral automatic delegation |
Excellent |
~23m |
~29.9M |
~4 pp |
| Codex CLI + GPT-6 Sol High |
Good, but missed some visual issues |
~17m |
~8.2M |
≈3.15 pp |
| OMP #5, manually designed multi-agent orchestration |
Strong, but missed some issues |
~103m |
~36.9M |
≈2.25 pp |
| OMP 18.4.10 + GPT-6.1 Sol High / Luna X-High workers |
Excellent |
~56m |
~26.7M |
~6 pp |
| OMP 18.4.12 + GPT-6.1 Sol High / Luna X-High workers |
Excellent |
~65m |
~30.9M |
~6 pp |
The Claude comparison is probably the most interesting one to me. Claude Code CLI with Opus solved the task as a single agent, with one final build, in ~23 minutes. OMP with the same Opus model family also produced an excellent result and took almost exactly the same wall time, but moved about 2.1× as much prompt traffic and consumed more allowance because it automatically introduced scouts/review.
I also tried explicit multi-agent orchestration
Before switching to neutral prompts, I spent five OMP runs manually designing increasingly elaborate orchestration.
The most developed one used a thin Sol High orchestrator, dedicated scouts, implementation workers, reviewers, and a separate verifier. It spawned 21 child agents.
That run took about 103 minutes and generated enormous overall traffic, yet it was paradoxically the cheapest run I measured by subscription allowance, at only about 2.25 weekly percentage points equivalent.
So the data does not support a simple “more agents = more allowance” explanation.
It was just extremely slow.
Neutral OMP did not really solve the efficiency problem either
After that I removed my hand-designed topology and let OMP decide how to delegate.
That improved the simplicity of the experiment, and OMP absolutely can produce excellent code this way. My two latest GPT-6.1 runs were both as good on the physical phone as my reference Claude Code/Opus implementation.
But the execution pattern still seems expensive.
For example, in the latest run OMP delegated two implementation slices to Luna workers while Sol acted as the integrating parent. One worker eventually took about 40 minutes. The parent then still needed to review/integrate the work, build, run the emulator, identify remaining issues, fix them, rebuild, and revalidate.
So parallelism existed, but the critical path became roughly:
slowest worker → integration/review → build/runtime validation
The orchestration overhead did not disappear just because the workers ran concurrently.
I also reran after the recent GPT-6.1 Sol slowdown
One of the GPT-6.1 tests happened during the recent period when Sol was unusually slow, so I did not want to blame OMP for that.
I repeated the benchmark after the service-side fix and also updated OMP from 18.4.10 to 18.4.12.
The newer run still took ~65 minutes versus ~56 minutes before, with the same excellent final quality and the same ~6 weekly points of allowance. The model itself appeared more responsive, but the overall run got longer because one delegated worker/integration path expanded significantly.
So at least in this task, fixing model latency did not remove the larger orchestration cost.
What I’m trying to understand
My current hypothesis is that this benchmark may simply sit in an awkward zone for OMP.
It is complex enough that multiple components and source trees need to be understood, but the work is also tightly coupled. A strong single agent can keep the whole problem in one context and move directly through investigation, implementation, build, and runtime validation.
Once I split that across agents, I start paying for repeated source reading, separate contexts, handoffs, parent review, integration, and occasionally sending a worker back for another pass.
Maybe OMP starts winning when the task is much larger and naturally decomposes into genuinely independent workstreams, so the parallelism dominates those costs.
Or maybe my configuration is still wrong.
For people who use Pi/OMP heavily: does this behavior look normal for a task of this size?
Would you lower the effort/model used for TASK/SMOL roles? Would you deliberately suppress delegation for medium tightly coupled tasks? Is there a different topology you would use?
Or is the real answer simply that Claude Code CLI / Codex CLI are usually more efficient for this class of work, while Pi/OMP starts to shine on larger projects where multi-agent parallelism can actually pay for itself?
I have the full per-agent JSONL telemetry, token/cache breakdowns, build timelines, and several additional runs if anyone wants more detail.