Why I Changed My Codex Evaluation Method Three Times: Comparing LLM Speed and Cost

This post is also available on X. View Post

I saw a post on X suggesting to “use whatever scores 70 or above.” At first, I agreed. But when I switched Codex from fast mode back to normal mode, I experienced a severe latency shock. It reminded me just how critical execution speed really is. (Unfortunately, I can’t recall the exact post.)

According to OpenAI’s coding agent charts, Sol produces high-quality outputs but is expensive. Terra and Luna, on the other hand, appeared relatively cost-effective. While Luna is introduced as a fast and economical model, is it actually still fast and cheap once you scale its reasoning effort up to xhigh or max to squeeze out coding quality?

I remember that when they were first introduced, the model speeds were shown with lightning icons: Luna > Terra > Sol (4, 3, and 2 lightnings, respectively). Now, however, all of them are labeled with 4 lightnings.

It was hard to make a judgment based solely on the model descriptions and charts. So, I decided to delegate the benchmarking process to the LLM itself. Consequently, this experiment evolved into three stages:

1. Comparing on a Small Coding Problem: We compared the base speed and cost of the models on the same small coding problem.
2. Re-running Real-World Tasks from Myven: Recognizing that a tiny problem doesn’t represent real-world development, we re-ran actual tasks from Myven.
3. Measuring Multi-turn Costs with an Initial Review: Factoring in the reality that implementation and review repeat as a project grows, we measured multi-turn costs including the first review edit.

I didn’t plan these three stages from the start. The limitations of the previous stage naturally forced the next experiment. In the process, I discovered a truth more important than the individual performance of each model.

[!NOTE] The actual performance of a coding agent is not determined by a single model in isolation. Rather, it is a combined outcome of the task, reasoning effort, test suite, evaluator, and review loop working together.

Success/Failure Criteria by Benchmark Stage

To conduct the experiment, we defined clear success and failure criteria for each stage.

Admittedly, the second success condition is harsh. As a result, almost all outcomes ended in failure—we will dive into the details below.

1. The First Experiment: Comparing Speed and Cost on a Small Problem

The goal of the first experiment was simple. I wanted to see which combination was the fastest and cheapest when keeping other conditions constant and only changing the model and reasoning effort.

Compare Targets and Setup

• Configurations Compared:
  • Luna xhigh / max
  • Terra xhigh / max
  • Sol high
• Additional Configurations Added:
  • Terra high
  • Sol medium

I directly ran the same benchmark-prompt using the Codex CLI command codex exec. We tried to keep the conditions as identical as possible:

• Used the normal mode with service_tier="default" (no fast or priority processing)
• Temporary session to minimize user configurations and MCP side-effects
• Changed only the model and reasoning effort
• 3 iterations per configuration
• Collected execution time and token metrics via JSONL events
• Inspected both the response format and the actual runtime behavior of the generated patch

The task was a relatively small problem involving analyzing and fixing a TypeScript concurrency bug. We limited the scope so that the model wouldn’t have to explore the entire repository or repeat multiple types of tests. The prompt content was as follows:

You are participating in a controlled coding-agent benchmark.

Do not use tools. Analyze the TypeScript implementation below. It is intended to provide a TTL cache with single-flight loading, authoritative invalidation, retry after loader failure, and per-caller cancellation that must not cancel the shared load.

type Entry<V> = {
  value?: V;
  expiresAt: number;
  pending?: Promise<V>;
};

export class AsyncCache<V> {
  private readonly entries = new Map<string, Entry<V>>();

  async get(
    key: string,
    loader: (signal?: AbortSignal) => Promise<V>,
    ttlMs = 1_000,
    signal?: AbortSignal,
  ): Promise<V> {
    const current = this.entries.get(key);
    if (current?.value !== undefined && current.expiresAt > Date.now()) {
      return current.value;
    }
    if (current?.pending) return current.pending;

    const pending = loader(signal).then((value) => {
      this.entries.set(key, { value, expiresAt: Date.now() + ttlMs });
      return value;
    });
    this.entries.set(key, {
      value: current?.value,
      expiresAt: current?.expiresAt ?? 0,
      pending,
    });
    return pending;
  }

  invalidate(key: string): void {
    this.entries.delete(key);
  }
}

Return exactly one valid JSON object, with no Markdown fence and no surrounding prose. Keep it under 700 words. Use this shape:

{
  "findings": [
    {"code": "STALE_RESURRECTION|REJECTION_POISON|SHARED_ABORT", "why": "..."}
  ],
  "patch": "a complete replacement implementation as a JSON string",
  "tests": ["exactly four concise test cases"],
  "tradeoff": "one concise sentence"
}

Include each finding code only if the code proves it. The replacement must preserve single-flight loading while ensuring: invalidate during an in-flight load cannot repopulate stale data; a rejected load can be retried; one caller aborting does not cancel the shared loader or other callers; an aborted caller rejects promptly.

Small Problem Results: Sol medium and Terra high Outperform

Out of 21 total runs, the core comparison was as follows:

From these results, the conclusion was clear:

• For Speed: Sol medium
• Balanced Cost & Speed: Terra high
• Lowest API Cost: Luna xhigh
• The max tier simply inflated reasoning tokens and latency without delivering extra quality.

Specifically, Sol medium was 26.4% faster and 16.8% cheaper than Sol high. Terra high was also 28.2% faster and 15.9% cheaper than Terra xhigh. At least for this problem, dialing up the reasoning effort by one notch did not yield any quality improvement.

Luna was also a surprise. Although its unit price is low, its reasoning time in xhigh and max became bloated. Luna max cost roughly the same as Terra xhigh but was about 3 times slower.

Based on this alone, it seemed like I should just select Sol medium or Terra high as default values for Myven development. But after finishing the experiment, one concern lingered.

“Isn’t this problem way too simple?”

2. Why the Small Benchmark Felt Lacking

The first experiment was useful for comparing the model’s reasoning speed and token efficiency. However, it was far removed from actual development.

In the small benchmark, the model’s only job was to read a single file and generate a patch. In contrast, real development in Myven involves these tasks working in tandem:

• Navigating multiple directories and existing implementations
• Editing GitHub Actions and shell scripts
• Confirming AWS and Terraform operating contracts
• Updating ADRs (Architecture Decision Records) and runbooks
• Running existing/new tests and diagnosing failure causes
• Reflecting code reviews, re-testing, and re-adjusting

The total time in the first benchmark was almost entirely model reasoning time. But in a real project, the breakdown of total time is very different:

$$\text{Total Time} = \text{Reasoning} + \text{Repo Exploration} + \text{Tool Invocation} + \text{Testing} + \text{Rework after Failure}$$

As the project size grows, the weights of the latter factors increase dramatically. A 10-second difference between Sol medium and high means nothing in a project where the full test suite takes 5 minutes. On the flip side, if high catches an operational contract missed by medium on the first try, the overall task completes much faster, even if individual calls are slower.

So, instead of repeating simple problems, I analyzed actual Myven development session records. Since task difficulty was entangled with model selection in the original sessions, the fairest method was to re-run past tasks from the exact same starting commit.

3. The Second Experiment: Re-running Real Myven Tasks

I selected three tasks of different natures from actual sessions:

1. Start of the First Task: Implementing a GitHub Actions workflow to turn the staging environment on/off at scheduled times.
2. Transition from Plan to Implementation: Restoring the existing PR deployment image when the staging environment turns back on based on an agreed plan, while preserving related runtime contracts.
3. Post-Review Implementation: Addressing code review feedback that misinterpreted AWS IAM simulation responses and adding tests.

Using the starting commits of each task, I created isolated repositories and assigned them to six configurations:

• Terra medium / high / xhigh
• Sol medium / high / xhigh

This made a total of 18 runs. To avoid touching the source tree, all runs were executed in isolated clones made using git archive. Network, AWS, and GitHub access, as well as commit/push operations, were strictly prohibited.

Success Criteria Was Not “Plausible-looking Code”

This time, we aligned the evaluation criteria closely with production deployments. A run was marked as PASS only if it satisfied all the following:

1. Codex CLI terminated normally within the time limit.
2. Actual code changes were present.
3. Passed git diff --check.
4. Adhered to the allowed file scope.
5. Passed syntax and static analysis checks.
6. Passed tests verifying the requested feature.
7. Passed hidden regression tests that were not exposed to the model.

This structure resembles the evaluation method of SWE-bench. Much like OpenAI’s SWE-bench Verified, the key was not just solving the new issue but ensuring that existing behavior was not broken.

The Success Rate was 2/18

In actual project tasks, the outcomes were starkly different from the small benchmark.

Only two out of 18 runs passed the strict criteria:

• Terra high on the first task
• Sol xhigh on the first task

However, a “FAIL” should not be interpreted as the model accomplishing nothing.

• Plan → Implementation Task: All six configurations implemented a significant portion of the core logic to preserve the deployment image. The failure point was mistaking a Lambda lookup error for a fresh initial state, falling back to the ECR image, and breaking the existing plan-on failure contract.
• Post-Review Task: Every configuration correctly shifted direction to parse resource-specific results of the IAM simulation. However, they all missed edge cases like optional top-level fields or unexpected types.

In short, the proportion of runs that reached a production-ready state in a single one-shot attempt was just 2/18.

[!TIP] Fixing the Evaluator Itself During the experiment, we uncovered bugs in the evaluator. Some implementations were correct in logic, but the evaluator demanded literal identity in step IDs or YAML formatting (e.g., accepting only on or "on" but not both). We corrected the evaluator (making it fail the start commit, pass the correct answer commit, and pass logically identical variants) to ensure reliability. To evaluate agents properly, you must first validate the evaluator.

4. Why a Multi-turn Benchmark Was Necessary

With a one-shot success rate of 2/18, a natural question arose. Real-world development is rarely a one-shot game.

As projects grow, we cycle through implementation → testing → review → modification → re-verification. Since token usage and latency accumulate through these loops, one-shot metrics do not tell the whole story of development cost.

However, letting the model run indefinitely in a multi-turn setup created a different issue. Models with higher reasoning tiers (especially xhigh) tended to explore paths not mentioned in the initial review, finding unrelated potential issues and extending execution times indefinitely.

Thus, we set clear limits on the multi-turn experiment:

1. Provide only a single blocking issue found in the first evaluation as feedback.
2. Limit the fix strictly to that issue without expanding to other files or scopes.
3. Prohibit the use of subagents.
4. Run the overall evaluation exactly once after the fix (no second review opportunity).

The two runs that passed one-shot were treated as having “zero additional implementation time.” This experiment was designed to measure “the realistic total time and cost to implement the initial request and fix a single review point.”

5. Allowing One Review Tripled the Successes (from 2/18 to 6/18)

The multi-turn results were as follows:

• One-shot Success: 2/18 Success after 1st Review: 6/18
• Recovery Rate after 1st Review: 25% (4/16)
• Average Latency Added by Review Fix: 1,664.9s (approx. 27.7 mins)
• Average API Cost Added by Review Fix: $6.49
• Combined Total Duration (Request + Review): approx. 2.18 hours
• Total Estimated Cost: $29.99

Providing just one review tripled the number of successful tasks. However, a review loop was not a magic bullet for all failed tasks.

6. The Nature of the Failure Mattered More Than the Model

Analyzing the multi-turn results by task category revealed stark differences based on the nature of the issue.

1) Clear Omissions (5/6 Success)

• Task: Restricting manual execution inputs to an on/off choice, or restricting scheduled triggers to the default branch. The locations and fixes were obvious.
• Outcome: 5 out of 6 runs passed after the review.
• Insight: These were solved within 36–53 seconds. Terra medium, Terra xhigh, and Sol high recovered easily. Clear omissions do not strictly require a high reasoning effort like xhigh.

2) Complex Fallback State Machines (0/6 Success)

• Task: Failing on specific exceptions (access-denied, throttled, etc.), preventing ECR fallback, and allowing fallback only under strict recovery conditions.
• Outcome: 0/6 passed—total failure even after the review.
• Insight: The review turn alone added 123–261 seconds per configuration. Sol xhigh worked the longest but still failed. This outcome underscores the need for reproductive tests rather than higher reasoning effort. Instead of writing verbose review text, we should provide fixtures mimicking AWS responses and test cases that the model can run directly.

3) Narrow Parsing Boundaries (Only Terra xhigh Passed)

• Task: Safely handling non-string formats and parsing optional properties in top-level aggregated fields.
• Outcome: Only Terra xhigh passed (taking 77.8s).
• Insight: Other configurations received feedback about optional fields but failed to integrate them safely into their code. This was a clear showcase of xhigh’s precise reasoning capabilities.

7. Model Rankings Shifted Across the Three Stages

The most interesting takeaway was that the preferred model kept shifting as we expanded the evaluation scope.

graph TD
    A[Stage 1: Small Problem] -->|Result| B(Sol medium / Terra high dominant)
    C[Stage 2: Real One-shot Task] -->|Result| D(Only Terra high / Sol xhigh pass)
    E[Stage 3: Multi-turn Review] -->|Result| F(Terra xhigh leads with 2/3 successes)

1. Small Problem: Sol medium was the fastest, Terra high had the best cost-to-speed balance, and max was unnecessarily slow.
2. Real One-shot Task: Only Terra high and Sol xhigh passed. The speed rankings of simple tasks did not translate to success rates in real-world codebases.
3. Multi-turn Review: Terra xhigh emerged as the most resilient, recovering to achieve 2/3 success. Sol xhigh was the most expensive and slowest, with no additional recoveries.

Ultimately, there is no single answer to “Which model is best?” The better question is:

“Is the failure I need to resolve a simple omission, a complex state machine, or a regression that the model cannot see?”

8. Principles for Operating Coding Agents in Myven

Based on these results, we now apply the following guidelines in our development workflows:

1. Start Initial Implementation with Terra medium or high
   • For small, localized changes, start with Terra medium.
   • For tasks involving multiple files or operational contracts, Terra high is a safe default. Avoid running xhigh right off the bat.
2. Keep Simple Reviews in the Same Configuration
   • Simple edits like missing guard clauses or incorrect types can be solved quickly by the same configuration. Keep the review feedback narrow and concrete instead of upgrading the model tier.
3. Promote to Terra xhigh for Complex Parsing and State Logic
   • xhigh excels at tight edge cases. However, since it is slow and costly, only upgrade to it once the complexity of the issue is confirmed after the first review.
4. If the First Review Fails, Do Not Increase the Reasoning Effort
   • Urging the same model to “try harder” wastes tokens and time.
   • What the agent needs is concrete feedback: failing tests, response fixtures, expected state transition logs, or executable verification commands. Breaking the task into smaller sub-tasks is far more effective.
5. Sol’s Premium Price Tag is Unjustified
   • Sol was fast in simple settings. However, in real-world multi-turn runs, Sol xhigh cost a whopping $8.45 with only a 1/3 success rate. (Note: This is based on three Myven AWS tasks and may vary in other domains.)

9. Lessons from Delegating Benchmarks to LLMs

Initially, I thought delegating the benchmark design and run to an LLM would give me a straightforward answer. The first experiment did exactly that—showing that Sol medium was fast, Terra high was cost-effective, and max was redundant.

However, attempting to apply this to real-world development highlighted the missing pieces: repository exploration and regression tests. Moving to real Myven tasks tanked one-shot success rates. Recognizing that actual work is iterative, we had to expand to a multi-turn setup.

The benchmark evolved through these stages:

$$\text{Reasoning Speed on Small Problem} \to \text{One-shot Quality on Real Project} \to \text{Total Dev Cost with Initial Review}$$

Understanding all three perspectives is essential for making sound operational decisions in real-world workflows.

10. Limitations of This Experiment

This experiment has several clear limitations:

• Sample Size: Due to resource and token constraints, we ran only one iteration per model/task combination. To filter out stochastic variance, we should run at least 5 iterations, which was not feasible here.
• Domain Bias: The three real-world tasks were focused on Myven’s AWS infrastructure and operations. The results might differ for frontend layouts, CRUD APIs, or refactoring tasks.
• Multi-turn Depth: We allowed only one review turn, whereas real workflows might utilize multiple turns and test output streams.
• Cost Estimation: Costs are based on token counts reported by the Codex CLI using standard API pricing and may differ from actual invoicing.

Conclusion: The Feedback Loop Trumps the Model

An agent’s coding capability is not defined by a single benchmark score on a model card. Real-world performance is a product of the model, the reasoning budget, the repository architecture, the test suite, the evaluator, and the granularity of the review loop.

Currently, automating evaluations or delegating them entirely to LLMs remains difficult, and maintaining a robust loop harness is a constant challenge. The path to “perfect one-click automation” is still far away.

We must focus on creating environments where the model can easily understand constraints and test its code directly, rather than simply blaming the model.