Does Raising Reasoning Steps Make Coding Agents Better?

Does Raising Reasoning Steps Make Coding Agents Better?

I compared the medium, high, and xhigh settings of Terra and Sol across 60 sessions in normal mode. Along with time, tokens, and estimated cost, I measured functionality, regression, and edit scope based on strict validator criteria, as well as the results of the fresh-context first-review complement.

There is a common hypothesis when choosing a coding agent: raising the reasoning steps will make it slower and more expensive, but will yield better results. There is also an expectation that using a stronger model will help it find relevant paths in complex repositories better, and that even if the first implementation is lacking, the completeness will improve through reviews.

The results of this experiment were more complex than that.

In the balanced sample, the fastest setting was terra medium. For both Terra and Sol, the medium setting was faster and used fewer tokens than xhigh across all 9 task-iteration pairs. This efficiency gap was robust. On the other hand, the quality ranking could not be definitively determined. This was because there were only three tasks, and the strict edit scope rules judged even the related test files allowed in the public tasks as out-of-scope.

In other words, what these results strongly showed was the time and token cost of xhigh. Whether higher reasoning does not improve quality, or whether it modifies more unnecessary paths, needs to be measured again with adjusted scope rules and a wider variety of tasks.

What I Wanted to Confirm in the Experiment

This benchmark was designed to answer the following questions:

1. Does raising the reasoning steps from medium to high and xhigh increase the number of completed tasks that meet all success criteria?
2. Does the relatively faster Terra family maintain its speed and cost advantages in actual repository tasks?
3. Does the additional time and cost of the Sol family lead to improved quality?
4. When the first implementation fails, can a fresh-context agent recover the results by complementing only the first problem identified by the deterministic validator?
5. Do higher reasoning steps lead to unnecessary modifications while exploring more paths?

My expectations before the experiment were as follows:

• medium will be fast and cheap but will have many omissions in complex tasks.
• high will be the sweet spot between speed and quality.
• xhigh will be the slowest and most expensive but will show the highest task completion rate in difficult tasks.
• Sol will be slower and more expensive than Terra but more stable in complex integration tasks.
• The limited complement calls of fresh-context will turn some incomplete results into completed tasks.

As a result, the expectations regarding speed and resource usage were relatively clearly confirmed. The quality-improving effect of higher reasoning was not observed under the strict criteria, but it could not be concluded that there was no effect because the results were sensitive to the scope definition. In the fresh-context first-review complement, transitions to strict completion were not observed either.

Three Generalized Tasks for Public Release

The three tasks were created based on maintenance patterns that repeatedly appear in actual product development. However, in this article, product names, repository structures, file and function names, API addresses, commit and execution identifiers, and implementation details of the hidden validator have been removed or generalized.

These tasks do not represent current defects or security vulnerabilities in active production services. They are problems designed to measure the implementation behavior of models using isolated repository snapshots and benchmark agreements.

Task A: Normalizing External Event Identifiers

I assumed a situation where an external platform’s event payload can use both the older flat format and the new nested format. The implementation had to satisfy the following conditions:

• Support both the existing format and the new format.
• Adhere to the designated priority when both values exist.
• Do not raise exceptions on null, empty values, or incorrect data types.
• Consistently use the normalized identifier in storage and subsequent routing.
• Accurately handle specific target selection and the default fallback.
• Do not modify unrelated payload fields.

Although it looks like a small code change, it is a task that requires understanding input boundaries, storage paths, subsequent branching, and regression tests together.

Task B: Refresh Stability of Async Detail Screens

On a screen that asynchronously loads detailed information, a stable UI state had to be maintained even when initial loading, refreshing of the same item, errors, and rapid item switching overlapped.

• Show a clear and accessible loading state during the initial request.
• Maintain existing information while refreshing the same item.
• Express both the existing information and the error state even if the refresh fails.
• Reflect only the success results of the latest request on the screen.
• Prevent delayed old responses from overwriting the latest screen.
• Appropriately remove the information of the previous item when a different item is selected.
• The state should not break even under abnormal responses.

This is a task where understanding the existing state model and request race conditions accurately is more important than the code itself.

Task C: Reflecting Browser Read State

I asked the model to implement a small client that securely reflects the read state of notifications or announcements to the server in the browser.

• Obtain a security token with same-origin credentials before the request.
• Correctly configure security headers and JSON requests.
• Maintain a failed state on HTTP errors, invalid tokens, and network exceptions.
• Remove the unread indicator on the screen only after the server request succeeds.
• Operate safely even in server-rendered environments without a browser DOM.
• Prevent duplicate requests.
• Do not arbitrarily change the injected address and functionality for testing.

Although this task has a small amount of code, it had to simultaneously align existing module boundaries, the browser environment, security request sequences, and UI reflection timing.

How Task Completion Was Determined

Simply finishing the model process normally or having the model respond that it had “completed” was not considered task completion. Final completion was defined as satisfying all of the following five axes simultaneously. The task completeness in this article does not mean actual deployment or release approval, but the rate of simultaneously satisfying the benchmark’s strict success criteria in a single session.

1. Functionality: Satisfy the pre-fixed behavioral contract.
2. Regression: Do not break existing unrelated behavior.
3. Strict Edit Scope: Modify within the scope allowlist locked in the validator. This allowlist does not completely match the reasonable modification scope of the public task, requiring a separate sensitivity analysis.
4. Process: Execution and result collection end normally.
5. Validation Integrity: The deterministic validator reproduces the same result in an isolated environment, and the candidate code and verification code do not contaminate each other.

This distinction is important when interpreting the results. However, a post-audit revealed issues with the construct validity of the scope configuration. The public task allowed related implementations and independent tests, but the hidden allowlist treated some related test files as out-of-scope. Therefore, the strict incompletion rate in this article is not the “rate of not being able to write code at all,” but the “rate of not satisfying all locked success criteria simultaneously,” and scope failures among them should not be immediately interpreted as unnecessary modifications.

Metrics and Score Criteria

Since a single pass/fail makes results that almost completed the functionality and results that could not even find the requirement look like the same failure, I decomposed the results into quality, completeness, execution reliability, and efficiency.

Quality and Completeness Metrics

Each score converted the ratio of sessions that passed the corresponding condition into a scale of 0 to 100. To prevent tasks with a large number of assertions from excessively dominating the score, a single session counted as a maximum of 1 case on each axis. While this method aligns weights by task, it has an all-or-nothing cliff. For example, in Task B, session-level functionality pass was 0/18, but individual functional contracts met 67/108, or 62.0%. Therefore, the session axis score and the coverage by contract must be viewed together.

Task completeness is not the average of functionality, regression, and scope scores. Even if the three conditions pass individually in different executions, it is not calculated as complete. All success criteria must be satisfied simultaneously in the exact same session.

Execution Reliability Metrics

Normal process termination and validation integrity were also recorded from 0 to 100. In this experiment, all settings scored 100 on both axes. Since this is not evidence of a difference in model performance, but evidence that 60 results were collected without tool errors or validator contamination, it was not included in the model ranking score.

Efficiency Metrics

Efficiency scores were calculated only from the initial balanced sample n=9, where the number of samples per setting was identical. The top-performing setting on each axis was set to 100, and the rest were converted to relative scores.

Speed Score = Fastest protocol wall-clock median / Median of the setting × 100
Token Efficiency = Total tokens of the fewest 9 sessions / Total tokens of the setting × 100
Cost Efficiency = API-converted cost of the lowest 9 sessions / API-converted cost of the setting × 100

For time, the median was used to reduce the influence of a single long run. For tokens and cost, which are actually cumulative resources, the totals of the same 9 sessions were used.

Auxiliary Composite Score

The score below is an exploratory example calculated for operational choices at the time. It is not used as a basis for quality ranking.

Auxiliary Composite Score = Functionality 25% + Regression 15% + Scope 15% + Task Completeness 15%
                            + Speed 10% + Token Efficiency 10% + Cost Efficiency 10%

This score is a utility function determined after checking the results, not a statistical performance score of the model. It partially double-counts quality by including functionality, regression, scope, and completeness (which is their intersection) together, and simultaneously reflects costs calculated from tokens and token unit prices. Since the ranking changes depending on the weights, it is excluded from the text’s conclusion, and raw axes and sensitivity analysis are prioritized.

Test Plan

The six settings compared were:

• Terra: medium, high, xhigh
• Sol: medium, high, xhigh

All execution conditions were locked identically:

• Use isolated exec execution of Codex CLI
• Run first implementation and complement as independent --ephemeral calls respectively
• Use normal service tier, fast mode disabled
• Concurrency of 1
• Model-internal multi-agent disabled
• Recreate each session fresh from the same repository snapshot
• Separate original work tree and candidate workspace
• One-time executions excluding user personal settings
• Use pre-fixed functionality and regression contracts
• Use a deterministic validator that does not rely on network time or arbitrary waiting

Step 1: Balanced Comparison

Every combination of the three tasks and six settings was executed independently three times.

3 tasks × 6 settings × 3 times = 54 logical sessions

Because each setting has exactly 9 samples, this balanced sample was used when comparing the speed, tokens, and cost of the model and reasoning steps.

Step 2: Expanding the Number of Iterations to 5 for Uncertain Cells

For only three task-setting combinations where the initial results were mixed or the failure types differed per iteration, I added a 4th and 5th run.

3 selected cells × 2 additional runs = 6 sessions
Final Total = 60 logical sessions

The expanded cells were as follows:

• Task A / sol high
• Task A / terra xhigh
• Task C / terra xhigh

Actual coding agent tasks rarely end with the first response. Humans look at verification results to request the next complement, and automated harnesses also try to finish the task by controlling the loop based on test and review results. Therefore, practical quality judgment should include not only the average quality of the first run but also how stably it completes when the same conditions are repeated.

However, the 5-run expansion in this experiment was not extending a single session up to 5 turns. It was repeating the same task-setting combination up to five times in isolated, independent sessions to confirm non-determinism and reproducibility. The implementation → validation → complement loop itself was measured separately in the C1 protocol below. In other words, the 5-run expansion reinforces the reproducibility and completion stability evidence required when running an automatic completion loop, but does not directly show whether the task converges within 5 turns.

The 5-run expansion makes the sample size across all settings imbalanced. Therefore, the first 54 balanced samples were used for fair speed/cost comparisons of all settings, and in the quality analysis, they were included as evidence to judge the variability and repeated completion of the selected task cells.

C1: Fresh-context Multi-turn Restriction

Each logical session proceeded in the following order:

1. The model performs the first implementation.
2. The deterministic validator evaluates functionality, regression, and scope.
3. If the first implementation fails and there is a publicly disclosable product defect, only the single first finding with the highest priority is delivered to the model.
4. The model complements only that finding.
5. The entire validation is executed again, and the session ends.

The first implementation and complement were executed as isolated codex exec --ephemeral calls, respectively. The complement call inherits the repository state of the first implementation as-is, but does not inherit the previous conversation or search process, receiving only the single deterministic first finding. This design was not a mistake, but was intended to control quality degradation due to long-context contamination and cumulative context, and to provide the exact same complement information to all settings.

Therefore, multi-turn here does not mean continuing the same conversation thread, but refers to the workflow stages of work performance → external verification review → complement of findings. The measured value is also interpreted not as the “recovery rate of the same conversation,” but as the strict completion transition rate after single-finding complement of a fresh-context agent inheriting the modified code state. This is an experiment that controlled the impact of long-context quality rather than measuring it.

If the first implementation passed, there was no complement call. Even when only out-of-scope edits remained, no separate complement was given. This restriction was to prevent the model from finding more problems and modifying endlessly, and to compare the operating cost up to the first implementation and fresh-context single-finding complement.

As a result, there were 80 actual Codex calls in 60 logical sessions.

Token and Cost Calculation Method

Prices used the official price list of July 13, 2026, which was locked at the start of the experiment.

Each Codex call was calculated independently. If the input of one call exceeded 272,000 tokens, the input unit price was doubled, and the output unit price was multiplied by 1.5. Out of the 80 calls, 77 exceeded this long-context threshold.

Token metrics should be interpreted as follows:

• input tokens includes cached input.
• cached input tokens is a subset of input tokens.
• output tokens includes reasoning output tokens.
• Since reasoning tokens is a subset of output tokens, it was not added back to the total tokens.
• Total tokens was calculated as input + output.

Cost is an API list price conversion estimate calculated using the fixed price list and the token usage reported by the CLI. It is not the actual billed amount of Codex subscription fees or quotas, but the value obtained by applying the public API unit price of the same model to the tokens reported by the CLI. It is also not a value cross-checked against actual invoices. Furthermore, because Codex CLI does not expose cache-write tokens separately, any unobserved additional cache-write costs are not included in this estimate.

Result 1: Fair Comparison Between Settings

First, let’s look at the initial balanced sample of n=9 per setting.

The time in this table is the protocol wall-clock, which includes not only model calls but also deterministic validation and waiting for review approval. The median pure model call times were 231 seconds for terra medium, 268 seconds for terra high, 335 seconds for sol medium, 412 seconds for sol high, 563 seconds for terra xhigh, and 789 seconds for sol xhigh. Even in pure model time, Terra and Sol’s medium were faster than xhigh in all 9/9 task-iteration pairs respectively.

The graph above is the median pure model execution time excluding the validator and approval wait. Only the initial balanced sample of n=9/setting was used, and lower is better. terra medium was the shortest at 231 seconds, and sol xhigh was approximately 3.4 times longer at 789 seconds.

Under strict criteria, raising the reasoning steps increased time and tokens, but did not increase completions. However, because this completion rate is sensitive to the scope allowlist, it cannot be used as a basis to confirm quality effects.

• terra medium, terra high, and sol medium all passed 3/9.
• terra medium was the fastest and had the lowest estimated cost among these three.
• terra xhigh had a median about 2.7 times longer than terra medium but passed one fewer case.
• sol xhigh used the most tokens and cost, and had zero strict completions.

If only the related test files are recognized as a reasonable modification scope, the results change significantly.

The gray bars represent the strict completion rate applying the pre-fixed allowlist as-is, and the green bars represent the sensitivity analysis allowing even the test files directly related to the implementation. The green bars should not be read as separate success rates or official pass rates observed in re-runs. It is an auxiliary metric showing how much the completion rate changes depending on which scope rule is used to judge the same output.

This sensitivity analysis shows that the strict completion ranking was heavily influenced by the definition of the evaluation scope, not just the model behavior. Therefore, while terra medium is supported as the default speed/cost candidate for this sample, it cannot be concluded as the best setting for quality. In addition, the 54 sessions repeated three fixed tasks three times, so the diversity of independent tasks is only three.

Multi-Axis Scores by Setting

If I divide the same balanced sample into functionality, regression, scope, completeness, and efficiency, the meaning of the results changes.

sol xhigh was not a setting that failed to perform the tasks at all. Its functionality score was 66.7 and regression was 100. Because the scope score of the strict allowlist was 33.3, its completeness became 0, but in the sensitivity analysis allowing related tests, it changes to 6/9. sol high also had high regression safety and was 6/9 under the adjusted criteria.

terra medium showed the best observed values in the three efficiency axes. However, because the multi-axis score also includes the problematic scope axis and completeness (its intersection), this is not read as a comprehensive quality/efficiency victory.

Multi-Axis Scores by Task

The fact that the type of task had a greater impact than the setting is clearer in the scores by task.

Tasks B and C both scored 0 on the session-level completion table, but the reasons were different. In Task B, all sessions failed to meet the entire functional contract simultaneously, but individual functional contracts were met in 67/108 (62.0%) cases. In Task C, almost all sessions passed functionality and regression validation, and the 20/20 strict scope failures were all cases where a single related test file was outside the allowlist. This result is evidence that session axis scores, contract-specific coverage, and scope sensitivity must be reported together.

Result 2: Quality Variability Observed in 5-Run Expansion

The results of the 5-run expansion for the three selected cells are as follows:

Under strict criteria, both sol high and terra xhigh for Task A were 4/5. Both settings passed functionality and regression verification 5/5, and each became a scope failure once because the related integration test file was outside the allowlist. Allowing related tests makes both 5/5.

Within the same task and the same number of successes, terra xhigh was more advantageous:

• The median was about 95 seconds shorter.
• Token usage was about 28% lower.
• The 5-run cumulative estimated cost was a little less than half.
• The observed IQR was also much smaller.

However, the sample size is only five. The Jeffreys 95% interval for 4/5 task completeness is very wide, approximately [0.371, 0.977]. Therefore, this result is an observation in that specific task, not evidence determining the general superiority of one model over the other.

Looking at the 5-run expansion results again through the multi-axis score:

The two settings for Task A have the same quality profile. In this case, the efficiency difference where terra xhigh used less time, tokens, and cost becomes the actual selection criterion.

Task C’s terra xhigh had strict completeness of 0, functionality of 60, and regression of 100. The strict failure signature was divided into three scope failures and two unmet functionalities, but all three scope cases were due to the related test file. Under the sensitivity criteria allowing this, it changes to 3/5. Although randomness affected functional contract fulfillment, the strict completion of 0/5 alone cannot judge the scope control ability.

When including the 5-run expansion results in the quality judgment, sol high and terra xhigh showed the same functionality, regression, and adjusted completion quality in Task A, while in Task C, the functionality fulfillment of terra xhigh fluctuated across runs. Therefore, a consistent quality advantage of higher reasoning was not confirmed in this set of tasks. This conclusion concerns the repeated reliability of the three selected cells and is not a general quality ranking comparing all settings 5 times.

Result 3: Total Tokens and Cost

Usage across the entire experiment, including the 5-run expansion, is as follows:

About 93.35% of input tokens were recorded as cache reads. Without cache, the cost of repeatedly loading context in large repositories would have been much larger. Conversely, even with a high cache ratio, the total estimated cost reached about $195 because of three overlapping reasons:

1. Most calls exceeded the long-context threshold.
2. Input and output tokens both increased in higher reasoning steps.
3. Sessions requiring review complement after the first implementation used additional calls.

Looking only at the initial 54 balanced samples, a total of 120.03M tokens and about $179.12 were used. The additional 6 sessions added about 11.74M tokens and $15.84.

The overall average was about $3.25 per session. The simple converted cost divided by 17 strict completions is about $11.47, but because the denominator changes depending on the scope sensitivity, this is not viewed as a productivity metric to be used for model purchasing decisions.

Quality Advantage of Higher Reasoning Was Not Confirmed Even with 5-Run Expansion

Including not only the initial balanced sample but also the 5-run expansion, there is no evidence that higher reasoning produced higher quality in this task group. In Task A, sol high and terra xhigh were identical with functionality/regression of 5/5 and adjusted completion of 5/5. In Task C, terra xhigh fluctuated across iterations, with functionality and adjusted completion at 3/5. Observations that xhigh is more stable or complete than lower reasoning settings did not emerge.

However, the 5-run expansion selected only the three cells whose initial results were uncertain, while the remaining settings stayed at 3 runs. Because it is an imbalanced sample with expansion targets chosen after checking the results, one cannot directly compare the cumulative completion rate of all settings or conclude that “higher reasoning generally has no quality effect.”

Under strict criteria, functional correctness and task completeness were also separated. A significant portion of this difference occurred in scope rules rather than model behavior.

Task A’s sol xhigh passed functional validation 3/3, but its strict scope was 0/3. Task C’s 20/20 scope failures were also all due to a single related test file. Allowing related tests changes sol xhigh’s initial completion from 0/9 to 6/9. Therefore, this cannot be used as evidence that “higher reasoning modified more unnecessary paths.”

Generally, it is possible that higher reasoning steps explore more paths, expanding the patch. However, the current data did not verify that hypothesis. To confirm this, the following must be measured separately under scope rules matching the public prompt:

• Number of modified files and diff size
• Whether files completely unrelated to the requirements were modified
• Distinction between new tests and existing test enhancements
• Functional contract coverage and actual regression
• Blind human review’s judgment of minimum patching

Therefore, the conclusion within this experiment is that no quality advantage of higher reasoning was observed even when including the 5-run expansion. The presence and scope of general quality effects and scope control effects must be re-confirmed in experiments repeating all settings the same number of times and adjusting the scope.

Was There an Effect in Complementing Just One First Review Finding in Fresh-Context?

In this protocol, after preserving the file state of the first implementation, I delivered only the first finding of the deterministic validator with the highest priority in a separate ephemeral call. A total of 20 sessions used this fresh-context complement, but 0 sessions transitioned to completion under strict criteria. This value means a strict completion transition rate of 0/20 for fresh-context single-finding complements.

This result does not measure the recovery ability of the model receiving reviews in the same conversation thread, because I intentionally removed long-context influence using --ephemeral. Additionally, 18 out of the 20 cases were the same first finding of the same Task B, and 2 cases were the exact same finding of Task C, so it was not 20 diverse review problems. Therefore, one cannot generalize that “models cannot reflect reviews” or “multi-turn reviews are generally ineffective.”

However, there are practical lessons:

• Review findings must be distinguished as to whether they are independent or have a sequential relationship.
• When having the model fix only the first finding, full regression validation must be run again.
• Complex state management or exploration of existing module boundaries may place a higher importance on quality during the planning phase than on fresh-context single complements.
• Before increasing the number of complements, one must check whether the validator accurately separates actual product defects from scope issues.

Analysis Methods and Interpretation Principles

Speed prioritized the median over the average, and variation between iterations was confirmed via IQR. To avoid hiding the uncertainty of small success samples, the Jeffreys 95% interval was used for the 5-run cells.

I also adhered to the following principles:

• Comparisons between settings use the initial n=9 results where the sample sizes are equal.
• The 5-run expansion results are used to judge quality variability and repeated completion within the selected task cells.
• Do not lump functionality, regression, scope, and process failures into a single “model failure.”
• Tokens and cost are calculated per call and then summed.
• Do not double-count reasoning tokens with output tokens.
• Do not interpret overall task completeness as the model’s universal coding ability score.
• Present the sensitivity difference between strict scope criteria and related-test-allowed criteria together.

Particularly after the 5-run expansion, the total sample sizes of sol high and terra xhigh are larger than other settings. If the cumulative cost or number of successes in this state is used as the setting ranking, selection bias occurs.

How to Apply in Practice

The operational strategy directly supported by this data is not to choose xhigh from the start when prioritizing time and tokens. The default setting for quality cannot be finalized before experiments with adjusted scope.

The cost-effectiveness graph shows the estimated API list price per setting of the initial balanced sample divided by the number of completions under the related-test-allowed criteria. terra medium was about $2.05 per adjusted completion, and sol xhigh was about $10.20. Lower is more cost-effective, but because the denominator is scope sensitivity-adjusted completion rather than strict completion and is not the actual Codex subscription bill, it is used only as an indicator for decision-making between settings, not as an absolute price tag.

Small and Clearly Bounded Maintenance

terra medium is a good candidate for the default speed/cost setting in this fixture suite. It was the fastest in protocol wall-clock and pure model time, and used the fewest tokens and cost. However, because strict completeness and the auxiliary composite score are sensitive to the scope definition, they are not used as evidence of quality superiority.

Tasks Requiring Exploration but with Clear Modification Scope

terra high can be used, but there is no evidence yet that it has higher quality or is safer than medium. The need for additional time and tokens compared to medium must be judged by the characteristics of the task.

Tasks Requiring Repository-Wide Path Exploration

xhigh was consistently slower and used more tokens than medium. While setting range limits when using it is useful as a general operational safeguard, this experiment does not prove xhigh’s tendency to exceed scope.

• Perform only investigation and planning first.
• Finalize the files to be modified and the scope of responsibility.
• Do not implement collateral problems found; leave them as a separate list.
• Run scope validation as well as functionality after modification.

Rather than using higher reasoning as a default, it is easier to control time and cost by planning in medium or high and applying higher reasoning only to tasks that truly need it.

Validated Operational Principles and Next Experimental Hypotheses

When translating these results into actual project operations, one must distinguish confirmed conclusions from unverified hypotheses.

Therefore, the most rational operational hypothesis at present is to start the first implementation with medium to control costs, and selectively upgrade the complement setting based on validation results. Particularly, terra medium → sol high is a candidate to combine speed and regression safety, but to make this a recommended default, a cross-setting experiment is required where terra medium and sol high complement the same first implementation results respectively.

Limitations

This experiment has clear limitations:

• Only three fixed tasks were used.
• Task difficulty and integration scope were not identical.
• Only three selected cells were expanded to 5 runs, while the others remained at 3.
• Model outputs have non-determinism, so the same setting can choose different paths.
• Costs are estimates based on a fixed price list, not actual billed amounts.
• Cache-write costs are outside the CLI observation scope.
• C1 is a fresh-context complement that only hands over the file state of the first implementation, and the long-context effect of the same conversation was not measured.
• 18 out of 20 C1 complement cases concentrated on the same first finding of the same Task B.
• The public task and hidden scope allowlist collided on some related test files.
• Session-level functionality scores can hide partial fulfillment of contracts.
• The three tasks do not represent all planning, design, migration, and operational responses of full product development.

Therefore, this result is a comparison of behaviors within a specific fixture suite. It should not be read as an absolute score of the models or a leaderboard applying to all coding tasks.

Price Sources

• GPT-5.6 Terra Model Document
• GPT-5.6 Sol Model Document
• Introducing GPT-5.6

Prices and calculation rules used the July 13, 2026 snapshot locked at the start of the experiment.

Summary

• Compared medium, high, and xhigh of Terra and Sol in normal mode.
• Collected 60 logical sessions and 80 Codex calls by repeating balanced samples of 3 tasks and 6 settings three times in independent sessions, and then expanding three selected cells to 5 independent sessions.
• Separated functionality, regression, strict scope, strict completeness, speed, tokens, and cost, and additionally reviewed contract-specific coverage and scope sensitivity.
• terra medium was the fastest in protocol wall-clock and pure model time, and had the lowest estimated cost.
• For both Terra and Sol, medium was faster and used fewer tokens than xhigh in all 9/9 task-iteration pairs respectively.
• Strict completion was 0 to 3/9 by setting, but allowing related tests changed most to 6/9, making the quality ranking sensitive to the scope definition.
• Task B had a session functionality pass of 0/18, but met 67/108 (62.0%) of individual functional contracts.
• The auxiliary composite score is a post-hoc utility function double-counting correlated axes, and is not used as a model quality ranking.
• For Task A, both sol high and terra xhigh were 4/5, but terra xhigh was faster and used fewer tokens and estimated cost.
• A consistent quality advantage of higher reasoning was not confirmed in this task group, even when including the 5-run expansion. However, because only three selected cells were expanded, one cannot finalize that there is no general quality effect.
• Total usage was 131.78M tokens, estimated cost of approximately $194.96, and protocol wall-clock total of approximately 8 hours 23 minutes.
• C1 is a 2-stage fresh-context complement handing over the file state. The strict completion transition rate of this restricted protocol was 0/20, but it does not represent the same-thread conversation recovery rate.
• In practice, there is a basis to start with medium when prioritizing time and tokens. The default setting for general quality must be measured again with more diverse tasks and adjusted scopes.
• Upgrade strategies using different settings for first implementation and complement (e.g., terra medium → sol high) are promising follow-up hypotheses but were not measured in this experiment.