CI Got Slow, So I Decided to Buy Back My Time

AI-generated conceptual illustration.

I started out trying to cut cloud costs, but now I find myself leaning toward a slightly more expensive computer. The reason is Myven’s CI: the automated tests that check whether things still work whenever I change the code. Lately, waiting for those results has been taking a while.

More features mean more tests. Having more tests is a good thing, but it also stretches the gap between making a change and finding out whether it works. Waiting once is fine. Repeating that wait several times a day feels different.

GitHub Actions was getting expensive, so I initially installed a runner on a computer I already owned. A runner is the program that actually executes CI jobs. Using existing hardware saved money, but as the tests grew longer, time became the bottleneck.

So I tried running the same job on AWS instances.

The cost of finishing a job matters more than the hourly rate

An always-on server is usually sized with some headroom for peak demand. It needs to handle sudden increases in traffic, which means some capacity sits unused during quieter periods.

For a compute-heavy job that runs and then finishes, like CI, the calculation is a little different. If the instance runs only as long as needed and then shuts down, a higher hourly rate can still work out cheaper if the job finishes sooner.

Here are the recorded times for a subset of Myven’s tests.

These times include environment setup and cleanup for this test job. They do not represent Myven’s entire CI pipeline or a general measure of instance performance.

The same job finished 1 minute and 46 seconds sooner. That is about a 24% reduction in elapsed time, or roughly a 32% increase in the rate of completing the same work.

The interesting part was the cost. In the pricing I compared, C8g.xlarge cost about 10% more per hour than C7g.xlarge. Applying that difference gives a simple calculation:

Cost per run ≈ hourly rate × elapsed time

With the C7g run cost set to 100:
C8g run cost = 100 × 1.10 × (330 sec ÷ 436 sec)
             ≈ 83

Despite the higher hourly rate, the compute cost per run comes out about 17% lower. Finishing sooner means spending less time paying that higher rate.

This comparison assumes a 10% difference in hourly rates; it is not a statement of current AWS pricing. The actual bill also depends on the full time the instance stays running and additional costs such as storage. Still, this job made it clear that choosing by hourly rate alone could cost me more.

Serial and parallel execution need different thinking

There is a distinction to make here. Making a computer process work faster and having it process multiple jobs at once are two different approaches.

Serial execution means running tests one after another. The next test starts when the previous one finishes. If there is only one stream of test execution, adding CPU cores may leave much of that extra capacity unused. For the compute-bound parts of that serial work, better performance per core is what helps.

Parallel execution means running multiple independent tests at the same time. More cores then provide room to distribute the work. But the test tooling also needs to be configured to run more jobs concurrently. Moving to a larger instance while continuing to run tests one at a time can mean paying for more resources without saving much time.

The C7g.xlarge and C8g.xlarge comparison above did not come from increasing the number of parallel tests. It compared the same job across two instance generations. That result alone does not tell us whether adding more cores would produce a similar speedup.

If the work can run in parallel, a larger instance becomes an option

That led to the next question: if the tests can be split into independent groups, what happens when I increase both the instance size and the number of parallel jobs?

If an instance with twice the hourly rate finishes the job in half the time, the calculated cost per run stays the same.

2× hourly rate × ½ the execution time = the same cost per run

That would mean getting the result twice as fast for the same money. For CI, that is an appealing trade.

But this is the ideal case where the entire run takes half as long. Real CI mixes setup steps that must happen in sequence with tests that can run concurrently.

Suppose a 10-minute job includes 2 minutes that must run serially and 8 minutes that can be parallelized. Even if the parallel portion runs twice as fast, the total is still 2 min + 4 min = 6 min. At twice the hourly rate, the cost per run becomes 2 × 6 ÷ 10 = 1.2× the original. In this case, the choice is to spend 20% more to cut the wait by 40%, rather than halve the time at the same cost.

Even that assumes evenly distributed work and no additional overhead. In practice, one slow test might keep running after everything else has finished, or tests sharing a database or test data might interfere with each other. Before increasing concurrency, I need to isolate test data and check where the job spends time waiting beyond the CPU.

So the C7g versus C8g comparison is based on recorded runs. The effect of doubling the instance size is something I still need to test. I need to make parallel execution possible and measure how much time it actually saves.

What I wanted to save was time, after all

At first, I wanted to spend a little less on CI. But as I cut costs, the wait for test results grew longer. That waiting was not free either.

If I start something else while waiting, it takes time to get back into the original task. Even when I want to check one more small change, the thought of another long CI run makes me hesitate. Those costs never appear on the compute bill, but they are easy to feel while developing.

In this comparison, the faster instance reduced both elapsed time and the cost per run. I will need more measurements to see whether the next step does the same. But finding the computer with the lowest hourly rate no longer feels like a sufficient goal.

I wanted to save a little more money, but I do not want to keep losing time waiting for CI. I think it is time to pay to get some of that time back. And if buying back time also makes the bill smaller, as it did here, all the better.