A real-world experiment with Claude Code, Jira, GitLab and OpenTelemetry to measure agent costs, reduce token waste and find out whether AI agents are actually cheaper than human developers.

Why do we celebrate the empty gas tank?

My tax advisor asked me a fairly reasonable question:

“Could I use agents to automate some of the boring work?”

It is the kind of question I hear more often now. Not from people looking to replace an entire department, but from professionals who spend too much time moving information between systems, checking documents, preparing standard responses and repeating processes that are necessary without being particularly intelligent.

My instinctive answer was yes

Much of that work can probably be automated. At least parts of it. An agent can collect information, apply rules, prepare a result and hand it to a human for review.

The more important question is whether doing so actually makes economic sense.

Later that same day, I saw someone in a chat group enthusiastically announcing that he was constantly running out of tokens.

Apparently, this was evidence that he was doing serious work with AI and I have to confess, I found that logic slightly confusing.

Formula One teams do not celebrate the first driver who runs out of fuel. Cloud architects rarely brag that they exhausted the budget before lunch. Even in consulting, consuming every available resource is generally not considered a performance indicator unless the contract is unusually generous.

So I decided to test both ideas.

Could an agent complete a real piece of work more cheaply than a human?

And could I build the workflow in a way that did not treat token exhaustion as an unavoidable side effect or even worse as a indicator of success of agentic systems?

For the experiment, I used something I understand well enough to measure properly: software development.

Not a synthetic coding benchmark. Not “build me a calculator.” A real Jira ticket in a real application, with several repositories, tests, branches, GitLab Merge Requests and a human review at the end.

That experiment worked, but it also processed almost 19 million tokens for the first successful ticket.

Which made the second question considerably more urgent.

---

What would it cost a human?

Before building anything, I needed a reasonable comparison.

A developer does not receive a Jira ticket and immediately begin typing the final solution. The developer reads the description, explores the repository, reconstructs the relevant architecture, identifies the likely cause, changes the code, runs tests, handles Git, creates a Merge Request and documents the result.

Depending on the ticket, that might take 30 minutes, half a day or considerably longer.

The financial comparison is not simply:

AI subscription versus developer salary

The more useful comparison is:

cost of investigation
+ cost of implementation
+ cost of verification
+ cost of review
+ cost of fixing mistakes

An agent can only be cheaper if it reduces the total amount of human attention required without creating additional risk somewhere else.

That meant my experiment needed more than generated code. It needed a controlled development process.

---

The agent needed to work like part of the team

My application, Modulyn, is spread across several independent GitLab repositories:

• modulyn-core
• modulyn-api
• ib-to-zoho
• infrastructure
• several domain modules

A ticket may affect one repository, or it may reveal a genuine dependency across several.

I did not want Claude to search the entire workspace, change anything that looked related and deliver one enormous commit accompanied by a cheerful summary.

The workflow had to resemble the way I would expect a human developer to work:

1. Read the ticket in Jira.
2. Investigate the relevant repository.
3. Explain the likely cause.
4. Propose an implementation plan.
5. Wait for my approval.
6. Create a feature branch.
7. Implement only the approved changes.
8. Run the relevant verification.
9. Commit and push the result.
10. Open a draft GitLab Merge Request.
11. Document the outcome in Jira.
12. Leave the final review and merge to me.

From my side, the process should begin with one command:

.\ticket.cmd MODULYN-5

This is close enough to assigning a ticket to an AI developer to be useful and comparable in my little experiment.

The important difference is that the AI developer does not decide what reaches production.

---

Jira tells the agent where to begin

Because Modulyn is not a monorepo, repository discovery quickly became one of the main cost drivers.

If Claude searches every repository for every ticket, it spends time and tokens reading architecture, configuration and implementation details that may have nothing to do with the issue.

I therefore added a Jira field called Suggested Component. Its values correspond to the repositories in the workspace.

Figure 1: Jira gives the agent a strong routing hint without giving it unlimited authority.

A value such as:

Suggested Component = modulyn-core

means that Claude should start its investigation there.

It does not mean the repository can be modified automatically. The implementation plan still needs approval. It also does not prevent Claude from identifying a real dependency elsewhere.

The field narrows discovery and the approved plan authorizes the work.

That distinction is important because efficiency should not come from making the agent blind. It should come from making exploration intentional.

---

The workflow has a planning phase and an implementation phase

The process lives in a Claude Code skill called jira-deliver and the first phase is read-only.

Claude reads the Jira ticket through Atlassian MCP, resolves the suggested repository and investigates the likely cause. It identifies the affected files, relevant tests and possible deployment implications.

It then produces a self-contained plan and stops.

At this point, Claude is not allowed to create a branch, edit files, update the Jira status or push anything.

I review the plan and either adjust it or approve it.

Only then does the second phase begin.

Claude checks for unrelated local changes, fetches the latest default branch, creates a Jira-keyed branch and implements the approved work. It runs the repository-specific verification, reviews its own diff, commits the change, pushes the branch and creates a draft Merge Request.

Finally, it writes the result back to Jira and updates a ticket. Something I have to admit, is way more effort a human developer most likely will take and this produce a great papertrail of changes the human still benefits of.

This separation matters for governance, but it also became one of the most important levers for controlling token usage.

I simply did not understand that yet.

---

The first ticket worked

The first substantial ticket concerned Zoho (a bookkeeping tool) connectivity.

Modulyn could display the integration as connected even when configuration or refresh-token issues meant that the user actually needed to disconnect and reconnect the account.

Claude investigated the issue, produced a plan, implemented the change, verified the result and opened the Merge Request.

From a functional perspective, the experiment was successful.

The agent had taken a real Jira ticket through a real engineering process. It had performed work that would otherwise have required a developer to investigate the behaviour, locate the relevant code, make the changes and prepare them for review.

This was the answer to my tax advisor’s first question: Yes, agents can automate a meaningful part of professional work.

Then I opened Grafana.

---

The agent had processed almost 19 million tokens

I had connected Claude Code to an observability stack:

Claude Code
    → OpenTelemetry Collector
    → Prometheus
    → Grafana

Each ticket run received labels such as:

workspace=modulyn
workflow=jira-deliver
jira_issue=MODULYN-5

This allowed me to measure tokens, estimated cost, cache usage, model activity, agent time, commits, edit activity and Merge Request events per ticket.

The first complete run showed:

• 18.96 million processed tokens
• $9.96 in estimated API-equivalent cost
• approximately 21 minutes of Claude active time

Figure 2: The workflow delivered the ticket, but the resource consumption was difficult to defend.

Ten dollars is not expensive compared with an hour of experienced engineering work, so that part of the experiment was promising.

However, almost 19 million tokens for a single ticket did not look like a system I wanted to scale carelessly.

It also explained why people could run out of token allowances so quickly. The agent did not need to generate millions of words. It only needed to carry a large context through enough model turns.

---

A 98 percent cache hit rate did not make the workflow efficient

The dashboard showed a cache reuse rate of approximately 97.7 percent and initially, that looked healthy. Claude was not rebuilding the entire context from scratch during every request, but cache reuse is not the same as zero cost.

The workflow repeatedly carried a large context through planning, implementation, verification and Git operations:

large context
→ model request
→ tool call
→ model request
→ file read
→ model request
→ test output
→ model request
→ Git operation
→ model request

A context of roughly 100,000 tokens processed across dozens of requests quickly becomes several million processed tokens and yes, the cache made that cheaper than reconstructing the context each time. It did not make the context small.

This is where the Formula One comparison became useful. A team may recover energy efficiently and still lose the race because the car is carrying unnecessary weight.

The token cache was doing its job. The workflow architecture was not.

---

The large instruction file was not the main problem

My first suspect was the workspace-level CLAUDE.md.

It contains shared instructions for several repositories, architecture conventions and rules for working inside Modulyn. Perhaps I had simply given the agent too much documentation.

Claude Code’s /context all view showed approximately:

• 118,800 tokens in messages;
• 21,800 tokens in system tools;
• 9,100 tokens in the system prompt;
• 2,400 tokens in memory files;
• 2,300 tokens in loaded MCP tools;
• 1,900 tokens in skills.

Figure 3: Accumulated investigation history was much larger than the workspace instructions.

The workspace instructions contributed around 2,000 tokens. They were not free, but they were not responsible for a 19-million-token run. So it became very clear, the real weight came from the accumulated conversation.

Claude had searched files, evaluated possible causes, followed dead ends, read tool output and discussed the implementation plan. After I approved that plan, the entire investigation remained attached while Claude edited the code, ran tests, committed the result and interacted with GitLab.

The context window still had plenty of capacity. Nothing forced Claude to compact or discard the history automatically.

The agent was therefore paying to reread its complete research notebook while performing tasks that only required the final conclusion.

---

change the workflow instead of buying more tokens

This is maybe something the AI companies don’t like to hear, but the easy solution would have been to increase the token allowance. But this is like having a broken toe and just take a aspirin. That would have solved the symptom while preserving the waste.

Instead, I changed the workflow.

Medium effort became the normal setting

The first version used high reasoning effort for every ticket. That may be appropriate for an architectural change involving several systems. It is excessive for an ordinary bug fix or a very small feature with limited impact.

So I decided to switched the default to medium effort. This should be enough for most of the tasks and we can increase it any time if there is a heavy lift and shift operatation planned.

The next real ticket, which added a missing logout option, consumed approximately:

• 9.11 million tokens
• $4.44 in estimated cost

That was roughly half the consumption of the earlier run. The tickets were different, so this was not a controlled comparison. Still, the result was strong enough to keep medium effort as the default.

The highest reasoning setting should be selected because the task requires it, not because it exists.

Repository routing became stricter

When Jira identifies one valid component, Claude now begins only in that repository. The skill instructs the agent to use targeted searches, avoid broad architecture surveys and refrain from enumerating sibling repositories without evidence.

Claude can still expand the scope when the code points to a genuine dependency, but ut must explain why.

This reduced the amount of irrelevant context created before implementation even began.

The plan became a real implementation handoff

The plan now contains enough information for implementation to start without replaying the whole investigation:

• exact repositories
• likely files
• root-cause evidence
• implementation steps
• acceptance criteria
• build and test commands
• safety implications
• deployment considerations
• branch and Merge Request requirements.

A useful plan is not a summary of what Claude has been thinking. It is more the contract for the next phase the LLM and the user agreed on.

Approval now clears the planning history

This was the most important change.

Once I approve the plan, Claude clears the exploratory context and starts implementation with the ticket and the approved plan. And this is implicitly stated in the Claude documantion but completly underestimated. The standard context of a Sonnet 5 model can carry up to 1m token, so my 118k token were far below the automatic context compaction.

The workflow now looks like this:

investigation context
→ approved self-contained plan
→ clean implementation context

The implementation phase no longer carries every search result, dead end, file read and tool response from planning.

This saves tokens, but it also makes the agent more predictable. Claude begins implementation from an approved scope instead of a large collection of observations that might invite additional work.

Duplicate Git instructions were removed

The skill already defined how branches, commits, pushes and Merge Requests should be handled.

Claude Code was also loading additional built-in Git guidance.

I disabled the duplicate instructions for ticket sessions. This was not the largest saving, but unnecessary context remains unnecessary even when it is well written.

---

The next run used 3.5 million tokens

The next ticket was a small but real issue in Modulyn’s About page. The page displayed the versions of all modules correctly, while modulyn-core itself appeared as “Unknown.”

Jira suggested modulyn-core.

Claude started there, found that Spring Boot’s build-info goal was not configured and proposed the required Maven change.

I approved the plan. The planning context was cleared. Claude made the change, verified that build-info.properties was generated, pushed the branch and opened a draft Merge Request.

The run consumed:

• 3.50 million processed tokens
• $2.13 in estimated API-equivalent cost
• approximately 12 minutes of Claude active time
• one commit
• one actual GitLab Merge Request
• one file changed
• seven final lines added

Figure 4: The optimized workflow used substantially fewer tokens while still completing the full delivery process.

Compared with the earlier 18.96-million-token ticket, this represented:

• 81.5 percent fewer processed tokens
• 78.6 percent lower estimated cost

Again, for the sake of transparancy and clarity, the tickets were not identical, so I would not call this a laboratory benchmark. It was, however, enough to prove the principle.

Running out of tokens was not an unavoidable property of agentic work. A significant part of the consumption came from workflow design.

---

Was the agent cheaper than a human?

For this particular ticket, the estimated API-equivalent cost was $2.13.

Even after adding my time to review the plan and Merge Request, the agent was cheaper than asking a developer to investigate the repository, identify the missing build configuration, implement the change, verify it and prepare the handoff.

That does not mean agents are automatically cheaper and there are more and more companies finding this out the hard way.

And even if they are cheaper, a low token bill does not compensate for incorrect code, weak testing or hours of human cleanup. The relevant measure is not the cost of generation. It is the cost of reaching a reviewable and correct outcome.

In this case, the agent delivered:

• the investigation;
• a proposed solution;
• the implementation;
• verification;
• a feature branch;
• a commit;
• a draft Merge Request;
• a Jira handoff.

The human remained responsible for approval and review.

That is a useful economic model. The agent performs the structured work. The human spends attention at the decision points.

The saving comes from reducing human execution time, not from pretending human judgement is obsolete.

---

The dashboard also taught me not to trust every AI metric

During the optimized run, Claude reported 160 lines added.

GitLab showed the final result:

• one file changed;
• seven lines added;
• zero lines removed.

Both figures were correct. They described different things.

Claude’s line counter included intermediate edits, replacements and rewrites. GitLab showed the final delivered diff.

The same problem appeared elsewhere:

Claude edit activity ≠ final delivered diff
MR creation events ≠ unique Merge Requests
processed tokens ≠ unique context
estimated API cost ≠ necessarily the billed subscription cost

During an earlier run, telemetry recorded two Merge Request creation events while GitLab contained one actual MR. A timeout or ambiguous response had affected the operational counter.

This is why an AI dashboard should not be allowed to grade its own homework.

Claude can report internal activity. GitLab remains authoritative for commits, diffs, Merge Requests and pipeline results. Jira remains authoritative for the requested work and acceptance criteria.

The useful picture comes from connecting those systems.

---

The agent leaves evidence behind

After implementation, Claude writes a concise handoff into Jira.

It includes:

• what changed;
• what was verified;
• what was not verified;
• the branch name;
• the draft Merge Request link;
• the remaining risk.

Figure 6: The result is documented where the work began.

This matters because an agent should not merely produce code and disappear.

A reviewer needs to understand why the change exists, which acceptance criteria it addresses, what was tested and what remains uncertain.

That evidence also makes the economics measurable. I can compare the cost of the run with the result that reached GitLab, rather than counting generated text and calling it productivity.

---

What I would now tell my tax advisor

Yes, agents can automate boring professional work and they are particularly useful when the process involves collecting information, following repeatable rules, interacting with several systems and preparing a result for human review.

But the business case depends on the workflow around the model.

An agent with unlimited context, unclear scope and no measurement can turn a simple process into a very sophisticated waste generator.

The useful design principles were not especially glamorous:

• route the agent to the most likely source;
• separate investigation from execution;
• require approval before changes;
• compress the investigation into a self-contained handoff;
• clear context when it is no longer needed;
• use expensive reasoning selectively;
• verify outcomes in the systems where they actually exist;
• keep the human at the decision points.

This applies beyond software development.

A tax agent reviewing documents should not carry every earlier client file into every later action. A support agent should not search the entire knowledge base when the product and issue category are already known. A finance agent should not rerun a full investigation after a human has approved the proposed treatment.

Context is useful. Uncontrolled context is expensive.

---

The real experiment was not whether Claude could write code

I already expected Claude to be capable of changing a Maven configuration, so the interesting question was whether an agent could take part in a real workflow at a cost that made sense.

The answer is yes, provided the workflow is designed for efficiency and reviewability.

My process now looks like this:

Jira ticket
→ targeted investigation
→ implementation plan
→ human approval
→ planning context cleared
→ feature branch
→ implementation
→ verification
→ commit and push
→ draft GitLab Merge Request
→ Jira handoff

It is not an autonomous software company.

It is a controlled system that transfers structured work from a human to an agent while keeping judgement and responsibility with the human.

That is probably also the answer for my tax advisor.

Do not begin by asking whether an agent can do everything.

Begin with the boring process, define the boundaries, measure the result and make sure the machine is not celebrating because it ran out of fuel.