Build your own agent harness, part 2: every turn pays for the whole conversation

Last time, we found out that the model doesn't remember us. The "conversation" is a list that our harness keeps and resends, in full, with every request. Nice trick. Elegant, even.

But wait a minute. In full? Every time? My inner accountant just woke up 🤓 If every turn resends everything said so far, then every turn also pays for everything said so far. So what does a simple "ok" actually cost?

Today we add a system prompt and a token counter, and we watch the bill grow under our eyes.

The system prompt is just a message

First, a small addition everyone has heard of: the system prompt. The text that tells the model how to behave before the user says anything.

It sounds special. It isn't. Here it is in Agent.java, a plain Java text block:

final String SYSTEM_PROMPT = """
        You are a concise assistant for software developers.
        Answer in at most three sentences unless asked otherwise.
        """;

And here is where it goes:

// The system prompt is just the first message of the history: it is resent every turn too.
List<ChatMessage> history = new ArrayList<>();
history.add(SystemMessage.from(SYSTEM_PROMPT));

That's it. A SystemMessage, added as the first item of our list. Since we resend the whole list every turn, the system prompt travels with every request too. It sets the assistant's default behavior, and the model weighs it against what the user asks. Keep that "weighs" in mind, we'll test it.

The API counts tokens, not money

Now the counting. In step 01 we only kept the message from the model's response. This time we keep the whole ChatResponse:

var response = model.chat(history);
AiMessage answer = response.aiMessage();

because it carries something precious: the token usage of the call.

// Every turn pays again for the whole history: input tokens grow even if we only say "ok".
var usage = response.tokenUsage();
totalInput += usage.inputTokenCount();
totalOutput += usage.outputTokenCount();
double cost = (totalInput * PRICE_INPUT + totalOutput * PRICE_OUTPUT) / 1_000_000;
int contextSize = usage.inputTokenCount() + usage.outputTokenCount();
display.usage(usage.inputTokenCount(), usage.outputTokenCount(), cost, 100.0 * contextSize / CONTEXT_WINDOW);

For each call, the API tells us how many tokens went in and how many came out. We add them up for the session and turn them into dollars.

Into dollars… with which prices? That's the catch: the response tells you how many tokens, never how much money. Same for the size of the context window. The harness has to know them, so they come from our configuration:

// Prices and window size are provider-specific: the API never tells us, we have to know them.
final double PRICE_INPUT = Double.parseDouble(env("LLM_PRICE_INPUT", "0.10"));   // $ per million tokens
final double PRICE_OUTPUT = Double.parseDouble(env("LLM_PRICE_OUTPUT", "0.50")); // $ per million tokens
final int CONTEXT_WINDOW = Integer.parseInt(env("LLM_CONTEXT_WINDOW", "1050000"));

The defaults match gpt-6-luna. For another model, check your provider's pricing page and set the three environment variables.

Remember, my goal in these articles is to learn and share. Sometimes my approach is naive, just so that things stay simple enough.

Last thing: contextSize. The input of the last call is the whole history, plus the answer that just came out: that's the current size of the conversation, in tokens. Divided by the window size, it tells us how full the context is.

display.usage(...) only renders all this: after each answer, a grey line shows calls · in / out tokens · session cost · context %. The counting itself stays in the step file, because it's the whole point of this article.

And that's all that changed since step 01. Small, right?

Run it

export LLM_API_KEY=...
export LLM_PRICE_INPUT=0.10 LLM_PRICE_OUTPUT=0.50 LLM_CONTEXT_WINDOW=1050000   # adjust to your model
jbang 02-tokens/Agent.java --inspect

"ok" is not free

Let's say "Hello", then "ok" a few times. The laziest conversation ever:

you> Hello

assistant> Hello! How can I help?

  1 call · 30 in / 11 out · session $0,0000 · context 0,0%

you> ok

assistant> What would you like help with?

  1 call · 48 in / 30 out · session $0,0000 · context 0,0%

you> ok

assistant> I’m here whenever you’re ready.

  1 call · 85 in / 34 out · session $0,0001 · context 0,0%

Look at the grey lines. The output stays small, the model doesn't have much to say to "ok" (who would?). But the input grows at every turn. We don't pay for "ok", we pay for the history that comes with it.

Now ask something that needs a long answer, then say "ok" again. The input of that "ok" jumps: the long answer is now part of the history, and it will be resent, and paid for, at every single turn from now on.

Let's do a bit of math (just a bit, promise). If each turn adds roughly the same amount of text to the history, turn 1 sends 1 chunk, turn 2 sends 2 chunks, turn 10 sends 10. The total input for 10 turns is 1 + 2 + … + 10 = 55 chunks, not 10. The conversation grows linearly, but its cost grows roughly with the square of the number of turns. Stateless has a price 💸

The system prompt applies to every request

Now let's check our system prompt is doing its job. Ask:

Explain how garbage collection works in Java.

That's a topic that usually fills pages (and conference talks, and books…). Here you get three sentences at most: the system prompt is part of every request, and the model follows it.

Then ask:

Explain it in detail.

And now you get a longer answer. Did the model just ignore its instructions? No: the prompt says "unless asked otherwise", and now you asked. A system prompt sets a default, not a hard limit. The model weighs it against what the user asks.

What the JSON shows

Open inspect.log and look at any request: the system message is the first item of input. In every request. The "special" system prompt is just one more message we resend, and pay for, every turn.

The context window fills up

The last number on the grey line is the context usage. With a window of more than a million tokens, our "ok" conversation barely moves the needle. But that percentage only goes one way. Every turn adds to it, nothing ever removes anything, and one day the history won't fit anymore.

What do we do then? Not today's problem. But keep an eye on that number 👀

Limits

Let's stay honest:

• The cost is an estimate: prices come from our configuration, not from the provider.
• With a reasoning model like gpt-6-luna, the output tokens include the model's hidden reasoning. You pay for thinking you never see.
• Providers may cache a repeated prefix of the request and bill it cheaper. Our counter ignores that, so it's an upper bound.
• Usage data is optional in the API. If a provider doesn't send it, tokenUsage() is null… and at the step-02 tag, the agent crashes with a NullPointerException, right after a perfectly good answer. Oops 🙈 I fixed it afterwards with a simple if (usage != null) (this commit, so pull main if you want it). The tag keeps the original code, and the lesson: a harness treats optional fields as optional.

As usual, the code, a README to run it and things to try are in the step-02 folder.

Next time

So far, our model can only talk. How does a model that only produces text read a file or check the time? Next article: a first tool, and the realization that the model never runs anything itself.

Tell me what you think 💞