My AI Agent Worked. My Tests Didn't.

A few months ago I shipped the first version of the AI assistant in AWS Announcements Hub. The site has a chatbot, and behind that chatbot sits an AI agent. That agent runs with Strands on Amazon Bedrock AgentCore and uses Amazon Nova through Amazon Bedrock. Its job is narrow: search more than 20,000 AWS announcements and answer questions about how AWS services have evolved over time.

This series walks the full cycle of taking an agent to production: test before deploying → observe what it does in production → operate and improve with what you see → secure the boundary. This is part 1, test.

Getting the first version working was surprisingly quick. Getting to the point where I could change it without wondering what else I had just broken was a different story.

This post is not a list of best practices. It is what I ran into along the way: three silly failures my tests never caught, what they forced me to change about how I test the agent, and the workflow I ended up with. If you have an agent in production, you will probably recognise the starting point.

The day my green tests weren’t testing what mattered

I already had unit tests, and they all passed. The Python functions behind the agent were covered: the filters worked, the data transformations worked, and search returned exactly what I expected. Green from top to bottom.

And writing those tests has never been cheaper: today you ask the AI to generate the whole battery of unit tests for a function and you have it in seconds, practically for free. So the problem was never having tests (I had plenty). The problem was that I was testing what was easy to test, not what actually broke.

Having lots of green tests isn’t the same as testing what matters. AI generates unit tests in seconds; deciding which behaviour deserves a test is still on you.

The problem showed up the moment I stopped asking the agent textbook questions and started treating it like a real user would. Since the assistant answers in several languages, I deliberately tried the same intent in English and in Spanish, and that is where the illusion fell apart.

The first thing I asked, in English, was for the oldest announcement in the database:

Me: “What is the first announcement?”

Agent: “I’m specialized in AWS announcements and services only.”

That is a perfectly valid question about AWS announcements, and the agent flatly refused it. I assumed it was bad luck with the wording, so I asked for exactly the same thing in Spanish:

Me: “¿cuál fue el primer anuncio?” (what was the first announcement?)

Agent: [returns a list of announcements, newest to oldest]

This time it did search. But I had asked for the first announcement and it handed me, in practice, the latest one. Same intent, different language, different result, and both wrong.

The third one worried me the most. I asked about a specific year:

Me: “Are there announcements from 2004?”

Agent: [replies with a plausible list, without touching the database]

It made up a believable answer from the model’s own knowledge, without checking the real data. If I hadn’t known the dataset myself, it would have slipped right past me.

Three questions, three failures, and the telling part is that each one lived in a different layer of the system: the first was an undue refusal of a valid query, the second a parameter the model picked wrong, and the third simply not using the tool when it should have. None of my unit tests noticed any of it. That was when it clicked that I was testing the tools, but not testing the agent.

The trap is right here: testing the functions behind the agent isn’t the same as testing the agent. The model decides whether to call them, with which parameters, and how it reads the result, and no unit test touches that layer.

Why unit tests weren’t enough (and why they stayed)

Unit tests missing these failures doesn’t mean they are useless. I replaced nothing: a tool is ordinary software, and deterministic software gets tested deterministically. A test like this one is still fast, cheap and worth having to catch a broken filter or a bad transformation:

def test_search_returns_s3_results():
    results = search_announcements(service="S3", year=2025)

    assert results["count"] > 0
    assert all(
        r["service"] == "S3"
        for r in results["results"]
    )

The catch is that above that layer sits another one the unit test never touches: the model decides whether to call the function at all, with which arguments, how to read the result and how to turn it into an answer. A flawless search_announcements() is worth nothing if the agent decides not to call it. That decision is exactly where my three questions fell apart.

So I added a second layer of tests. And here is the distinction that took me a while to see: these are no longer normal tests. A unit test calls your function and checks what it returns. This layer instead boots the real agent (with its model, its system prompt and its real tools), throws a natural-language question at it and watches what it does. It doesn’t test a function; it tests the model’s decision.

I built it in three pieces. The first is describing each case as data, not as test code. I use a dataclass for that:

from dataclasses import dataclass, field

@dataclass
class EvalCase:
    id: str
    query: str                              # what I ask the agent
    tool_expected: str | None               # the tool it MUST call (or None)
    must_contain: list[str] = field(default_factory=list)      # substrings required in the answer
    must_not_contain: list[str] = field(default_factory=list)  # substrings forbidden

An EvalCase is not a test: it is the description of a behaviour I care about. For example, “if I ask for the first announcement, the agent must call search_announcements and must NOT give me the refusal message”:

EvalCase(
    id="search-first-announcement-en",
    query="what is the first announcement?",
    tool_expected="search_announcements",
    must_not_contain=["I'm specialized in AWS"],   # must not refuse
    must_contain=["/announcement/"],               # must return real links
)

EvalCase(
    id="scope-weather",
    query="what's the weather in Madrid?",
    tool_expected=None,                            # out of scope: must NOT search
    must_contain=["I'm specialized in AWS"],       # here the refusal IS correct
)

The second piece is the one that really separates this from a unit test: a function that runs the real agent and captures, besides the text answer, which tools it called and with which parameters. Simplified:

def run_agent_locally(query: str) -> dict:
    agent = Agent(model=model, system_prompt=SYSTEM_PROMPT, tools=tools)

    response, tools_used, tool_calls = "", [], []
    async for event in agent.stream_async(query):
        if "data" in event:                        # text the agent streams out
            response += event["data"]
        if "current_tool_use" in event:            # each time it decides to use a tool
            call = event["current_tool_use"]
            tools_used.append(call["name"])
            tool_calls.append({"name": call["name"], "params": call["input"]})

    return {"response": response, "tools_used": tools_used, "tool_calls": tool_calls}

Notice there isn’t a single assert here: I just run the agent and keep what it did. The tool_calls (the tool name and the exact parameters the model chose) is precisely what a unit test never sees, and what let me catch the reversed-ordering bug.

The third piece is the check. With what the agent did in hand, a plain assert is enough for most cases:

def score_case(case: EvalCase, result: dict) -> tuple[bool, list[str]]:
    failures = []
    if case.tool_expected and case.tool_expected not in result["tools_used"]:
        failures.append(f"expected tool '{case.tool_expected}', got {result['tools_used']}")
    for s in case.must_contain:
        if s not in result["response"]:
            failures.append(f"missing from answer: {s!r}")
    for s in case.must_not_contain:
        if s in result["response"]:
            failures.append(f"should not appear: {s!r}")
    return (len(failures) == 0, failures)

And I wire it all to pytest so each case is an independent test and the whole suite runs with the same pytest as always:

import pytest

@pytest.mark.parametrize("case", EVAL_CASES, ids=[c.id for c in EVAL_CASES])
def test_eval_case(case):
    result = run_agent_locally(case.query)
    passed, failures = score_case(case, result)
    assert passed, f"Eval '{case.id}' failed:\n" + "\n".join(failures)

The upshot is that pytest tests/test_behavioral.py runs the dozens of questions against the real agent and tells me, case by case, which ones it passes and which it doesn’t. There is no second LLM evaluating this: the real agent runs and normal code checks the result. For each case I can ask whether it called a tool, whether it was the right one, which parameters it used, whether it refused something it shouldn’t have, or whether the answer contains the kind of link I expect. The exact wording almost never matters, and that is a good thing, because a model can give the same correct answer in ten different ways.

But the second bug, the reversed ordering in Spanish, taught me that watching which tool runs isn’t enough. Take “What was the first AWS announcement?”: both of these calls use the correct tool, and only one answers the question.

search_announcements(sort_order="asc")   # correct
search_announcements(sort_order="desc")  # same tool, wrong question

So I split out a third kind of check focused only on the arguments: that sort_order="asc" is used for first or oldest requests, or that all_history=True shows up when a question needs the full history instead of the default window. It is a class of failure that is easy to miss by eye: the trace shows the correct tool, everything looks reasonable, but the model is asking the tool the wrong question.

When code can no longer judge

Deterministic checks get me surprisingly far, but they hit a ceiling: they don’t tell me whether the final answer is good. Imagine the tool returns ten Amazon EC2 announcements and the agent shows only two without saying why, invents a title, answers in English to a Spanish question, rewrites the URLs, or describes the oldest result as if it were the newest. Writing an assert for each of those gets absurd fast.

For that handful of cases I use an LLM as an evaluator. The judge receives the original query, the data the tools returned, the agent’s final answer and a single concrete criterion to evaluate:

Query:
"¿cuáles son los últimos anuncios de EC2?"

Tool output:
[10 EC2 announcements]

Agent response:
"Here are the latest EC2 announcements..."

Evaluation:
Is the response written in the same language as the user query?

I run these evaluations with Amazon Nova through Bedrock, but I use them sparingly. If normal code can tell me whether sort_order is correct, normal code wins every time: faster, cheaper, deterministic. I keep the LLM judge for when the property I actually care about requires understanding the answer. And I don’t treat it as ground truth either: it is just another model, so the evaluation prompt matters and its verdicts need the occasional sanity check.

With that, my test stack ended up split across four layers that don’t compete with each other, each guarding a different boundary of the system:

Layer	What I check	Typical failure it catches
Unit tests	Deterministic tool code	Broken filter or transformation
Behavioral evals	Tool choice, refusals, response contract	Agent refuses a valid query
Parameter evals	Arguments the model picks	sort_order="desc" instead of "asc"
Quality evals	Semantic quality of the final answer	Wrong language or unfaithful summary

From bug to test: eval-driven development

With the layers clear, I went back to the three failures from the start, and here is what shifted my thinking the most: none of the three was the same kind of problem, even though all three looked like “prompt stuff”.

The refusal of a valid query really was a prompt issue: the system prompt had examples glued to specific services and didn’t make this broader type of question clear. I fixed it by changing the instructions and examples. The reversed ordering in Spanish, on the other hand, wasn’t a language problem: I had Python code trying to guess sort direction from English words (first, latest), and primero simply had no case. The fix wasn’t to patch the dictionary but to push that decision up to the model (I added sort_order to the tool interface) so it could map the user’s intent to an explicit value. That was a design problem at the boundary between the model and the tool. And the made-up 2004 answer was something else again: the model knew enough AWS to improvise something plausible, so it did; I solved it by making it explicit that questions about announcement data must be grounded in the database. A behavioural constraint.

None of them required changing the foundation model, and none was “prompt engineering” plain and simple. Agent behaviour is spread across the prompt, the tool schemas, the deterministic code and the model itself, and the point of the eval suite is that it tells me which boundary is the broken one.

That changed my workflow entirely. Before, touching the prompt looked like this:

flowchart LR
    A[Change prompt] --> B[Ask 4 questions by hand]
    B --> C["Seems fine"]
    C --> D[Deploy]

And the gap is always the question you didn’t think to repeat: you fix one behaviour and quietly break another. Now the loop is different:

flowchart LR
    A[Find a failure] --> B[Reproduce it]
    B --> C[Add it to the suite]
    C --> D[Fix prompt, tool or code]
    D --> E[Run the suite]
    E --> F[Deploy]

This is commonly called eval-driven development. It is like TDD, except the thing you’re testing is partly probabilistic. The goal isn’t to force an LLM to behave like a deterministic function; it is to stop evaluating every prompt change from memory and by eye.

A fair question here is: what about Amazon Bedrock AgentCore Evaluations?

AgentCore now has a managed evaluations capability, with built-in and custom evaluators, that overlaps with part of what I built. I still see value in keeping my small suite inside the repo: many of my most useful cases are deliberately boring assertions (“this query must produce sort_order='asc'”), cheap, easy to understand, running in my normal deployment flow and coming straight from bugs I’ve already hit. For a larger system, or if I wanted continuous managed evaluation over production traces, AgentCore Evaluations becomes much more interesting. I don’t see them as mutually exclusive.

Where to start (and what I still don’t cover)

Today, before each deploy, the agent runs about 30 behavioral cases (tool selection, scope, multilingual), 11 parameter cases (mostly sort_order and all_history) and 8 quality cases (language consistency, faithfulness to the data, link formatting). That’s 49 in total, and it’s on purpose: I would rather have 49 cases that came from behaviours I genuinely care about than hundreds of generated test prompts I’ve never looked at. The suite is still small enough that I understand every case in it.

If you have an agent and no evals yet, I wouldn’t start by picking a framework. I’d start with the behaviours you care about. Write down five queries the agent must get right, two it should reject, which tool you expect for each (if any) and one observable property that tells you whether it got it right. Run those seven against the real agent and, when the next bug appears, add the query that exposed it. That’s already a regression suite. You add the semantic evaluators when the deterministic asserts start falling short.

To get started: five queries the agent must get right, two it should reject, the tool you expect for each, and one observable property that tells you whether it got it right. Run them against the real agent and grow the suite with every new bug.

There is a gap this doesn’t close, and it’s worth saying plainly: my suite can only contain queries that, at some point, occurred to me. Real users are far better at finding the ones I never imagined. The other half of the problem is seeing exactly what the agent did in production (its model calls, tool calls and parameters) and turning those real failures into new eval cases.

That is exactly what I cover in Part 2, My Agent Is in Production. What Is It Doing in There?: setting up observability and closing the loop by turning production traces into eval cases. Part 3, I Have Traces From My Agent. Now What?, covers what I do with those traces to improve the agent day to day. And Part 4, My Agent Is Public. What If Someone Tries to Manipulate It?, adds the safety boundary around the agent.

---

The agent I talk about in this post, the AWS Announcements Hub AI assistant with its tools and its eval suite, is the one you can try at news.playingaws.com.