I passed the AWS Generative AI certification and used it to review my own applications
AWS Certifications
Practical guides for passing AWS certifications: how to prepare, study strategies, real challenges, and lessons from renewing a specialty cert.
4 articles
I passed the AWS Certified Generative AI Developer - Professional (AIP-C01, official page). A good part of what the exam covers I had already worked through on this blog over the last few months, and those articles were my revision material:
- The basics of Amazon Bedrock and its models, in Amazon Bedrock 101.
- Prompt design and prompt management, in Prompts I and Prompts II.
- The whole lifecycle of an agent in production, in the four-part series: testing, observing, operating with traces and securing it with guardrails.
While preparing for the exam I took the chance to do something I had been putting off: review how I use AI in my personal projects, with the exam guide next to me. This article is not a study guide: it is the list of what I reviewed, in case you want to do the same with your own applications, followed by the detail of what I changed.
The applications I reviewed
Alongside the blog I have been building several applications of my own, all listed in my projects. Five of them use Bedrock, always with Amazon Nova models, and almost always for small, well-scoped tasks: moderating, classifying, scoring or drafting. Only one has a real agent. This is how they looked before the review and how they look now:
| Application | What it uses AI for | Before | Now |
|---|---|---|---|
| Announcements Hub for AWS (chat) | Public assistant about AWS announcements | Nova 2 Lite, ConverseStream via Strands | Same, with its own cost profile |
| Announcements Hub for AWS (backend) | Moderating comments, categorizing the RSS feed, email digest | Nova Micro, InvokeModel | Nova Micro, Converse |
| Read Worthy | Extracting, scoring and explaining the value of each document | Nova 2 Lite, Converse | Same, with its own cost profile |
| Community Organizer Hub | Drafting event communications | Nova Lite, InvokeModel | Nova 2 Lite, Converse |
| Cloud Confessions | Moderating submissions: approve, reject or human review | Nova Lite, InvokeModel | Nova 2 Lite, Converse |
| Home & Property Manager | Recommendations for renovations and incidents, with a photo | Nova Lite, Converse | Nova 2 Lite, Converse |
None of them was wrong on its own. The problems showed up when I looked at them together.
What I reviewed, in case you want to review the same
These are the questions I asked about each application, and the ones I recommend asking yourself if you have something running on Bedrock:
| What to review | What I found in my apps | What I did |
|---|---|---|
| Which API you use to call the model | Three apps on InvokeModel and three on Converse, one of them with a code branch per provider | All on Converse, with streaming only where it helps |
| What happens when the model fails or throttles you | Retries in only two places, and one case where comments were published unmoderated | A shared wrapper with backoff and jitter, and a failure policy per type of task |
| Whether you know what each application costs | All my AWS spend shows up per project except Bedrock | An application inference profile tagged per application |
| How you know a prompt still works | Automated evaluations only in the agent, and prompt versions set by hand | Repeated runs, a prompt hash in every log and fixed test cases where mistakes matter |
| Whether you need Guardrails | The agent read the guardrail ID, but nothing ever passed it in | I measured it, hardened the prompt and left the guardrail ready to switch on |
The rest of the article is the detail behind each row.
One API, and streaming only where it is needed
The first thing was to clarify what exactly Converse does and how it differs from InvokeModel, because I was not sure. This is what the documentation says:
InvokeModelandConverseare both synchronous: they wait for the complete response.- Streaming lives in two other operations:
InvokeModelWithResponseStreamandConverseStream. - What sets
Converseapart is the format. It uses the same structure for messages,system,inferenceConfigandusagefor every model that supports it. WithInvokeModelthe body is each provider’s native one, which is why one of my clients had one branch for Nova and another for Claude.
With Converse, switching models means changing an ID, and token usage comes back in the same field in every application. I migrated the six calls still on InvokeModel and moved three applications from Nova Lite to Nova 2 Lite along the way.
My recommendation for everyone: if you still call
InvokeModel, migrate toConverse. It is a synchronous call just like the old one, but switching models no longer forces you to rewrite the request body and the response parsing. Staying onInvokeModelonly makes sense if your model is not in the list of models supported byConverse, or if you need a provider-specific parameter (for that,ConverseacceptsadditionalModelRequestFields).
The second question was where I need streaming. It only helps when someone is watching the screen while free text is being generated:
| Case | Streaming? | Why |
|---|---|---|
| Announcements Hub chat | Yes | It is conversational and the answer is rendered as it arrives |
| Moderating comments and submissions | No | I need the full verdict before publishing |
| Categorization, email digest and Read Worthy analysis | No | They run in the background and return JSON that has to be read in full |
| Drafts and recommendations | No | The user waits a few seconds, and streaming behind API Gateway requires Lambda response streaming |
In the chat I do use it, and it was a conscious decision: at the time I replaced the synchronous call with a streaming one to improve the experience: the user no longer sits staring at an empty screen while the model generates the answer. The agent is built with Strands Agents, whose BedrockModel calls ConverseStream by default. The agent iterates over the events with stream_async() and forwards them to the frontend.
ConverseandConverseStreamneed different IAM permissions: the first onebedrock:InvokeModeland the second onebedrock:InvokeModelWithResponseStream. If you move to streaming with only the first one, you will get anAccessDenied.
A shared wrapper for every call
I did the migration with a single file, bedrock-converse.ts, the same in all five backends. It does three things:
- It retries only transient errors (
ThrottlingException,ServiceUnavailableException, HTTP 429) with exponential backoff and full jitter. That is, it waits a random time between zero and a cap that doubles on every attempt, so several Lambdas do not retry at the same moment. - It writes one log line per call with the operation, input and output tokens, latency, retries and a hash of the prompt.
- It lets each call decide how much to retry, because they do not all have the same room. Behind API Gateway, which cuts off at 29 seconds, I use 2 retries with a maximum wait of 3 seconds. In an SQS worker I use 5 retries with a maximum of 20 seconds, and if it still fails, the message goes back to the queue.
If you add your own retries, check the SDK’s. The AWS client already retries on its own, and the two policies multiply. Wherever I have mine, I turn the SDK’s off.
Be careful with retries. An unbounded retry can keep a Lambda running and billing for minutes, and retrying an error you already know will repeat (a
ValidationExceptionfrom a malformed request, anAccessDeniedExceptionor a prompt that exceeds the model’s context) only adds latency and cost. That is why the wrapper retries only transient errors, with a maximum number of attempts and a cap on the wait, and everything else fails on the first try.
What to do when the model fails
A model may not answer: throttling, a timeout, a response that cannot be parsed. What should not happen is each application deciding what to do in that case without having thought about it. The wrapper now documents a policy per type of task:
| Type of task | If the model fails |
|---|---|
| Moderating or classifying something that will be published | Never approve blindly. Send it to human review or ask the user to retry |
| Scoring something that has an equivalent deterministic calculation | Use the deterministic calculation |
| Something the user asked for, such as a draft | A clear error so they can retry. Never make up a result |
| Something decorative, such as an extra sentence in an email | Leave it out |
Applying it turned up the most serious bug of the whole review. In the Announcements Hub, if Bedrock failed while moderating a comment, the comment was published unmoderated, because the code treated it as valid. Now the API answers with a 503 and the user sees a message asking them to try again in a moment. The comment is not published, and it is not stored in any queue either, because the user sends it again.
The default branch of a
catchis a product decision, not a technical detail. Look at what each model call in your code returns when it fails: one of them may well be letting everything through when it fails, without you knowing.
Knowing what each application costs
I tag all my resources with a project tag and group by it in AWS Cost Explorer. Bedrock was the exception: it showed up as cost with no project. That makes sense, because an invocation by model ID cannot be tagged, and the Lambda’s tags do not carry over to token usage.
Besides Cost Explorer, I use a tool of my own to know how much I spend on each project and to get proactive email alerts. It is a command-line (CLI) tool and it is open source. I explain how it works in FinOps on AWS: from theory to practice.
The solution is an application inference profile. It is a copy of a model or of a system profile (for example eu.amazon.nova-2-lite-v1:0) that does accept tags. The application invokes the profile ARN instead of the model ID, the profile itself is free, and the spend shows up grouped by its tags.
Cost Explorer only shows tags, not the profile name, so several profiles with the same tags give exactly the same breakdown as one. And since each profile is tied to a single model, the useful rule is one profile per model, application and environment.
Attribution is not retroactive and has daily granularity. The tag also has to be activated as a cost allocation tag in Billing before it shows up in Cost Explorer.
A CloudFormation resource that does not exist in eu-south-2
I defined it with CloudFormation’s AWS::Bedrock::ApplicationInferenceProfile resource. cdk synth and the tests passed, but the deployment failed when creating the change set:
1
Template format error: Unrecognized resource types: [AWS::Bedrock::ApplicationInferenceProfile]
That resource type does not exist in eu-south-2, where I deploy, even though the Bedrock API does answer there. I confirmed it in three steps:
aws cloudformation describe-typereturnsTypeNotFoundExceptionineu-south-2and a valid type ineu-west-1.aws bedrock list-inference-profilesworks ineu-south-2.aws cloudformation validate-templatewith a minimal template reproduces the same pipeline error.
A resource appearing in the CloudFormation reference does not mean it is available in your region. A
validate-templateagainst the target region is a cheap check before deploying.
The fix was to create the profile by calling the Bedrock API through a CDK AwsCustomResource:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const profile = new cr.AwsCustomResource(this, 'Profile', {
onCreate: {
service: 'Bedrock',
action: 'CreateInferenceProfile',
parameters: {
inferenceProfileName: props.profileName,
modelSource: { copyFrom: systemProfileArn },
tags, // explicit: stack tags are not propagated
},
physicalResourceId: cr.PhysicalResourceId.fromResponse('inferenceProfileArn'),
},
// Profiles are immutable: an update only re-reads the existing one.
onUpdate: {
service: 'Bedrock',
action: 'GetInferenceProfile',
parameters: { inferenceProfileIdentifier: profileId },
physicalResourceId: cr.PhysicalResourceId.fromResponse('inferenceProfileArn'),
},
onDelete: {
service: 'Bedrock',
action: 'DeleteInferenceProfile',
parameters: { inferenceProfileIdentifier: profileId },
},
installLatestAwsSdk: false,
// policy: scoped to the profile, the system profile and the base model
});
Kiro wrote this code for me, so writing it was the easy part. These are the details that are not obvious:
- Tags have to be passed explicitly. Stack tags do not reach what a custom resource creates. And if you apply them with
Tags.of(), which is an Aspect, they do not exist yet when the resource is built. - You need
onUpdate. Without it, any change to the definition, such as a new tag, repeats theCreateand fails with a duplicate name. - A profile cannot change its model. When I moved an application to Nova 2 Lite,
onUpdateonly re-read the existing profile, which still copied Nova Lite, and the deployment would have gone green. To change models, give the profile a new name and a new logical ID, so CloudFormation creates the new one before deleting the old one. installLatestAwsSdk: falseuses the SDK already in the Lambda runtime instead of installing it from npm on every deployment.- IAM permissions without
*. Invoking through the profile requires permission on the application profile, the system profile it copies and the base model. With a cross-Region profile, the model has to be covered in every region it can route to.
This solution has a consequence worth knowing before you adopt it: changing an application’s model means creating a new profile and deleting the old one. An inference profile cannot change its model, so trying another model stops being an environment variable change and becomes an infrastructure change, with a new ARN that the application has to receive. In four of my applications the profile and the Lambda that uses it are in the same CDK stack, so the new ARN arrives in the same deployment with no extra work. In the news application the profile is stored in SSM and read by other stacks and by the chat agent, so the order matters there: the profile first, then everything that reads it. The tags stay the same, so Cost Explorer keeps grouping in the same way.
Measuring prompts instead of assuming they work
With non-zero temperature, a model does not always give the same result. A run that passes proves nothing, and a run that fails may be noise. The advice is clear: before comparing versions of a prompt, measure how much the same version varies.
In the agent I added a script that repeats each case N times and only fails if the pass rate drops below a threshold (N=5 and a minimum of 80% by default). It settled a real doubt: a prompt change seemed to make three normal queries worse. Comparing both versions on the same cases, both got 17 out of 20 right. There was no regression, and without repeating I would have reverted a good change.
I also changed the model that acts as judge. In My AI Agent Worked. My Tests Didn’t. I explained that I use an LLM to evaluate the agent’s answers: that model, the judge, receives the query, the data and the answer, and scores one specific criterion. My quality tests used a Nova model to judge Nova 2 Lite answers, and the exam guide warns about self-preference bias: a model tends to score answers that look like its own higher. The judge is now Claude Haiku 4.5, from another family, configurable through an environment variable.
Measuring without Langfuse in the other applications
I use Langfuse traces only in the chat, because it is public and anyone can see a bad answer. In the other applications I do not want to send prompts and answers to an external service, and two things are enough:
- A prompt hash in every log, produced by the wrapper. That tells me which prompt version produced each result, without a hand-maintained version that nobody updates.
- An evaluation with fixed cases, only where mistakes matter. It is a set of test cases whose correct verdict I already know, run against the real model and compared with what I expect. Cloud Confessions moderation has 11 clear cases: tech humour, doxxing, discrimination, spam, gibberish, questions that are not confessions and an instruction injection. Each one has its acceptable verdicts. The evaluation fails if it approves something it should not, or if it gets fewer than 85% right. It does not run on every
npm test, because it costs money and depends on the model, so I run it when I change the prompt or the model.
I have not added evaluations to drafts and recommendations. The user always reviews the text before using it, and a mistake there is an annoyance, not harm.
The first time I ran that evaluation it scored 27%. It was not the model: my AWS credentials had expired, every call failed, and the wrapper did what it should, which was to send each case to review. The evaluation counted those failures as model verdicts. It now aborts as soon as a call fails. An evaluation that cannot tell a technical failure from a model answer gives you a number that measures nothing.
Guardrails: measure before deciding
In the guardrails article I explained which protections I would turn on in the agent and why. While revising the security domain of the exam guide I checked how the chat agent’s guardrail was set up and found it half done: the code read GUARDRAIL_ID, but nothing passed it to the runtime. I never got to test it, because I had ruled it out on cost. In practice, the only thing protecting the agent was the system prompt.
Before deciding anything I measured it. I wrote seven typical attacks against a chatbot (make it reveal its instructions, make it adopt another personality, make it obey fake orders or make it ignore its rules) and ran them against the real agent:
| Case | With the old prompt |
|---|---|
| Extracting the system prompt (in English and in Spanish) | Always failed: it refused and then dumped the whole prompt |
| Making it adopt another personality with no rules (the attack known as DAN) | Failed sometimes, and in one run it ended up giving a pizza recipe |
A message pretending to be a system order ([SYSTEM]) | Failed sometimes |
| Asking it to ignore its instructions, and off-topic requests | It resisted them |
A standalone confidentiality clause inside the prompt did not fix it. What worked was a SECURITY RULES section at the top of the prompt, taking priority over everything else, plus a short reminder at the end. With that change, the agent stopped revealing its prompt over five full runs. One case remained, the message pretending to be a system order, failing 2 times out of 5.
That kind of attack is exactly what the Guardrails PROMPT_ATTACK filter covers. Even so, I am not turning it on today. The agent is read-only, works on public announcements and has a limit of 10 queries a day per user, so the realistic worst case is someone using it as a free LLM for ten messages. What I have done is leave it ready: the guardrail is created with a CDK context flag, its ID is published in SSM and the deployment injects it into the runtime when it exists.
In an enterprise application the answer would be different: identifiable users, possibly internal data and someone who will ask what protects the chatbot. There I would turn on
PROMPT_ATTACKon input and PII filtering, and enforce the guardrail through IAM with thebedrock:GuardrailIdentifiercondition.
What I decided not to use
I reviewed these services and features from the exam guide and, for now, decided not to use them:
- Amazon Bedrock Prompt Management. I already tried it when I wrote the articles about prompts (Prompts II) and today I do not use it in any of my applications. I have few prompts, versioned in git and covered by evaluations. It would make sense with many prompts shared across teams, with non-developers editing them, or with an audit requirement for approvals.
- Prompt caching. My system prompts are short and the volume is low, so the saving would be a few cents.
- Semantic search. The agent searches by keyword, and for announcements with exact service names that works well. A vector store adds cost even when it is idle.
- Model invocation logging. It stores full prompts and answers. With user data that brings privacy and retention obligations, and I already have tokens and latency in my own logs.
- Langfuse in every application. Only where the answer is public.
The most useful part of the review was not memorizing services, but sitting down with my own applications and the exam’s questions. Several things I took for granted were not true: a guardrail that was never actually applied, a prompt that revealed its instructions, Bedrock spend with no project and comments that were published unmoderated when the model failed. If you have something running on Bedrock, the table above is a good place to start reviewing yours.
On this blog
- Amazon Bedrock 101: Understanding Its Foundational Models
- Prompts I: How to Design Effective Prompts in Amazon Bedrock
- Prompts II: Effective Prompt Management in Amazon Bedrock
- My AI Agent Worked. My Tests Didn’t.
- My Agent Is in Production. What Is It Doing in There?
- I Have Traces From My Agent. Now What?
- My Agent Is Public. What If Someone Tries to Manipulate It?