Student Notes — Chapter 4: Answers from the IT knowledge base (RAG), with the policy page cited

AWS in Action: AI edition · Chapter 4

Before you start

Nothing is needed to read this chapter or to run its offline checks beyond Python and the development tools listed in the course repository’s requirements-dev.txt. The offline checks use a fake knowledge base and read local files, so they need no AWS account and cost nothing. The live steps (deploying the stack, uploading and syncing the policy pages, asking the real model) need an AWS account, in a Region where Amazon Bedrock and S3 Vectors are both offered; they create real resources and cost money, so you delete the stack at the end. Do them from the AWS console first, with CloudShell, a terminal that opens in the browser, as the alternative. Sign in as an IAM or IAM Identity Center user, not as the root user: as the root user the sync step stops with an AccessDeniedException that names the root user. Before the live steps, check that you have a Region with both services, a budget alert, Amazon Nova Lite and Amazon Titan Text Embeddings V2 in the Bedrock Model catalog, and the course code on your computer, cloned or downloaded from the course repository. The steps are written out in two places: SETUP.md at the top of the course repository, and the section Before you try in chapters/04-policy-answers/README.md, which gives a console route beside each command. This chapter builds on chapter 3: you should know what a bucket, CloudFront and a template are. Recap: a bucket is a container for files in S3, CloudFront is the front door that serves the chat page from a private bucket and forwards /ask to the API, and a template builds a stack that is created and deleted as one unit. Have paper or a notes app open for the Flashcards and the Model answer.

Three ways to follow this chapter, the same as before, and you can mix them:

• Watch: read the chapter and follow the build on screen. Nothing needed.
• Copy and run: copy the folder chapters/04-policy-answers/ and run its checks.
• Build with AI: open chapters/04-policy-answers/prompts/, run each prompt in an AI coding tool, then run that prompt’s check. The checks for the whole chapter are in chapters/04-policy-answers/prompts/CHECK.md.

Summary

This chapter makes the helpdesk answer from the company’s own IT policy pages and say which page it used. You put three policy pages in a folder of the private bucket from chapter 3 and create a Bedrock knowledge base, a searchable index of those pages. When an employee asks a question, the function makes one retrieve-and-generate call: Bedrock finds the closest pieces of the pages, a model writes the answer from them, and the reply carries citations. The function turns the citations into a short list of pages with excerpts, and the chat page shows the first cited page beside the answer. You extend chapter 3’s template, sync the pages, try it, and delete the stack.

Key concepts

Term	Definition
Retrieve and generate	Two steps in one call: look up the pieces of your own documents that match the question, then have a model write the answer from those pieces.
Knowledge base	A searchable index of your documents that Amazon Bedrock builds from a data source and keeps in a vector store.
Data source	Where a knowledge base reads its documents from. Ours is the policies/ folder of the private bucket.
Chunk	A small piece of a document, a few sentences long, that is searched and quoted on its own. By default about 300 tokens, kept to whole sentences.
Embedding	A list of numbers that places a piece of text so that similar meanings sit close together.
Vector	The list of numbers that an embedding is. The question is turned into one too, so the closest chunks can be found.
Vector store	Where the vectors are kept and searched. Ours is Amazon S3 Vectors: a vector bucket holding a vector index.
Sync (ingestion job)	The step that reads the data source, chunks it, embeds it and writes the vectors. Run it again after a page is added, changed or removed.
Citation	A pointer from part of an answer to the piece of source text it came from, naming the document.
Sources list	What our function returns beside the answer: each cited policy page once, with a short excerpt.
Model ARN	The full name of the model, or of its inference profile, that writes the answer, built from the ModelId and ModelRoute parameters, which the call needs in this form.

Explanations

Orientation: where this chapter sits

Chapter 3 built the door. This chapter fills the room behind it with the company’s own words. In the finished helpdesk the model reaches an IT knowledge base, and the answers are cited:

Today you build the knowledge base part of the picture.

Try this first: an employee asks “How do I reset my password?” A model alone answers from what it learned in training. Should the answer come from the model’s memory alone, from the company’s own password page found when the question arrives with that page named in the reply, or from a person who copies the steps out of the page by hand? Pick one and say why. The chapter builds one of them.

By the end you can:

• explain how a question goes from the pages in S3 to an answer with a cited page;
• read the new template pieces and the handler and say what each is for;
• sync the pages, ask a question, check the citation and delete the stack.

Part 1. The goal: an answer with its page

A student in an open-book exam does not recite from memory. They find the right page, then write the answer, and they can show you the page. That is what the helpdesk will do. An employee asks “How do I reset my password?”, the helpdesk looks the question up in three IT policy pages, and the reply carries the answer and the names of the pages it used. The chat page then shows the first cited page beside the answer, so the employee can check it.

The code for this chapter is in the course repository under chapters/04-policy-answers/. The three policy pages are in chapters/04-policy-answers/app/policies/: password-reset.html, vpn-access.html and new-laptop.html. The function is chapters/04-policy-answers/app/ask/handler.py, the chat page is chapters/04-policy-answers/app/web/index.html, and the stack description is chapters/04-policy-answers/template.yaml. Chapter 3’s files are still there. A prompts/ folder holds one prompt per build step with a check for each, and a prompt is finished when its check passes.

Part 2. Why a model needs your documents

Three situations first, because each shows a way a model alone fails. You have met the first kind in chapter 1: a confident answer that is wrong.

Situation 1: the stale answer. The company changes its password steps.

Policy page:   "Reset it in the staff portal."
Model memory:  "Call the help line."   (illustrative)

The model keeps repeating the old steps. Cause: a model’s memory was fixed when it was trained, while a page can be edited today.

Situation 2: the missing page. An employee asks about a rule that no page covers.

Question:  "Can I use my own phone for work?"
Pages:     none about phones     sources: []

A confident answer might still appear. Cause: with nothing to retrieve, there is nothing to cite. The reply then carries an empty sources list, which is the signal to check the answer and to add the missing page.

Situation 3: the wrong chunk. A question about the VPN is answered from a laptop set-up page.

Question:  "VPN will not connect"
Cited:     new-laptop   (a laptop set-up page)

Cause: the search picks the pieces that match best, and one piece of the wrong page happened to match. The cited page and its excerpt give it away, which is why they sit beside the answer. Part 6 returns to these three.

The fix for these three is the same idea. Retrieve and generate means two steps: first look up the pieces of your own documents that match the question, then have the model write the answer from those pieces. Think of a librarian who fetches the right binder before answering. The model answers from your pages, not from memory alone.

Part 3. Pages in S3 and a knowledge base

A library catalogue lists the books on small cards; you search the cards, then fetch the book. The index at the back of a textbook points you to the right pages. A knowledge base is a searchable index like that for your documents. Amazon Bedrock builds it from a data source, here the policies/ folder of the bucket, and keeps it in a vector store, here Amazon S3 Vectors. The pages stay in the bucket; the knowledge base keeps a searchable index of them.

Two ideas explain how pages become searchable. First, the chunk. A manual cut into index cards gives cards that each hold one idea; a loaf cut into slices gives slices you serve one at a time. When Bedrock reads a data source it first splits each document into chunks, so that the right small piece can be found. By default a chunk is about 300 tokens, kept to whole sentences; tokens are from chapter 1. For content converted from HTML, the documentation says the chunker may follow logical boundaries such as sections, so say “about”.

Second, the embedding. On a map, two cafés on the same street have nearby coordinates. An embedding gives a piece of text coordinates for its meaning: it is a list of numbers, called a vector, that places the text so that similar meanings sit close together. Books shelved by topic work the same way: neighbours are related. Each chunk is converted to an embedding and written to a vector index. The question is turned into numbers too, and the closest chunks are the ones used. The documentation notes that embeddings allow texts to be compared with numbers.

Read the picture from left to right: pages, chunks, embeddings, vector store. This preparation runs when you sync, not each time a question arrives, like a chef’s preparation before service. Remember it as: prepare the index once, then search it for each question.

Our vector store is S3 Vectors, a bucket type built to store and query vectors, with a vector index that holds the vectors. The AWS documentation describes it as cost-effective vector storage with no infrastructure to provision, best suited to workloads that are queried infrequently, which fits a helpdesk demo. The template creates a vector bucket and a vector index. The index is set to a size of 1024 numbers per vector, because the embedding model used here, Titan Text Embeddings V2, outputs 1,024 numbers by default, and it compares vectors by cosine distance. Use a Region where Amazon Bedrock and S3 Vectors are both offered.

The data source tells the knowledge base which pages to read:

S3Configuration:
  BucketArn: !GetAtt SiteBucket.Arn
  InclusionPrefixes:
    - policies/

The prefix policies/ means files under that folder are read, so the chat page in the same bucket is not indexed. One private bucket holds both, and the prefix keeps them apart.

Now the step that surprises people. Uploading a page does not make it searchable. The documentation says that each time you add, change or remove files in the data source you must sync it so it is re-indexed. A sync is an ingestion job:

aws bedrock-agent start-ingestion-job \
  --knowledge-base-id "$KB" \
  --data-source-id "$DS"

You check the job until its status is COMPLETE, and the documentation says a short wait may still be needed after that for new vectors to be available for querying. Until then the knowledge base does not know the new page.

The command above needs three things first: the stack is deployed and complete, the page and the policy pages are already in the bucket under policies/, and you run it as a user that is not the root user. KB and DS are the values of the KnowledgeBaseId and DataSourceId outputs; the deploy block in Part 7 reads them from the stack. In the console the same sync is Amazon Bedrock → Knowledge base → your knowledge base → Data source → Sync, with a green banner when it works.

Part 4. Retrieve and generate

An employee’s question travels through the search and then to the model:

Read it from left to right: the question is searched against the index for the closest chunks, the model writes the answer from those chunks, the reply carries the answer plus citations, and the cited page is shown beside the answer. One call does the search and the writing.

In chapter 2 the function called the model with the Converse API. Now it calls the knowledge base through the Bedrock agent runtime with one retrieve-and-generate request:

response = agent.retrieve_and_generate(
    # …
)
answer = response["output"]["text"]

The elided settings are the question as input, then a configuration with the type KNOWLEDGE_BASE, the knowledge base id, the model’s ARN, and the length cap and temperature, as in chapter 2. The model’s ARN is its full name, which the template builds from the ModelId and ModelRoute parameters: most Regions serve the model only through an inference profile (a model id with a geography in front, as in chapter 2), and then the ARN names that profile. The knowledge base id comes from the stack. Both reach the function as environment variables. The generated answer is in output, and the cited source chunks are in citations. Chapter 2’s guards are unchanged: a blank, non-text or over-long question is refused with 400 before any call, and a failed call is logged and answered with a safe 502.

A citation is a footnote: it points from part of an answer to the piece of source text it came from and names the document. A recipe card that names its cookbook lets you go and check. Each retrieved reference in a citation holds the chunk’s text, metadata about the document and the location of the document in the data source. The function’s sources helper keeps the S3 location, turns it into a page name under policies/, lists each page once in the order first cited, and cuts the chunk’s text to 300 characters as the excerpt. A name outside the pattern for policy pages is dropped.

You can build and test the handler with an AI coding tool. The prompt chapters/04-policy-answers/prompts/03-handler.md writes the call and chapters/04-policy-answers/prompts/04-handler-tests.md writes the tests, which replace the knowledge base with a fake and check the answer, the cited pages, the dropped locations, the 400 and the 502. If the default model cannot answer knowledge base questions in your Region or account, pass another model id as the ModelId parameter.

Part 5. The cited page beside the answer

A good reply is one the page can use directly. This is what the function returns for a good question (the answer text here is illustrative):

{"answer": "1. Open the staff portal …",
 "sources": [{"page": "policies/password-reset.html",
              "excerpt": "…"}]}

answer is the model’s text. sources lists each cited page once with a short excerpt, and if nothing is cited it is an empty list. The function hands the page a list; the page decides what to show.

The page is a shop window that shows the shelf the item came from. After a good answer it adds a “Sources:” line with one button per page and shows the first page at once. Showing a page sets the panel title and the excerpt as text, and loads the page into a frame beside the chat:

const PAGE = /^policies\/[A-Za-z0-9._-]+\.html$/;
if (!PAGE.test(source.page)) return;
frame.src = "/" + source.page;

The first line is a pattern: a name must start with policies/ and end in .html. A name that does not match is not shown, and the function checks the same pattern, so a strange location from the knowledge base is stopped twice. The frame loads the policy page from the same address as the chat page, through CloudFront, so it comes from the private bucket and the template’s CloudFront setup already serves it. The frame carries an empty sandbox attribute, which applies all of the browser’s restrictions to the embedded page, so the policy page is shown as a page. The policy pages have no scripts and no links, and a test checks it. The prompt chapters/04-policy-answers/prompts/05-cited-page-panel.md builds the panel, and chapters/04-policy-answers/tests/test_policy_panel.py checks it.

Part 6. What to check

Retrieval can go wrong in three ways, and you have met these three: the wrong chunk, a missing page, and an answer with no source. This table turns them into a routine:

Symptom	What it usually means	What you do
The cited page does not fit the question	The best-matching chunk came from the wrong page	Read the excerpt, improve the pages, sync again
The reply has no sources	No page matched, or the page was not synced	Treat the answer as unchecked; add or sync the page
The answer matches an old page	The page changed after the last sync	Sync again

An answer with no source is one to check, not one to trust. A model can write a fluent answer without a source, and the empty list is your warning. Give each topic its own page, so the best match is the right page.

What this chapter leaves open, and where the course deals with it:

What is still open	What the helpdesk will do	Built in
Anyone with the API’s own address can still call it	Sign-in and roles	Chapter 7
The helpdesk answers but does not act	Create and read tickets through tools	Chapter 5
Guardrails check the input and the generated answer, not the pages retrieved at runtime	Guardrails for input and output, with this gap named	Chapter 7
Nobody watches cost or errors	Logs, cost and an alarm	Chapter 9

Part 7. Extend the template, test, delete

Templates in this course are cumulative: chapter 4 holds the resources chapter 3 held, plus what it adds. Chapter 3’s twelve resources stay as they were. The file is chapters/04-policy-answers/template.yaml, and it adds five:

Piece	What it is for	Name in the template
Vector bucket	Holds the vector index	PolicyVectorBucket
Vector index	Stores the vectors and finds the closest	PolicyVectorIndex
Knowledge base role	What Bedrock may do while indexing and searching	KnowledgeBaseRole
Knowledge base	The searchable index of the policy pages	PolicyKnowledgeBase
Data source	Points at the policies/ folder	PolicyDataSource

The knowledge base role is a staff badge for Bedrock itself: it can be assumed by the Bedrock service for this account and for knowledge bases in this Region, and it lets Bedrock use the embedding model, list and read the policy pages, and use the vector index. The function’s role also gains permission to retrieve and generate, and the template gives the function the knowledge base id and the model’s ARN as environment variables. The data source is set to delete its vectors when it is removed, so the vector bucket is empty when the stack deletes it; a vector bucket can be deleted only when it is empty. Resources that can carry tags are tagged chapter: session-04 plus your project and environment. Every name you choose starts with Project and Environment, as in chapter 2 (helpdesk-dev-policies for the knowledge base), and nothing has a Retain deletion policy. The template adds two outputs you use: KnowledgeBaseId and DataSourceId; the ModelInUse output from chapter 2 shows the model or profile id the stack chose. The prompt chapters/04-policy-answers/prompts/02-knowledge-base-stack.md builds this and its tests, and cfn-lint template.yaml is its check.

Costs money. The live steps create real resources, ingestion calls an embedding model, and each question calls a generation model. Deploy, try and delete in one sitting.

aws cloudformation deploy --stack-name helpdesk-dev-ch4 --template-file template.yaml --capabilities CAPABILITY_IAM --parameter-overrides Project=helpdesk Environment=dev
BUCKET=$(aws cloudformation describe-stacks --stack-name helpdesk-dev-ch4 --query "Stacks[0].Outputs[?OutputKey=='SiteBucketName'].OutputValue" --output text)
SITE=$(aws cloudformation describe-stacks --stack-name helpdesk-dev-ch4 --query "Stacks[0].Outputs[?OutputKey=='ChatUrl'].OutputValue" --output text)
KB=$(aws cloudformation describe-stacks --stack-name helpdesk-dev-ch4 --query "Stacks[0].Outputs[?OutputKey=='KnowledgeBaseId'].OutputValue" --output text)
DS=$(aws cloudformation describe-stacks --stack-name helpdesk-dev-ch4 --query "Stacks[0].Outputs[?OutputKey=='DataSourceId'].OutputValue" --output text)
aws s3 cp app/web/index.html "s3://$BUCKET/index.html" --content-type text/html
aws s3 cp app/policies "s3://$BUCKET/policies" --recursive --content-type text/html

The first command creates the stack, with Project and Environment set so every name starts helpdesk-dev. The next four read its outputs. The last two copy the chat page and the policy pages into the bucket. Then run the sync from Part 3 and check the job until it prints COMPLETE:

JOB=$(aws bedrock-agent start-ingestion-job --knowledge-base-id "$KB" --data-source-id "$DS" --query "ingestionJob.ingestionJobId" --output text)
aws bedrock-agent get-ingestion-job --knowledge-base-id "$KB" --data-source-id "$DS" --ingestion-job-id "$JOB" --query "ingestionJob.status" --output text

Open the address in $SITE, ask “How do I reset my password?” and check that the password reset page appears beside the answer. From a terminal:

curl -s -X POST "$SITE/ask" -H "content-type: application/json" -d '{"question": "How do I reset my password?"}'

The reply holds the answer and a sources list. These commands are also in chapters/04-policy-answers/README.md, and the whole live check is in chapters/04-policy-answers/prompts/CHECK.md.

A hotel guest checks out and the room is cleared; a room left occupied keeps being charged. A stack you do not delete keeps costing money. The bucket still holds the page and the policy pages, and a bucket can be deleted only when it has no objects in it, so empty it first:

aws s3 rm "s3://$BUCKET" --recursive
aws cloudformation delete-stack --stack-name helpdesk-dev-ch4
aws cloudformation wait stack-delete-complete --stack-name helpdesk-dev-ch4

The wait command polls until the stack is gone. Deleted stacks are hidden from the console’s stack list by default, so a list that no longer shows helpdesk-dev-ch4 confirms it.

Practice: what do you do?

Decide each one before reading the call: what you do, and what the employee sees.

Problem	Our call and reason
You edit a policy page and upload it	Sync: start an ingestion job and check it until COMPLETE
An employee asks about a rule no page covers	The reply carries no sources: treat the answer as unchecked, add the page and sync
A VPN question cites the laptop set-up page	Read the cited page and excerpt, give each topic its own page, and sync again
A citation points outside the policies folder	The function drops it, and the page would not frame it
You finish the chapter	Empty the bucket, delete the stack and wait until it is gone

Flashcards

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. What does retrieve and generate mean?

Two steps: first look up the pieces of your own documents that match the question, then have the model write the answer from those pieces. Like an open-book exam: find the right page, then write. The model answers from your pages, not from memory alone.

2. Why can a model alone give a stale or made-up answer?

Its memory was fixed when it was trained, so it can repeat old steps or invent a plausible one, and nobody can check where it came from. A page can be edited today, which is why the answer is built from the current page.

3. What is a knowledge base?

A searchable index of your documents that Amazon Bedrock builds from a data source and keeps in a vector store. Like a library catalogue: you search the cards, then fetch the book. The pages stay in the bucket.

4. What is a chunk, and why split pages into chunks?

A chunk is a small piece of a document, a few sentences long, searched and quoted on its own. By default about 300 tokens. Splitting lets the search find the best small piece and send just that beside the question.

5. What is an embedding?

A list of numbers, called a vector, that places a piece of text so that similar meanings sit close together, like map coordinates for meaning. The question is turned into numbers too, and the closest chunks are used.

6. What does the sync do, and when must you run it?

It is an ingestion job that reads the data source, chunks it, embeds it and writes the vectors. Run it after any page is added, changed or removed, and check it until COMPLETE; uploading alone does not make a page searchable.

7. Which pages does the data source read?

Files under the policies/ prefix of the site bucket are read. The chat page sits in the same private bucket but is not indexed, because the prefix keeps the two apart.

8. What does the retrieve-and-generate call return that a plain model call does not?

Besides the generated answer in output, it returns citations: the source chunks the answer used, each with its text, metadata and the location of its page. That is what lets the page name its sources.

9. What is a citation?

A pointer from part of an answer to the piece of source text it came from, naming the document, like a footnote in an essay. Our function keeps each cited policy page once, with a short excerpt of up to 300 characters.

10. How does the chat page show the cited page safely?

It shows a name when it matches the policy-page pattern, loads it from the same address in a frame with an empty sandbox attribute, and sets the title and excerpt as text. The function checks the same pattern first.

11. What do you do when a reply has no sources?

Treat the answer as unchecked. No page matched, or the page was not synced. Check the topic, add the missing page and sync. A model can write a fluent answer without a source, so the empty list is the warning.

12. What does chapter 4 add to the template, and how do you delete it cleanly?

Five resources: PolicyVectorBucket, PolicyVectorIndex, KnowledgeBaseRole, PolicyKnowledgeBase and PolicyDataSource. Empty the site bucket, run delete-stack, then wait stack-delete-complete; the data source deletes its vectors so the vector bucket goes too.

Model answer

The core question this chapter answers: “The helpdesk should answer from our IT policy pages and show which page it used. How does that work, from the pages to the cited page on screen?”

Four things a colleague listens for, in this order: pages in S3 · knowledge base · retrieve and generate · cited page. Skip any of these four and the answer is incomplete.

1. Pages in S3 — the policy pages sit in a folder of the private bucket
2. Knowledge base — a sync splits them into chunks, turns the chunks into embeddings and stores those in a vector index
3. Retrieve and generate — one call finds the closest chunks and writes the answer from them, with citations
4. Cited page — the function returns pages and excerpts, and the chat page shows the first beside the answer

The full answer, said aloud:

(Pages in S3) “The IT policy pages are files in a policies folder of the same private bucket that holds the chat page.

(Knowledge base) A Bedrock knowledge base reads that folder. When I sync, it splits each page into chunks, turns each chunk into an embedding, a list of numbers that places its meaning, and stores them in a vector index. Uploading alone is not enough; I sync after any change.

(Retrieve and generate) When an employee asks, the function makes one retrieve-and-generate call. Bedrock finds the closest chunks and a model writes the answer from them. The reply carries citations to the pages the answer came from.

(Cited page) The function turns the citations into a short list of policy pages with excerpts. The chat page shows the first cited page beside the answer, when its name matches the policy-page pattern, and an empty list is my cue to check the answer.“

Missing a point is a failure state. The follow-up question lands where you went quiet: finish without the knowledge base and the next question is how the pages become searchable, and finish without the cited page and the next question is how anyone checks the answer. Practise until the four come out without prompting, then stop talking and let the questions come.

Why this matters

In a conversation with a colleague: a team that adds a model to its helpdesk is asked where the answers come from and how a user can check. The four points answer both, and the helpdesk is a specific example.

On the job: a support lead auditing answers, an IT team updating a policy page, and an engineer costing a demo ask versions of the same questions. The sync, the citations and the choice of a vector store with no standing machine answer them.

In the rest of this course: chapter 5 gives the same helpdesk tools to create and read tickets, and chapter 7 returns to the gap that guardrails check the input and the generated answer, not the pages retrieved at runtime. A reader who knows these seven parts can predict why each piece is there.

After this unit

Assignment. The chapter project asks you to deploy the stack, upload your own policy pages, sync, ask questions, check the cited page, and delete it. Keep the template: the next chapter extends it. The brief itself is the project file.

What’s next. The next chapter gives the helpdesk tools, so that a request such as “Request Figma access” creates a ticket and a question about a ticket number reads its status.