Student Notes — Chapter 1: Meet the LLM: what it is, what it does well, where it fails

AWS in Action: AI edition · Chapter 1

Before you start

Nothing to install or prepare. This is a concept chapter: you build nothing and deploy nothing. Have paper or a notes app open, because the Flashcards and the Model answer ask you to say and write things in your own words.

Summary

This chapter introduces the large language model (LLM) through one running example, a company IT helpdesk handling VPN, MFA, laptop and access-request questions. It shows what a model is, a program that writes the text most likely to come next, and why a reply that sounds right is not the same as a reply that is right. It then sorts the work into what a model does well (summarise, classify, draft, answer from text) and four ways it goes wrong: it makes things up, it gets exact facts and sums wrong, it has a limit on how much text it takes in at once, and it struggles when many people ask together. It ends with a way to decide, task by task, whether to trust the model, check its result or keep a person on the decision. The rest of the course builds the helpdesk around exactly these strengths and weaknesses.

Key concepts

Term	Definition
Large language model (LLM)	A program trained on a very large amount of text. Given some text, it writes the text that most plausibly comes next, one small piece at a time. It is a type of foundation model: a model trained on a massive amount of diverse data that can produce many kinds of response.
Prompt	The input you give a model: a question, instructions, and any text you want it to use. Prompts are used for tasks such as classification, question answering, code generation and creative writing.
Token	A small piece of text a model reads or predicts as one unit. It can be a whole word, part of a word, or a punctuation mark.
Hallucination	A reply that is fluent and confident but invented or wrong: a fact, a step, a name or a source that does not exist.
Context window	The amount of text a model can take in at once. Your question, any pages you supply and the reply it writes share that space.
Trust	Letting the model do the writing, when the text it needs is supplied and a wrong reply would do no harm.
Check	A person or a system verifies the result against a source before anyone relies on it.
Keep a person on it	A person makes the decision, because a wrong outcome would cause harm or the decision has consequences.

Explanations

Orientation: the course and the helpdesk

This is the first chapter of AWS in Action: AI edition, a ten-chapter course that builds one thing: a company IT helpdesk assistant that answers employees’ questions about VPN, MFA, laptops and access requests. Think of this chapter as the plan of a house you are about to build: you walk through the finished rooms before you pick up a tool. Chapters 2 to 9 build the helpdesk, one piece at a time, using Python. Each build chapter creates what it needs on AWS and deletes it at the end. This chapter builds nothing.

One helpdesk, one piece per chapter. Here is what it can do at the end of each:

Chapter	What the helpdesk can do at the end of it
1. Meet the LLM	Nothing yet: you learn what a model can and cannot do
2. First answer	Answers “my laptop won’t connect to the VPN” sent to a web address (an API)
3. Chat page	Has a chat page employees open in a browser
4. Knowledge base	Answers from the company’s own IT pages and names the page
5. Tickets	Opens a ticket and reads its status back
6. Triage agent	Checks for an outage, tries a known fix, else hands the case to IT staff
7. Sign-in and safety	Knows who is asking, masks pasted passwords, blocks off-topic requests
8. Ship like a team	Deploys on every change, through staging to production
9. Run it	Has logs, a cost per ticket, an alarm and a quality check
10. Your roadmap	You can walk an interviewer through it, chapter by chapter

Why getting to know AI matters

Teams are already asking whether a model can take on part of their work, such as the first reply on a helpdesk. Someone has to answer that in plain words: what it is good for, and what must be checked. That someone does not need to build a model. You do not need to build an engine to drive a car well, but you do need to know that it needs fuel, that it brakes slower on a wet road and that it cannot steer itself into a parking space you did not choose. In the same way, a cook does not need to make an oven to use one, but a cook who does not know it runs hot at the back burns the bread. What you need is a working picture of what a model does. Every chapter of this course builds on that picture.

How a service desk works today, without a model

Follow one ticket through an IT desk that has no model:

Step	Who	What it costs
1. The employee raises a ticket from the helpdesk page	Employee	Waits in the queue
2. An IT admin picks it up and looks through dashboards, the knowledge base and the user database	IT admin	Manual research, on every ticket
3. The admin replies with instructions	IT admin	The same answer typed again and again
4. The employee tries the steps; if they fail, back to step 1	Employee	Another round trip

Every answer waits for an admin’s time and research. Think of a library with one librarian: every reader queues to ask, and the librarian walks to the same shelf for the same book many times a day.

Now the same desk with a model, side by side:

The model takes over the waiting and the manual research: it reads the company’s own pages and writes the first reply in seconds, or it opens a ticket with a summary so IT staff start the hard cases with the facts ready. It is accurate only when it is given those pages to read, and that is what this course builds.

Here is the finished system in plain blocks: a chat page, the model, the company pages, tickets and a hand-over to IT staff. Each block is one chapter’s work.

And the same blocks as AWS services. Each box shows the chapter that builds it. You do not need these names today; each one is introduced in its own chapter.

Try this first: an employee types “My laptop won’t connect to the VPN” into a chat assistant, and it answers at once with five confident steps. Do you follow the steps, check them against the company’s own VPN page, or pass the ticket to a person on the IT desk? Pick one and say why. The chapter gives you a way to decide.

The questions this chapter prepares you for are asked in real rooms: a manager deciding whether to fund an AI pilot asks “can we let a model answer our helpdesk tickets?”; an engineer reading a complaint asks “why did it give a confident wrong answer?”; a team writing the rules for an AI feature asks “what should a person still check?”; a design review asks “what does a model not know about us?”. Knowing where a model fails is what makes it safe to use where it works.

The chapter has four parts: what an LLM is; what it is good at, and where a person stays; where it goes wrong; and deciding what to trust. By the end you can:

• explain what a model does and why a fluent reply is not proof of a correct one;
• sort a helpdesk task into trust, check or keep a person, and give the reason;
• answer “can we let a model answer our IT helpdesk?” in three points, aloud.

Part 1. What an LLM is

An employee types into the helpdesk: “My laptop won’t connect to the VPN.” A reply comes back in a second: “Try restarting your laptop, check that your Wi-Fi is working, and reinstall the VPN client.” It reads like a capable helpdesk agent wrote it.

Here is what happened. A large language model (LLM) is a program trained on a very large amount of text. It was shown so much text that it learned what usually follows what. Given your words, it writes the text that most plausibly comes next, one small piece at a time.

Two ways to hold the idea:

• A well-read colleague who has not worked here. They have read the whole library, so they speak well about almost anything. They have not seen your company’s VPN product, your outage page or this employee’s account.
• Autocomplete on your phone, scaled up. Your phone suggests the next word from patterns in what you type. A model does the same with vastly more reading, so it can continue whole paragraphs.

Look at the question again. The model recognised a familiar kind of problem from the words, and it wrote in a polite, helpful tone. What it could not see was the VPN product, the current outage status or the employee’s account. It filled that gap with common advice. So the reply was written in a helpful voice but was not an answer about this company’s setup.

How is a reply made? The question becomes a prompt. The prompt is split into tokens, small pieces of text that may be a word, part of a word or a punctuation mark. The model picks the next token, adds it to the text so far, and repeats until the reply is finished. The flow is drawn below.

It is like a chef tasting as they cook: the reply is built a spoonful at a time. Nothing in the loop looks facts up.

The consequence is the idea the whole course rests on: plausible is not the same as true. A model is trained to continue text well, not to look facts up. Its tone gives you no signal about whether it is right, in the way a stranger giving directions with total certainty tells you nothing about whether the directions are correct.

Part 2. What it is good at, and where a person stays

Give a new intern a ticket thread, a highlighter and an instruction: “shorten this”, “which team does this belong to?”, “write a polite reply asking for the error message”. They are fast and they are good. Now ask the same intern for the company’s gateway address with no document in front of them, and the quality drops. The difference is whether the material is in front of them.

A model works the same way. It does well at jobs where the material is already in front of it and the result is more text a person can read and judge. Four jobs cover most of what a helpdesk needs:

Job	Helpdesk example	What a person checks
Summarise	Shorten a 40-message ticket thread for the next agent	Nothing important was dropped
Classify	Label “I can’t log in” as access, VPN, MFA or laptop	The label fits the ticket
Draft	Write a polite reply asking for the error message	Tone and instructions are right
Answer from text	Answer “how do I reset MFA?” from a policy page pasted in	The answer matches the page

A second picture: a mail-room sorter reads each envelope and puts it in the right tray. No deep knowledge is needed, just careful reading of what is in front of them.

The strongest of the four is answer from text, because the facts are supplied. An open-book exam is easier than a closed-book one, for people and for models. Ask the same MFA question without the policy page and the model has to guess. In chapter 4 you build this: the helpdesk hands the model the company’s own policy page and shows the page next to the answer.

Each of the four jobs has a quick check: a person looks at the text and the result. That is a good sign for a task. If you cannot say how a reader would check the result, think again before handing it to a model.

Now where does a person stay? Follow one ticket through the desk:

The model labels the ticket and drafts the reply. A system fetches the policy page and the account facts, because the model does not hold them. If the ticket could change access or settings, a person approves or refuses first. Think of a pilot and an autopilot: the autopilot flies the plane, and the pilot signs for the flight. In the same way, someone has to be able to say “I own this answer”, explain a decision and fix a mistake. A model writes text; it does not answer for it.

Put together, the pros and cons of a model on the helpdesk look like this. The cons here are not the failures of Part 3; they are the running costs of the assistant, and each is a piece of the build:

Pros	Cons, and the chapter that handles each
Understands questions typed in everyday words	It knows nothing about your company until it is given the pages, which someone must keep up to date (chapter 4)
Writes a first reply in seconds, at any hour	Every question has a cost, which you have to watch (chapter 9)
Summarises long threads and labels tickets	Pasted passwords and off-topic requests need rules on what goes in and out (chapter 7)
Leaves IT staff the hard cases, with the summary ready	A change can quietly make replies worse, so a fixed set of questions is re-run after each change (chapter 9)

A new coffee machine in the office is a fair comparison: it serves anyone at any hour, but someone refills the beans, someone pays for them and someone notices when it starts making weak coffee. The pros are why we build the assistant; each con is a job in the build, not a reason to stop.

Remember: the model writes, a system supplies the facts, and a person approves anything that could change access or settings. In our view, the desk changes shape rather than disappearing: the tedious work moves to the model, and the decisions stay with people. That last sentence is an opinion, not a measured result, and reasonable people disagree.

Part 3. Where it goes wrong

Four things go wrong, and each has a chapter that fixes it:

What goes wrong	What it looks like	Handled in
It makes things up (a hallucination)	A menu that does not exist; “Done!” when nothing was done	Chapter 4: answers from company pages
It gets exact facts and sums wrong	A made-up server address; a wrong seat count	Chapter 5: tool use
It has a limit on how much text it takes in at once (the context window)	A long thread where the first complaint is lost	Chapter 4: send just the matching pages
Many people ask at once and share limited capacity	A Monday morning where some get no reply	Chapter 9: an alarm on token use spots it

Here is the failure first, for the exact facts and sums. Three questions to a model that was given no documents. The wrong replies are illustrative, written for these notes:

Question asked	A wrong reply	What is wrong
“What is the address of the Pune VPN gateway?”	A made-up server name that looks right	It was not shown the real one
“We bought 1,247 seats and assigned 1,089. How many are left?”	“168 seats are left.”	The right answer is 158; the reply was off by ten
“What is our new Figma access form called?”	A confident form name from last year’s process	It was not shown the new process

These replies do not sound uncertain. They are as polite and fluent as correct ones. To a model, a number or an address is text to continue plausibly, not a value to look up or compute. Think of reciting a phone number from memory: you may get most digits right and still dial the wrong person. The fix sits outside the model: fetch exact facts from the company’s own page or system, let code do the sum, and supply new information in the question.

A hallucination is a reply that is fluent and confident but invented or wrong. Think of a new hire who does not want to say “I don’t know” and makes up a plausible answer instead of checking, or a map with an invented road that looks complete until you drive there. When the model has little to go on, it fills the gap with something that fits, not something true. The more specific the detail, the more convincing the wrong answer.

The context window is the amount of text a model can take in at once. Picture a desk of a fixed size: a limited number of pages fit, and to add one another has to go. Your question, the pages you supply and the reply it writes share that space. The size is measured in tokens and differs between models, so this chapter states no figure; you read the current one from the model’s documentation when you choose a model. Bring the right three pages to the desk, not the whole filing cabinet. Choosing the matching pages is the retrieval idea you meet in chapter 4.

When many people ask at once, each question is its own request, and the requests share the same limited capacity. Amazon Bedrock’s documentation says model use is controlled by quotas on token usage, that the quotas which apply to an account may be lower than the published defaults, and that quota increases are not granted automatically. Picture a restaurant at lunchtime: many tables order at once and one kitchen serves the room. A busy moment is a real design problem, not a rare event. Chapter 9 watches token use and sets an alarm for this reason.

Each failure enters the reply flow at a different step, so each needs its own fix:

Now three real situations on the desk. In each, read the failure first, then the cause, then what a good desk reply says. The tickets, replies and logs are illustrative, written for these notes.

Situation 1: the MFA reset that did not happen.

Employee, ticket #4821, 08:52:
  New phone, and my sign-in prompt never arrives.
  Can you reset my MFA? I have a client call at 9:30.

Assistant, 2 seconds later:
  Done! I have reset your MFA token. Sign in again and
  you will be asked to set up your new phone.

Nothing was reset: the assistant has no way to change an account. It wrote the reply a helpful desk would write, not one based on something that happened. Cause: hallucination. The employee reads “Done!”, stops looking for help and misses the call. A good desk reply: “I can’t reset MFA from this chat. I have sent your ticket to the identity team, marked urgent for your 9:30 call. They will confirm it is you and do the reset.”

Situation 2: the long thread that lost its first complaint.

Ticket #5107, pasted whole: 41 messages
  1      Employee: Since the laptop swap, Outlook asks for my
         password every hour and the finance-ap mailbox is gone.
  2-40   Reboots, a VPN reinstall, driver updates.
  41     Employee: Still asking for my password. Any news?

Assistant:  Summary: VPN dropouts after a laptop swap. A reboot and
            a client reinstall were tried. Next: reinstall the VPN client.

The summary is fluent and sounds complete, but the real complaint is missing. It is like joining a long meeting halfway through: your summary of the meeting misses what was said first. Cause: the context window. The first message did not get enough attention, or did not fit. The next agent works on the VPN and the employee waits another day. A good desk reply: “Your first message was about Outlook asking for your password hourly and the missing finance-ap mailbox. The VPN steps were a side issue. I am assigning this to the mail team with that message on top.”

Situation 3: the Monday-morning burst.

Helpdesk log, Monday (illustrative)
  09:00  About 200 "I can't log in" questions arrive together
  09:01  The first replies go out normally
  09:02  The service answers "too busy" to many requests
         and nothing retries them
  09:05  Employee: I asked five minutes ago and got nothing.
         Is IT down?

The same question from everyone is the busiest moment, not a rare one. Many requests shared one limited capacity, with no waiting or retry. Cause: many people asking at once. An employee who gets no reply cannot tell whether the helpdesk is broken. A good desk reply: “We are getting a lot of questions this morning. Yours is in the queue and the answer will arrive here. If you are locked out and it is urgent, call the desk line.”

What the three good replies share: each says what happened, what happens next and who owns it.

Part 4. Deciding what to trust

A pilot’s pre-flight checklist is short and the same on each flight, and it catches the slips that feel unlikely. Deciding what to trust works the same way. Three questions sit underneath it: was the text the reply needs supplied, could a wrong reply cause harm, and who sees the reply before it matters? Drawn as a decision:

Three outcomes: trust the model with the writing, check the result, or keep a person on the decision. Applied to the helpdesk:

Ask yourself	Trust the model with it	Check it or keep a person on it
Is the source text supplied?	Answer from a policy page	An answer from what it remembers
Is an exact value involved?	Explaining a result	A number, address or price
Could a wrong reply cause harm?	Drafting a friendly message	Granting access or changing settings
Does it decide for someone?	Suggesting a category	Approving or refusing a request
Will anyone review it?	A draft a person sends	A reply that goes straight to the employee

The more a wrong answer would cost, the more checking it gets. In short: trust it with the writing, check it with the facts.

Each problem you have met has a fix in the system around the model, and that is how the course is laid out:

Problem	What the helpdesk will do	Built in
It makes things up	Answer from the company’s own pages and show the page	Chapter 4
Exact facts and sums	Let tools, such as a ticket system or code, supply them (tool use)	Chapter 5
Hard cases	An agent that hands over to a person	Chapter 6
Pasted secrets, off-topic questions	Guardrails: rules on what may go in and come out	Chapter 7
A change makes it worse and nobody notices	Run a fixed set of questions after each change (evaluation)	Chapter 9

Where do people still fit? This part is the teacher’s view, not a hiring statistic; no survey or job-market figure is claimed. A restaurant is more than a stove: someone orders the food, keeps the kitchen safe and tastes before serving. A model is the stove. Around it sit jobs that set it up (knowing what a good IT answer is, taking it from an idea to a working page, building and running it the same way each time) and jobs that check it and stand behind it (verifying replies, watching logs, cost and quality, and explaining to managers what it can and cannot do).

In plain words: DevOps is building and running the helpdesk the same way each time; end-to-end deployment is taking it from an idea to a page employees can use. Someone with IT knowledge can spot a wrong IT answer faster than a newcomer can. So the argument is that your existing job skills are not replaced; they are pointed at a new kind of system. Chapter 10 returns to how to explain this to an interviewer.

How you talk about a model shows whether you have this picture. Statements to use, and statements to avoid:

Say this	Not this
“It answers from the policy page we give it”	“It knows our systems”
“I would check any address, number or price against the source”	“It is usually right”
“A person approves access requests”	“The model handles tickets by itself”
“It can fail when many people ask at once”	“It scales without effort”
“In my view, my current job skills still matter here”	“An AI job is guaranteed”

Practice: sort five tasks

Decide each one before reading the call: trust the model with the writing, check the result, or keep a person on the decision.

Task	Our call and reason
Label an incoming ticket as VPN or access	Trust it: the text is in front of it and the label is easy to see
Tell an employee how long the VPN outage will last	Check it: the model cannot see the outage status
Work out seats left: 1,247 bought, 1,089 assigned	Check it: let code do the sum (the answer is 158)
Decide whether a contractor gets access to the finance reports	Keep a person on it: it is a decision with consequences
Draft a friendly reply asking for the error message, which a person sends	Trust it: it is writing, and a person sends it

Flashcards

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. In one sentence, what does a large language model do?

It writes the text that most plausibly comes next, one small piece at a time, based on a very large amount of text it was trained on. It does not look facts up, which is why a fluent reply is not proof of a correct one.

2. What is a token, and how is a reply built from tokens?

A token is a small piece of text: a word, part of a word or a punctuation mark. The model picks the next token, adds it to the text so far and repeats, like a chef tasting as they cook, until the reply is finished.

3. Name the four jobs a model does well, and why they have something in common.

Summarise, classify, draft, and answer from text. In these four the material is already in front of the model and the result is text a person can read and judge, so a quick check is possible.

4. Why is "answer from text" the strongest helpdesk job?

Because the facts are supplied in the question, such as a policy page, so the model does not have to guess. It is like an open-book exam: the same reader does better when the book is open.

5. Who stays on the desk, and why?

A person approves anything that could change access or settings, makes exceptions and answers for the outcome. A model writes text but does not answer for it, so someone must be able to explain a decision and fix a mistake.

6. What is a hallucination, and why is a specific wrong answer worse than a vague one?

A hallucination is a fluent, confident reply that is invented or wrong. Specific detail, like exact names or a "Done!", makes a wrong answer sound as if it came from a real source, so people believe and follow it.

7. Why does a model get exact facts and sums wrong, and what is the fix?

To a model a number or an address is text to continue plausibly, not a value to look up or compute. The fix sits outside the model: fetch the fact from the company's page or system, and let code do the sum.

8. What is a context window, and what shares it?

It is the amount of text a model can take in at once. Your question, any pages you supply and the reply the model writes share that space, like pages on a desk of a fixed size.

9. Why do many people asking at once matter for a helpdesk?

Each question is a separate request, and they share limited capacity. Busy moments such as Monday morning can hit the limit, so the system needs retries, separation of requests and a watch on token use.

10. What three questions decide trust, check or keep a person?

Was the text the reply needs supplied; could a wrong reply cause harm; and who sees the reply before it matters. The more a wrong answer would cost, the more checking it gets.

11. A model on the helpdesk has running costs even when it works. Name two and the chapter that handles each.

Any two of: the company pages it answers from must be kept up to date (chapter 4); every question costs money, so the cost per ticket is watched (chapter 9); pasted passwords and off-topic requests need rules on what goes in and out (chapter 7); a change can quietly make replies worse, so a fixed set of questions is re-run after each change (chapter 9).

Model answer

The core question this chapter answers: “A manager asks: can we let a model answer our IT helpdesk? What would you trust it with, what would you check, and why?”

Three things a manager listens for, in this order: trust it with · check · keep a person on. Skip any of these three and the answer is incomplete.

1. Trust it with — what a model does well
2. Check — where it goes wrong, and what you verify
3. Keep a person on — what a person decides

The full answer, said aloud:

(Trust it with) “A model does well at work on text it is handed: summarising a long ticket, labelling it, drafting a reply, and answering from a policy page we give it.

(Check) It goes wrong in four ways: it can make things up with total confidence, it gets exact facts and sums wrong, it has a limit on how much text it takes in at once, and a busy morning shares limited capacity. So an address, a sum or a new policy should come from a system, not from the model’s memory, and I would check replies against the source.

(Keep a person on) A person stays on anything that grants access, makes an exception or changes a system, because the model writes text but does not answer for it.“

Missing a point is a failure state. The follow-up question lands where you went quiet: finish without the last point and the next question is who is responsible when it is wrong. Practise until the three come out without prompting, then stop talking and let the questions come.

Why this matters

In a conversation with a manager: teams adopting models want people who can say, in plain words, what a model is good for and what to check. The three points give you a structure that leaves no obvious gap, and a specific helpdesk example makes it believable.

On the job: a manager deciding on an AI pilot, an engineer reading a complaint about a bad reply, and a team writing the rules for an AI feature ask versions of the same question. The decision chart and the problem-to-fix table answer it.

In the rest of this course: the chapters ahead answer the problems named here. A reader who knows the failures can predict why each build step exists.

After this unit

Assignment. The chapter project asks you to sort a list of helpdesk tasks into three groups, trust the model, check the result, and keep a person on it, and to give a reason for each. Keep your sorted list: the next chapter starts from it. The brief itself is the project file.

What’s next. The next chapter gives the helpdesk its first real answer: you call a hosted model through an API from a small Python function, put it behind an HTTP endpoint, create the whole stack from one template, and delete it afterwards.