Courses / Agentic AI from Scratch (Python)

Session 11 of 40

Memory: the conversation kept by a checkpointer, trimming long chats, and what not to keep

Notes

How to Add Memory to a Chat: Keep the Conversation So a Follow-up Question Works

Agentic AI Foundations · Session 11 — Memory: keeping the conversation

Before you start: what do you need to build a chat that remembers?

You need the code folder for this session, the code/ folder, open in your terminal. It holds helpdesk_chat.py, the two prompts in model_prompts/ (sort.md and answer.md), data/policies/ with the three policy files, and a build prompt in prompts/. Have your environment switched on, your three settings set as in session 1, and the libraries from session 9 installed (pip install langchain-core langchain-openai pydantic). Nothing new to install. There is no test set in this session: the test set from earlier sessions sends one question at a time with no history, so it cannot check a follow-up, and a multi-turn test would need a different design. Have paper or a notes app ready for the Flashcards and the Model answer.

Summary: how does a chat understand a follow-up question?

A language model keeps nothing between calls, so a question such as “And when must they be used by?” means nothing to it alone: who are “they”? This session turns the two-step helper from session 10 into a chat loop, in which your code keeps a list of earlier messages, adds each question and answer to it, and sends the list again with every new question, so the helper can read “they” as the days from the question before. You limit how many messages are kept, because the whole list goes out again with every call and a long one may not fit the model’s context window, and you see that the list holds whatever the person typed, so a password typed into the chat is sent to the model again.

Key concepts: memory, history, message, placeholder, KEEP_MESSAGES and follow-up defined

Term Definition
Memory In a chat, a list of earlier messages that your code sends again with every new question. The model itself keeps nothing between calls.
History The Python list that holds the earlier messages, as pairs such as ("human", text) and ("ai", text).
Message One question or one answer in the history; a question and its answer are two messages.
Follow-up A short question that depends on the one before it, such as “And when must they be used by?”.
MessagesPlaceholder A blank in a prompt template that takes a whole list of messages, not one piece of text.
KEEP_MESSAGES A setting for how many of the latest messages are kept and sent again; 6 means the last three exchanges.
Chat loop A loop that reads a line, calls ask_helper(question, history), prints the reply and updates the history.
ask_helper(question, history) The two-step helper from session 10, now taking the history as a second argument.

Explanations: how does the helper remember what was said?

Where are we? What does this session add to the HelpDesk Copilot?

A reply follows five steps: pick the policy section, build the request, read it as tokens, write the reply token by token, and give the answer. Today changes step 2. The request now carries the earlier messages as well as the policy and the question, so steps 3 to 5 read the whole conversation. The code is in code/: helpdesk_chat.py is the session 10 helper as a chat loop.

Why does the model forget, and what is memory?

Imagine a clerk who starts every phone call with a blank mind. You ask how many leave days can be carried over, they answer, you hang up. You call again and say “and when must they be used by?” The clerk cannot help: who are “they”? A friend who forgets everything each time you hang up is the same. You would start the second call by reading out the notes of the first.

That is what memory is here. A doctor reads the patient’s file before each visit; the file is the memory, not the doctor. In the Copilot the file is a Python list of pairs, ("human", text) and ("ai", text). The model gives each answer from what is in front of it, and if an earlier message is not in the list sent with the question, the model has never seen it. When you stop the program, the list is gone.

Here is a real conversation. The replies vary in wording from run to run:

You: How many unused annual leave days can I carry over?
Helper: You can carry over up to 5 unused annual-leave days into the next leave year.  [leave, Leave Policy 4.2, needs_human=False]
You: And when must they be used by?
Helper: Carried-over days must be used by 31 March.  [leave, Leave Policy 4.2, needs_human=False]
You: Who approves my leave request?
Helper: Your manager approves or declines a request.  [leave, Leave Policy 4.3, needs_human=False]
You: What if my manager does not answer?
Helper: A request that is not answered is not approved.  [leave, Leave Policy 4.3, needs_human=False]

The helper understood “they” because the earlier question and answer came with it.

How does the chat loop keep the history?

A shared notebook passed across a desk is the picture: the clerk reads the notebook, answers, and you write the answer into it. The chat loop does exactly that. Run it with python helpdesk_chat.py, type each line after You: and press Enter; an empty line ends the chat, and so does Ctrl+C, even while the helper is still answering.

history = []
try:
    while True:
        question = input("You: ").strip()
        if question == "":
            break
        reply = ask_helper(question, history)
        ...
        history.append(("human", question))
        history.append(("ai", reply["answer"]))
        history = history[-KEEP_MESSAGES:]
except (EOFError, KeyboardInterrupt):
    print()

After each answer the code adds the question and the answer to the list, then keeps only the last KEEP_MESSAGES. The list is added to after the answer, so the next question carries it. The try around the loop is the door out: Ctrl+C (or the end of input) jumps to the except line, which prints a blank line and ends the program quietly instead of with an error. That list is the entire memory. One trap: in Python history[-0:] is the same as history[0:], which is the whole list, so KEEP_MESSAGES = 0 does not switch memory off; it keeps everything. Use 2 or more to see the cap work.

What is a messages placeholder, and how do both chains get the history?

In session 10 each prompt was a system text and the new question. Think of a script with a gap for a whole earlier scene: the lines of that scene drop in between the opening and the new line. A messages placeholder is that gap. LangChain’s reference describes it as a placeholder that can be used to pass in a list of messages. In helpdesk_chat.py it sits between the system prompt and the new question, in both templates:

answer_prompt = ChatPromptTemplate.from_messages(
    [("system", answer_text), MessagesPlaceholder("history"), ("human", "{question}")]
)

ask_helper takes the history as a second argument and passes it to both chains:

def ask_helper(question, history):
    label = sort_chain.invoke({"question": question, "history": history}).label
    if label == "other":
        ...   # a person takes over, as in session 10
    policy = (DATA / "policies" / f"{label}-policy.md").read_text()
    values = {"question": question, "policy": policy, "history": history}
    reply = answer_chain.invoke(values).model_dump()

The sorter needs the history too, because a follow-up has no topic words of its own: “And when must they be used by?” says nothing about leave. The sorting logic is the same as session 10; only the history is new. Both chains also keep session 10’s with_fallbacks, so a typed reply the provider refuses with a 400 error is asked for once more as a tool call.

The prompts changed by one line each. model_prompts/sort.md now says that a short follow-up such as “and when must they be used by?” belongs to the same topic as the messages before it, so use them. model_prompts/answer.md says a short follow-up belongs to the same topic, so read the earlier messages to know what “it” or “they” means. The list gives the model the past, and the prompt tells it to use it.

How much history should the chat keep, and what are the limits?

Think of a desk notepad with room for six lines: write a seventh and the oldest line has to go. KEEP_MESSAGES = 6 is the new setting at the top of the file. Each question and each answer is one message, so 6 means the last three exchanges. Older messages fall off the front, so the list does not grow without end:

KEEP_MESSAGES = 6
...
history = history[-KEEP_MESSAGES:]

It is a trade. All of the list is sent each time, so every kept message adds to what goes out on every call. Keep too few and the thread is lost: with a small value the earlier messages fall off the list and a follow-up has nothing to point at. LangChain’s guide on short-term memory makes the same point: a full history may not fit in the model’s context window, which can cause context loss or errors, and it lists trimming the first or last N messages among the common ways to cope. Its guide also says conversation history is the most common form of short-term memory. Your history[-KEEP_MESSAGES:] is the simplest form of that trimming.

What does memory hold, and what should stay out?

A whiteboard in a meeting room keeps whatever anyone writes, for the next person to read. A note slipped into the clerk’s folder is read again at every later call. Memory kept this way holds what was typed, not only what was useful. If someone types a password into the chat, it is sent to the model again on the next question, and again until it falls off the end of the list. Decide what the chat must not hold before you build it, not after.

What are three situations where memory goes wrong, and what does each mean for the design?

  1. A follow-up is answered as if it were a new question. history was not passed to both chains, or the placeholder is missing from one of the two templates.
  2. The chat loses the thread after a few questions. KEEP_MESSAGES is small, so the earlier messages fell off the list.
  3. Someone types a password into the chat and it comes back with the next call. The history holds whatever was typed, and all of it is sent again.
What you learned The design decision Where it shows
The model keeps nothing between calls Keep a list and resend it history in the chat loop
A follow-up needs the past in both steps Put the placeholder in both templates, pass history to both chains MessagesPlaceholder("history")
Every kept message is sent on each call Cap the list KEEP_MESSAGES
The list holds whatever was typed Decide what must stay out The chat loop and your own rules

If something fails, check this table:

You see Do this
ModuleNotFoundError: langchain_openai Run the pip install line from session 9, with your environment switched on.
An error with 401 The service did not accept the key (a typing slip, or a key that is no longer valid); check your settings as in session 1. With no key at all, the program stops earlier with No API key.
A follow-up is answered as if it were a new question Check that history is passed to both chains and that the placeholder is in both templates.
Helper: no typed reply came back. The model’s reply could not become a typed reply. Ask again.

Basic assignment: build the chat and have a conversation that needs a follow-up.

Project step: build the chat. Start from your session 10 helpdesk_router.py, with the two files in model_prompts/ updated to mention follow-up questions. Write helpdesk_chat.py with your coding assistant using prompts/1-write-helpdesk-chat.md. Run it, ask about carrying over leave days, then “And when must they be used by?”, and check the answer is about 31 March. Then set KEEP_MESSAGES = 2, ask the carry-over question, then “Who approves my leave request?”, then “And when must they be used by?”, to see the helper lose the thread (do not use 0: history[-0:] keeps the whole list); and ask “What is the capital of France?” in the middle of a chat to see whether the next leave question is still understood. Stop when you have seen a follow-up work and one fail.

Practice: explain and decide. First, answer the Model answer question below aloud in four beats before reading it. Second, for each of these, say what you would change: (1) a follow-up is answered as a new question; (2) the chat forgets the topic after three questions; (3) a user typed a password earlier in the chat.

Flashcards: chat memory, message history and follow-up questions with answers

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. Does the model remember an earlier call? No. It keeps nothing between calls. A chat seems to remember because your code sends the earlier messages again with the new question.
2. What is memory in `helpdesk_chat.py`? A Python list of `("human", text)` and `("ai", text)` pairs. After each answer the code adds the question and the answer, and when you stop the program the list is gone.
3. Why does "And when must they be used by?" need the history? On its own it says nothing about who "they" are or what the topic is. With the earlier question and answer sent along, the helper reads "they" as the carried-over leave days.
4. What is `MessagesPlaceholder("history")`? A blank in a prompt template that takes a whole list of messages, not one piece of text. It sits between the system prompt and the new question.
5. Why is the placeholder in both templates? Both steps need the past. The sorter must know what a follow-up is about to give it the right label, and the answerer must know what "it" or "they" means.
6. What does `KEEP_MESSAGES = 6` mean? The code keeps only the last six messages, which is the last three exchanges, since a question and an answer are two messages. Older ones fall off the front of the list.
7. Why limit the history at all? All of it is sent each time, so every kept message adds to each call, and a very long history may not fit in the model's context window. A cap keeps the size steady.
8. What goes wrong if `KEEP_MESSAGES` is too small? The earlier messages fall off the list and the chat loses the thread: a follow-up has nothing to point at and is treated as a new question.
9. What is the risk of memory kept as a list? The list holds whatever the person typed. If someone types a password into the chat, it is sent to the model again on the next question, until it falls off the list.
10. A follow-up is answered as a new question. What do you check? Check that `history` is passed to both chains and that the placeholder is in both templates. Missing from either one means that step does not see the earlier messages.

Model answer: how does the helper understand a follow-up, and what are the risks?

The core question this session answers: “The helper understood ‘they’ as the carried-over days. Does the model remember our earlier chat? How does the follow-up work, and what are the risks?”

Four beats, in this order: the model forgets · the list is the memory · limit it · keep sensitive text out. Skip any of these four and the answer is incomplete.

  1. The model forgets: it keeps nothing between calls.
  2. The list is the memory: the code adds each question and answer to a list and sends it again with the next question, through MessagesPlaceholder("history") in both templates.
  3. Limit it: KEEP_MESSAGES keeps only the last messages, since the whole list is sent on each call.
  4. Keep sensitive text out: the list holds whatever was typed, and it is sent to the model again.

The full answer, said aloud:

(The model forgets) “No, the model does not remember our earlier chat. It keeps nothing between calls.

(The list is the memory) The follow-up works because my code keeps a list of the earlier questions and answers and sends it again with each new question, through a placeholder in both prompt templates. That is why it knew ‘they’ meant the carried-over days.

(Limit it) I keep only the last few messages with a setting called KEEP_MESSAGES, because everything in the list is sent on every call and a long one may not fit the context window. Too small and it loses the thread.

(Keep sensitive text out) And the list holds whatever was typed. If someone puts a password in the chat it goes back to the model on the next question, so I decide what must stay out.“

Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and the next question is what happens to a password typed into the chat. Use this answer when an interviewer or a design-review panel asks whether your chatbot remembers. Practise until the four come out without prompting.

Why this matters: where do you use chat memory at work?

When a user asks if the AI remembers them: you can say exactly what happens: the program keeps a list for the length of the chat and resends it, and the model keeps nothing itself.

In a design review: someone asks why the chat keeps only the last few messages. Every kept message is sent on every call, so you cap the list and say what the cap is.

In a privacy and security review: a reviewer asks what happens to something sensitive typed into the chat. You know it is sent again with each question while it stays in the list, so you decide up front what must stay out.

In the rest of this course: later sessions give the Copilot documents to search and agents that act, and the same questions come back: what does it keep, and what should it not?

After this unit: what is the assignment and what comes next?

Assignment. Build the chat and have a conversation that needs a follow-up. The brief for the project work is the project file. Keep helpdesk_chat.py with its two updated prompt files.

What’s next. Session 12 starts on the company’s documents: loading files and splitting them into pieces.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.