Courses / Agentic AI from Scratch (Python)
Session 4 of 40
The Chat Completions API: messages, streaming, the model landscape and choosing a model
Notes
How to Use the Chat Completions API in Python: Messages, Streaming, Open Models and Choosing a Model
Agentic AI Foundations · Session 4 — The Chat Completions API: messages, streaming, the model landscape and choosing a model
Before you start: what do you need to build a chatbot with the Chat Completions API?
You need the setup from session 1: Python with your environment switched on, and the three settings (LLM_BASE_URL, LLM_API_KEY and LLM_MODEL) at the top of each script or set in your terminal. No new library is needed. For the model comparison you need a second model to call: another model on the same provider, a second provider, or a model server on your own machine. Keep your project folder open, because the chatbot you build goes into it. Have paper or a notes app ready for the Flashcards and the Model answer.
Summary: how do messages, streaming and model choice work in the Chat Completions API?
A Chat Completions request sends the model a list of messages, each with a role, and the reply comes back as the next message. Each request is independent and stateless, so a chatbot remembers a conversation because your program keeps the list and sends the whole history with each turn. Streaming returns the reply in pieces as it is written, and because the endpoint format is shared, the same code can reach hosted models, open-weight models on a cloud platform and models on your own machine by changing the base URL, the key and the model, and you choose among them by comparing quality, cost and speed on your own questions. The project step builds v0, a chat in the terminal that keeps the conversation.
Key concepts: message, role, streaming, base URL and open-weight model defined
| Term | Definition |
|---|---|
| Message | One entry in the list sent to the model: a role saying who wrote it, and the text. |
| Role | The label on a message: system for standing instructions, user for the person asking, assistant for a reply from the model, and developer, which OpenAI’s o1 models and newer use for instructions in place of system. |
| Chat Completions request | A request that sends the model a list of messages plus settings such as the model name; the reply is the next assistant message. |
| Stateless request | A request the model answers on its own, with no memory of earlier ones. Your program supplies earlier messages again to keep a conversation. |
| Streaming | Returning the reply in pieces as the model writes it, instead of in one block at the end. |
| Chunk | One piece of a streamed reply, carrying a small delta of the text. |
| OpenAI-compatible endpoint | A server, hosted or on your own machine, that accepts requests in the Chat Completions format. |
| Base URL | The address your code sends requests to; changing it points the same code at another provider. |
| Open-weight model | A model whose trained weights are published, so a cloud platform, another company or you on a local server can host it. |
Explanations: how do I build a chatbot, stream a reply and pick the model behind it?
Where are we? What does this session add to the HelpDesk Copilot?
Until now each call sent one question and got one answer. This session turns that into a conversation. You will see how messages and roles are laid out, why the whole history is sent each time, how to stream a reply, how the same code reaches other models, and how to choose one. The code you use is two short scripts: stream_reply.py, which asks one question and prints the answer a few words at a time, and chat.py, a chat in the terminal that remembers the conversation because it sends the whole list again on each message. Both start with the three settings from session 1. The project step is v0, a chat in the terminal.
| Box | What it does for the employee |
|---|---|
| Terminal chat (v0) | Where you type a question and read the reply, with the conversation remembered |
| Streamed reply | Shows the answer as it is written, not after a wait |
| Model comparison | Puts the same chat to two models so you can choose with evidence |
What are messages and roles in a Chat Completions request, and why is the whole history sent each time?
Failure first. An employee asks the Copilot, “How many days can I carry over?” and gets a good answer. The next message is, “And when do I have to use them by?” The reply is:
I am not sure what “them” refers to. Could you tell me which days you mean?
This reply is illustrative, written for these notes. The model is not too small and the employee did not phrase it badly. The program sent just the latest message, so the model did not see the first question.
Why? A Chat Completions request sends the model a list of messages, each with a role saying who wrote it, plus settings such as the model name. The reply is the next message, written by the assistant. Think of a script for a play: each line has a speaker, and the model writes the next line. Or a form with labelled boxes: standing instructions in one box, the question in another.
The roles are labels on the lines:
| Role | Who writes it | What it holds in the Copilot |
|---|---|---|
system |
You, the developer | Standing instructions: who the Copilot is, how to answer |
user |
The employee | The question or follow-up, in their own words |
assistant |
The model | An earlier reply, sent back so the model sees what it said |
developer |
You, the developer | The same job as system for OpenAI’s newer models |
OpenAI’s reference describes a developer message as instructions from the developer that the model should follow regardless of user messages, and says its o1 models and newer take developer messages in the place of the older system messages. This course writes standing instructions as system, which is what many compatible servers expect, so check your own model and provider if a standing instruction seems to be ignored.
Here is a request in code, from chat.py. It starts the list with one system message that sets the model’s role for the whole chat:
messages = [{"role": "system", "content": "You are a friendly IT help desk assistant."}]
Each turn adds a user message, sends the whole list, and keeps the answer as an assistant message (the full loop is in the project step below).
Now the idea that fixes the failure. A stateless request is one the model answers on its own, with no memory of earlier requests. OpenAI describes each text generation request as independent and stateless, and says a multi-turn conversation is built by providing additional messages as parameters. Think of a call centre where the agent has a fresh notepad on each call: the agent can help once you read the case file out again. Or a colleague with no memory of yesterday, to whom you hand over the notes each morning before asking the next question. Alternating user and assistant entries in the list record where the conversation stands, inside a single request. In v0, “memory” is simply the list your program keeps and resends.
Turn 1 sends: [system, user1] -> reply1
Turn 2 sends: [system, user1, reply1, user2] -> reply2
Turn 3 sends: [system, user1, reply1, user2,
reply2, user3] -> reply3
The list grows by two messages a turn, and every
message in it counts against the context window.
The growth has a cost. The context window is the maximum number of tokens in a single request, and it includes input, output and reasoning tokens. As you include more turns, both the output and the window limit matter, which you met in session 2. The whole list is sent again on each call, so the input token count of each request grows turn by turn, because it carries more history than the request before. What one message costs depends on its own length, not on its position; later in the course you trim long chats.
How do I stream an LLM reply in Python?
Think of a live commentator versus a match report: you hear each moment as it happens. Or a printer that feeds a page line by line instead of waiting for the whole document. Streaming returns the reply in pieces as the model writes it, instead of in one block at the end. By default, OpenAI generates the model’s entire output before sending it back in a single response. Streaming lets you start showing the beginning while the model continues generating the rest. It changes when the employee sees the words, not what the words are.
You turn it on with stream=True. The API then sends chunks, and each chunk carries a small delta of the reply. OpenAI’s reference says the reply data is pushed to your client while it is being produced, as server-sent events. The course script is stream_reply.py. Its question is “In five short steps, how do I fix a laptop that cannot join the office Wi-Fi?”, and the part that streams is:
stream = client.chat.completions.create(
model=os.environ.get("LLM_MODEL", LLM_MODEL),
messages=[{"role": "user", "content": question}],
stream=True,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
The if guards against a chunk whose choices list is empty (OpenAI’s reference says the extra usage chunk, when requested, has an empty choices array) and against a delta that carries no text. print(..., end="", flush=True) keeps each piece on the same line and shows it at once. If nothing prints until the end, check that stream=True and flush=True are both there.
When to stream: when a person watches the screen, since text that appears at once feels quicker. When a program must read the whole reply before it acts, such as a check on the complete answer, it can wait for the end.
Hands-on lab and basic assignment. Run the script and watch the answer appear piece by piece, like in ChatGPT or Claude:
python stream_reply.py
Notice that the first words appear while the model is still writing the rest. Then change question to one of your own and run it again.
How do I point the same code at another provider or an open model?
Failure first. You point the code at a different provider and one call fails:
Error code: 400 - the request includes a field this provider does not support
This message is illustrative, written for these notes. Nothing is wrong with your code. One hosted provider documents a list of fields it does not support and says that supplying them results in a 400 error. A local model server says it supports a subset of the OpenAI API, and its example uses a placeholder key that its docs mark as required but ignored. “Compatible” is a promise about the format, not a promise that each setting works.
Now the way to reach another endpoint with the same code. Think of a travel plug adapter: the same charger works abroad if you change the part that meets the wall. Or a restaurant chain with one order-slip format: the slip is the same in each branch, while the kitchen behind it differs. An OpenAI-compatible endpoint is a server, hosted or on your own machine, that accepts requests in the Chat Completions format. To reach it with the same code you change three settings: the base URL, where to send; the key, who you are; and the model, which one to use. In the scripts those are LLM_BASE_URL, LLM_API_KEY and LLM_MODEL, at the top of each script or set in your terminal (the terminal wins); setting them in the terminal keeps the key out of the files.
| Where the model runs | What you change | What to check |
|---|---|---|
| A hosted provider with an OpenAI-style API | LLM_BASE_URL, LLM_API_KEY, LLM_MODEL |
Its documentation lists the settings you use |
| A model server on your own machine | LLM_BASE_URL, and a placeholder key if it asks for one |
The model you loaded supports what you call |
| A provider that says it is “mostly compatible” | The same three variables | Which fields it rejects or ignores |
One hosted provider describes its API as mostly compatible with OpenAI’s client libraries, and its guide says to pass its key to the api_key parameter and change the base_url. Write the code once, then test it on the endpoint you will actually use.
What is the model landscape, and how do I choose a model by quality, cost and speed?
Picture a car park with several vehicles and one set of keys that fits each of them: a delivery van, a small hatchback, a rented truck and your own bicycle. The shape of the key is the OpenAI-compatible request; what differs is how far each vehicle goes, what it costs and how fast it is. Models behind that one request format fall into a few groups.
| Choice | Where it runs | What you trade |
|---|---|---|
| Larger hosted model | A provider’s servers | Choose it when your test questions need it; compare its cost and speed |
| Smaller hosted model | A provider’s servers | Choose it for cost or latency, if your test questions still pass |
| Open-weight model on a cloud platform | A cloud service that offers it through the same API | You pick a size from what the platform lists |
| Open-weight model on your laptop | A local model server | Your machine sets the speed and the size you can run |
An open-weight model is a model whose trained weights are published, so more than one company, or you, can host it. That is why the same model can be offered by a cloud platform and by a local server. Amazon Bedrock, for example, documents OpenAI-compatible Chat Completions and Responses APIs, so code written for the OpenAI request format can point at it. Its user guide lists two open-weight models of one family and describes the smaller one as optimized for lower latency and local or specialized use cases and the larger one as optimized for production, general-purpose or high-reasoning use cases. That one sentence shows the trade in miniature: the same family, two sizes, one chosen for speed and one for harder work. Which models a platform lists changes over time, so read its current catalogue instead of trusting a name from these notes.
Failure first. A team picks the highest-ranked model from a leaderboard for a help desk bot. The replies are good, and the monthly bill and the wait for each reply are far higher than the budget. The cause: the leaderboard measured quality on other people’s questions, and said nothing about the team’s cost or speed needs. OpenAI’s own model-selection guide frames the choice the same way: to reduce cost or latency, choose the model and settings that fit the need, and compare candidates on the same task to see the trade-off between quality and cost.
A choice by three questions:
| Question | What to measure | How to measure it in the Copilot |
|---|---|---|
| Quality: is the answer right? | The pass count on your test set | Run the ten test questions through each candidate |
| Cost: what will it cost? | Prompt and completion tokens times the provider’s price | Read the token counts a reply carries, as in session 2 |
| Speed: how long does a reply take? | Seconds to the first word and to the last | Time the call, with streaming on and off |
Run the same ten test questions through two candidates and put the three numbers side by side. Choose the cheapest or fastest one that still passes the questions that matter. A model that scores a little lower on average may still pass the questions your employees actually ask.
How do I build v0, a chat in the terminal that keeps the conversation?
Think of a waiter who writes each order on one running ticket: when you ask for “the same again”, they read the ticket, not their memory. Or a group chat you scroll up through before replying. The model is like a waiter with no memory at all, so v0 keeps the ticket for it. For example, tell v0 in turn one that you work in the Pune office; in turn three ask “which holiday list applies to me?” and it can answer only because turn one is still in the list.
Project step: v0, a chat in the terminal. The script is chat.py. It starts the list with a system message, reads what you type, appends it as a user message, sends the whole list, prints the answer and appends it as an assistant message, until you type quit:
while True:
text = input("You: ")
if text.strip().lower() == "quit":
break
messages.append({"role": "user", "content": text})
response = client.chat.completions.create(
model=os.environ.get("LLM_MODEL", LLM_MODEL),
messages=messages,
)
answer = response.choices[0].message.content
messages.append({"role": "assistant", "content": answer})
print("Assistant:", answer)
Run python chat.py and try this:
You: My laptop will not connect to the Wi-Fi.
You: What was my problem again?
You: quit
The second answer knows your problem because chat.py sends the whole conversation each time. The model itself remembers nothing between calls. If it answers “I don’t know what your problem was”, check that each answer is added to messages before the next question.
Try a second model. Change just the model setting, then run chat.py again with the same messages. Edit the LLM_MODEL line at the top of the script, or in the terminal:
export LLM_MODEL="<another model name>"
On Windows, in PowerShell:
$env:LLM_MODEL = "<another model name>"
Same code, different model. Compare the answers and how fast they arrive.
Three situations from the Copilot, failure first, then cause. The replies are illustrative, written for these notes.
- The forgotten follow-up. The employee’s “when do I have to use them by?” gets “which days?”. The program sent just the latest message, so the model did not see the first question.
- The heavy chat. A long conversation gets heavier with each turn. The whole list is resent each time, so each call carries more and can approach the window limit.
- The call that fails elsewhere. The same code works on one provider and returns a 400 error on another. Compatible is not identical, and a provider may reject a field it does not support.
What this means for your design:
| What you learned | The design decision | Where it shows |
|---|---|---|
| Each request is stateless | Keep the list and send it each turn | messages in chat.py |
| A long list fills the window | Watch the token counts; trim long chats later | The token counts of each reply |
| Roles carry meaning | Standing instructions in system, not in the user’s turn |
The first message in the list |
| Providers differ | Keep three settings in one place at the top; test on the target endpoint | LLM_BASE_URL, LLM_API_KEY, LLM_MODEL |
Practice: explain and sort. First, answer the Model answer question below aloud in four beats before reading it. Second, for each of these four situations, decide whether the cause is a missing history, a role problem, an endpoint difference or the window filling, and write your reason in one sentence:
- A follow-up question gets “which days?”.
- A standing instruction written in the user’s message is overruled by a later user request.
- A call that works on one provider returns a 400 error on another.
- A long chat sends more input with each turn and approaches the window limit.
Flashcards: Chat Completions API, streaming and model choice questions with answers
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What does a Chat Completions request contain?
It contains a list of messages, each with a role and text, plus settings such as the model name. The reply comes back as the next message, written by the assistant.2. What are the roles, and what does each hold in the Copilot?
System holds standing instructions, user holds the employee's words, and assistant holds earlier replies sent back to the model. Developer is OpenAI's newer name for instructions, which replaces system for its o1 models and newer.3. Why does the program send the whole history each turn?
Each request is independent and stateless, so the model sees what is in the request. The list of earlier user and assistant messages is the memory, which is why a chat grows heavier each turn and fills the context window.4. What does streaming change, and when do you use it?
It shows the reply in chunks as the model writes it, instead of in one block at the end. It changes when the employee sees the words, not what they are. Use it when a person watches the screen; wait for the end when a program must check the whole reply first.5. What do you change to reach another provider with the same code?
The base URL, the key and the model name, held in `LLM_BASE_URL`, `LLM_API_KEY` and `LLM_MODEL`. Compatible is a promise about the format, so test on the endpoint you will use, since a provider may reject or ignore a field.6. What is an open-weight model, and where can you run one?
It is a model whose weights are published, so it can be hosted by more than one party. You can reach one through a cloud platform that offers it behind an OpenAI-compatible API, or run one on your own machine with a local model server.7. How do you choose a model by quality, cost and speed?
Run your own test questions through two candidates. Compare the pass count for quality, the prompt and completion tokens times the price for cost, and the time to the first and last word for speed. Pick the cheapest or fastest one that still passes the questions that matter.8. Why is a leaderboard not enough to choose a model?
It measures quality on other people's questions and says nothing about your cost or speed limits. OpenAI's own guide frames the choice as a trade-off between quality, cost and latency to be tested on the same task.9. Why can the same code fail on another provider?
Compatible is a promise about the format, not that each setting works. A provider may reject a field it does not support, for example with a 400 error, so test on the endpoint you will use.Model answer: how does a chatbot remember the conversation, and how would we switch it to a cheaper model?
The core question this session answers: “An interviewer asks: how does the chatbot you built remember the conversation, and how would you switch it to a cheaper model?”
Four beats the interviewer listens for, in this order: history is resent · roles and streaming · same call, other endpoint · choose by quality, cost and speed. Skip any of these four and the answer is incomplete.
- History is resent: each request is stateless, so the program keeps the list of messages and sends it with each turn.
- Roles and streaming: standing instructions go in the system (or developer) message, and streaming shows the reply as it is written.
- Same call, other endpoint: change the base URL, the key and the model, then test on the endpoint you will use.
- Choose by quality, cost and speed: run your own test questions through two candidates and compare.
The full answer, said aloud:
(History is resent) “The model has no memory between requests. My program keeps the list of messages, the system message, each user question and each assistant reply, and sends the whole list with each turn. That is the conversation.
(Roles and streaming) The standing instructions sit in the system (or developer) message so a later user turn does not overrule them, and I stream the reply so the employee sees words as they are written.
(Same call, other endpoint) Because many providers, cloud platforms and local servers accept the same request format, switching means changing three settings: the base URL, the key and the model name. I would still test on the endpoint I will use, because compatible does not mean identical.
(Choose by quality, cost and speed) To pick the cheaper model I would run my ten test questions through both, compare the pass count, the token counts times the price, and the time per reply, and choose the cheapest one that still passes the questions that matter.“
Missing a beat is a failure state. If you stop before the choice, the follow-up is how you know the cheaper model is good enough. Practise until the four come out without prompting.
Why this matters: where do you use chat history, streaming and model choice at work?
In a debugging session: a bot that forgets the last question is often a program that did not send the history. Knowing the request is stateless takes you straight to the cause.
In a cost review: the question “can we use a cheaper model?” has an answer you can defend: a test set, three numbers per candidate, and a decision rule.
In the rest of this course: session 5 changes what the reply looks like so a program can read it; sessions 6 to 8 write and test the main prompt; session 11 trims long chats and keeps memory properly; and module 7 returns to model choice when several agents each use a model.
After this unit: what is the assignment and what comes next?
Assignment. Stream a reply, send the same question to two models and compare quality, cost and speed, and run v0 for a three-turn chat. The brief for the project work, v0, is the project file. Keep v0: the next sessions change what it returns and what it can do.
What’s next. Session 5 covers replies a program can trust: why free text breaks a program, Structured Outputs with a schema described in Pydantic, handling a refusal, and function calling, where the model asks your code to run a function.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.