Courses / Agentic AI from Scratch (Python)

Session 37 of 40

Ship it: a chat page with Streamlit, an API with FastAPI, deployment and production monitoring

Notes

How to Ship an LLM App: A FastAPI API, a Streamlit Chat Page, Deployment and Monitoring

Agentic AI Foundations · Session 37

Before you start: what do I need before I build an API and a chat page for my LLM app?

You need the code folder for this session open in your terminal, and every command below runs from inside it, as the folder’s README.md does. It holds the Copilot as you left it after session 36, plus seven new scripts: code/settings.py, code/monitoring.py, code/service.py, code/api.py, code/api_client.py, code/chat_page.py and code/test_api.py. Two older scripts changed a little: code/audit.py gained a decision called awaiting_approval, and code/check_secrets.py now checks API_KEYS too. Install the new libraries with pip install fastapi uvicorn streamlit httpx pytest, and set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL and EMBED_MODEL in your terminal as in earlier sessions. Then run python build_index.py and python make_tickets_db.py. You do not need a cloud account: the service stays on your own machine, and the parts about where it could run are explained as ideas only. Have paper or a notes app ready for the flashcards and the model answer.

Summary: how do I put an API and a chat page in front of an AI assistant?

Until now you were the one running the Copilot, from a terminal. An API lets another program ask the same Copilot in the same way, and a chat page lets a colleague type a question in a browser. FastAPI builds the API from functions and type hints, and Streamlit builds the page from a short Python script. Two decisions matter more than the libraries. The server decides who is calling and in which role, from a key it looks up in its own table, and never from anything the caller types. A new ticket takes two calls: /ask only proposes it, and a second call, /approve, from the same person, is the only thing that writes it. Once it is reachable you also decide where it could run and which records you read to know it is behaving.

Key concepts: what do the terms for shipping an LLM app mean?

Term Definition
Web API A doorway that lets one program ask another for something and get a reply in an agreed shape.
FastAPI A Python library that builds a web API from functions and type hints.
Route One address of the API that does one job, such as /ask.
API key A long random word a caller sends to prove it holds access; the server looks it up to find the user and the role.
Streamlit A Python library that turns a script into a web page; the script runs again from top to bottom when something on the screen changes.
Session state Streamlit’s store for variables that must survive reruns, kept for each user session.
Thread id A value that says which conversation a message belongs to.
Request log One line for each API call: the route, status, decision and seconds, with no question or answer text.
Deployment Making an application available to its users, normally by putting it on a remote machine with a server program.
Monitoring Watching a live system through its traces, records and numbers, so trouble is noticed early.

Explanations: how do the API, the chat page and the records fit together?

What is a web API, and how does FastAPI build one?

Think of a restaurant hatch: you hand in an order in the agreed form and a plate comes back, and you never walk into the kitchen. A request that does not fit the form is turned back at the hatch. FastAPI declares the form with Pydantic models: the request body is read as JSON, converted and validated, and invalid data gets a clear error that shows where the mistake was. A reply type works the same way on the way out.

code/api.py has three routes:

Route What it does Needs a key
GET /health Says the API is up, whether tracing is on, and whether the index and the ticket database are built No
POST /ask Takes a question and gives back an answer; a ticket is only proposed Yes
POST /approve Takes a person’s yes or no on a proposed ticket; the only route that can write one Yes

The routes are plain def functions, not async def. The model call blocks, and FastAPI runs a path function declared with a normal def in a thread pool, so one slow question does not freeze the whole server. Each route checks the request and hands the work to code/service.py, which keeps the routes thin and lets a test swap in a fake Copilot. FastAPI also builds an interactive page at /docs from your type hints, where you can try each route.

How does the server know who is calling and which role they hold?

The failure first. Suppose the request had a role field and the Copilot believed it. Anybody could type "role": "hr" and ask for the HR-only policy. In session 35 the role was set in code and never chosen by the model; the same rule holds for a caller.

So the request carries a key, and the server looks the key up in its own table. code/settings.py builds that table from the API_KEYS setting, one entry for each person, written key:user:role and separated by commas. The role must be one the Copilot knows, employee or hr at this stage.

def current_caller(x_api_key=Header(None)):
    caller = settings.API_KEYS.get(x_api_key or "")
    if caller is None:
        raise HTTPException(401, "Missing or unknown API key.")
    return caller      # user and role, from the server's table

The request model has two fields, the question and a thread id. There is no user and no role in it, so a role typed into the body is ignored. A missing or unknown key gets a 401. A real deployment would use the company’s own sign-in instead of shared keys, and the sign-in would tell the server who the person is. The keys are secrets: set API_KEYS in your terminal, never in a file. python settings.py shows how many callers it found and SET or MISSING for each secret, never a key.

Why does a ticket take two calls?

A ticket is an action with a side effect, and in session 27 you made a person say yes before one was created. An API has no keyboard to ask on, so the yes has to become a second call. In code/service.py the Copilot is given an approver that only records the proposed title and says no. POST /ask therefore returns the status needs_approval, a pending_id and the proposed ticket_title, and no ticket exists yet.

A second call, POST /approve, carries the pending_id and a yes or no. It writes a ticket only for a pending_id the Copilot made, only once, and only when the person approving is the person who asked. Another person’s key gets a 403, a used or unknown id gets a 404, and a no writes nothing. The pending tickets are kept in memory, so restarting the API forgets them. The alternative, keeping ticket creation off the API, is safer but hides the approval step; if you prefer it, the README says to delete the /approve route.

How do I build a chat page with Streamlit?

Think of a whiteboard that is wiped and redrawn when someone acts. By default Streamlit runs your whole script again, from top to bottom, whenever something on the screen must change. Anything the page must remember has to be written in session state, a store for variables that survive reruns, kept for each user session. A browser reload resets the connection and the state with it.

The failure first: a beginner’s page keeps the conversation in an ordinary variable, the user types, the script reruns, and the conversation vanishes. The fix is in code/chat_page.py, which creates its memory only when it is not there yet:

if "thread_id" not in st.session_state:
    st.session_state.thread_id = str(uuid4())
    st.session_state.messages = []
    st.session_state.pending = None

The not in check matters: without it every rerun would wipe the conversation. The thread id is made once for each tab, so two tabs are two conversations. st.chat_input shows a box pinned to the bottom of the page and returns the typed text, or None when nothing was sent, and st.chat_message draws one message. The page takes a key in a password box in the sidebar and sends it in the X-API-Key header through code/api_client.py, whose two functions, ask_api and approve_api, call the API with httpx. The page never sends a name or a role, and never imports the Copilot. When a ticket is proposed it shows two buttons, Approve and Decline, and each one calls /approve. Streamlit builds the page from one short script, which keeps the whole session in Python.

What records show that a running service is healthy?

Three records describe the same question from three sides.

Record Where What it answers
The trace Langfuse, if you set its keys The steps of one run, grouped by thread id
The audit trail audit_trail.jsonl Who asked, what the Copilot did, who approved
The request log requests.log The route, status, decision and seconds of each call

A Langfuse trace represents one request, and Langfuse sessions group traces that belong to the same interaction, such as a thread in a chat. code/monitoring.py tags each trace with the person and the thread id. A proposed ticket shows awaiting_approval in the audit trail, and the call that approves it adds a second record with ticket_created and the person’s name. The request log holds no question text and no answer text, only facts about the call.

python monitoring.py counts the request log by status and gives the average and slowest time. A rise in error statuses, or in the average time, is the number to put an alarm on. A CloudWatch alarm, for example, watches one metric over a period and acts when it crosses a threshold. Langfuse can also hold a score for each trace, so you can run session 34’s grader on a sample of live answers and watch the average.

Where could the service run, and who restarts it?

FastAPI describes deploying as making the API available to users, normally on a remote machine with a server program, and it says you can run the server yourself or use a cloud service that does part of the work. This session keeps the service on your machine. These are the choices you would weigh later:

Choice Who looks after what
Your laptop You. Good for learning and demos.
A server you run You handle restarts after a crash, HTTPS, and how many copies run.
A managed web service The provider runs it. You supply the code and the settings.
A function service The provider runs a small piece of code for each event. A poor fit for a program that loads an index.

Whatever the choice, the settings and keys come from the host’s own secret store and not from a file. HTTPS is normally provided by a component outside the application server, called a TLS termination proxy. FastAPI advises that whatever restarts a crashed application is an external component, because a crashed app cannot restart itself. And a program that loads a model or an index uses server memory, and several worker processes normally do not share it, so each extra worker loads its own copy. Session 40 turns these into a short plan.

What goes wrong when an API has no care taken?

  1. A user reads the HR policy by typing a role. The server trusted a role in the request. Fix: look the role up from the key, and keep it out of the request.
  2. A ticket appears that nobody approved. Creation ran inside /ask. Fix: /ask only proposes, and /approve writes once for the person who asked.
  3. Errors double overnight and nobody knows. Nothing watched the request log. Fix: count by status, and put an alarm on errors and time.
What you learned The design decision Where it shows
A caller can type anything Decide the role on the server from a key code/api.py, code/settings.py
A yes is a person’s Two calls, and one writes once code/service.py
A page reruns Keep memory in session state code/chat_page.py
A running service needs eyes Trace, audit trail, request log code/monitoring.py, code/audit.py

Flashcards: test yourself on shipping an LLM app

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. What are the three routes of the Copilot's API? `GET /health` says it is up and needs no key. `POST /ask` takes a question and gives back an answer, only proposing a ticket. `POST /approve` takes a person's yes or no and is the only route that can write a ticket.
2. Why is the role not a field of the request? A caller could then type `"role": "hr"` and read the HR-only policy. The server looks up the role from the caller's key in its own table, so the caller cannot choose it.
3. What does a request with no key, or a wrong key, get? A 401. The key must be one the server knows from `API_KEYS`.
4. Why does a ticket take two calls? `/ask` only proposes the ticket, so no ticket exists yet. `/approve`, from the person who asked, is the call that writes it, so the yes is still a person's.
5. What stops a second approval, or another person's approval? The pending ticket is removed when it is used, so a second call gets a 404. Another person's key gets a 403, because only the person who asked may approve.
6. Why does a Streamlit page need session state? By default Streamlit reruns the whole script when something changes, so an ordinary variable starts empty each time. Session state keeps the conversation, the thread id and a waiting ticket across reruns of one tab.
7. Why do two browser tabs get two conversations? Each tab has its own session state, and the page makes a new thread id once for each tab.
8. What are the three records of a running service? The trace in Langfuse, the audit trail in `audit_trail.jsonl` and the request log in `requests.log`.
9. What goes in the request log, and what is left out? The route, status, decision, user, role, thread id and seconds. It holds no question text and no answer text.
10. Why does something outside the app have to restart it? A crashed app cannot restart itself, so the restart belongs to a component outside it.

Model answer: how do I explain shipping an LLM app in an interview?

The core question this session answers: “A colleague asks: how can another system use the Copilot safely, and how do we know it is behaving once people use it?”

Four beats a colleague listens for, in this order: a doorway with a key · the server decides the role · a person approves actions · records you can read. Skip any of these four and the answer is incomplete.

  1. A doorway with a key: the API has /health, /ask and /approve, and a key on the calls that act.
  2. The server decides the role: a key lookup, with no role in the request.
  3. A person approves actions: two calls, and the second writes once.
  4. Records you can read: trace, audit trail and request log.

The full answer, said aloud:

(A doorway with a key) “The Copilot sits behind a small API with three routes: a health check, an ask route and an approve route. Every call except the health check carries a key.

(The server decides the role) The server looks the key up in its own table to find the user and the role. The request body carries no role, so nobody can ask to be an HR person.

(A person approves actions) Asking only proposes a ticket. A second call from the same person writes it, once, so the yes is still a person’s.

(Records you can read) Each answer leaves a trace, an audit record and a request-log line without the text. I would put an alarm on rising errors and slower replies.“

Missing a beat is a failure state. The follow-up lands where you went quiet: skip the second beat and you are asked how the server knows the caller is allowed. Practise until the four come out without prompting, then stop talking and let the questions come.

Why this matters: why does shipping an LLM app matter at work?

In a conversation with a teammate or manager: the first questions after a demo are who can call it, what it can change, and who is told when it breaks. A key lookup, a two-call approval and three records are the answers.

On the job: an API is how a portal, a script or another team’s tool reaches your assistant. The doorway is where the access rule has to hold, so it is also where a reviewer looks first.

In the rest of this course: session 38 runs this same API on four different data packs, and session 40 turns the notes on where it could run into a deployment plan.

After this unit: what do I do after this lesson, and what comes next?

Assignment. Run pytest test_api.py, start the API with uvicorn api:app --host 127.0.0.1 --port 8000, call /health, then ask as the employee and approve the ticket. Make four tries and note the status code of each: the same approval again, no key, approval with the other person’s key, and a role field added to the body. Open the chat page with streamlit run chat_page.py in two tabs. Project step. Write code/settings.py, code/monitoring.py, code/service.py, code/api.py, code/api_client.py, code/chat_page.py and code/test_api.py (the build prompts are code/prompts/1-write-settings.md to code/prompts/7-write-test-api.md). The brief for the project work is the project file. Next session: the capstone setup, with the same Copilot on four different data packs.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.