Courses / Agentic AI from Scratch (Python)
Session 33 of 40
Seeing inside the app with Langfuse: traces, cost and timing
Notes
How to Trace a LangChain or LangGraph App with Langfuse: Find the Slowest Step and the Cost
Agentic AI Foundations · Session 33
Before you start: what do I need before I trace an app with Langfuse?
You need the multi-agent Copilot from sessions 29 to 32 and the code folder for this session open in your terminal. The folder adds three new scripts, code/tracing.py, code/trace_chain.py and code/trace_copilot.py, and changes code/multi_agent.py and code/ask_copilot.py. You also need a Langfuse project: a free cloud account, or your own Langfuse server. From the project settings you get a public key and a secret key, and you know the address of your region or server. Set them in your terminal as LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST (the course’s name for the address; code/tracing.py hands it to the Langfuse library as LANGFUSE_BASE_URL, the name the library reads), the same way you set LLM_API_KEY, and keep the secret key out of files you share. Install the libraries with pip install mcp langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters langfuse, set your model settings as in earlier sessions, then run python code/build_index.py and python code/make_tickets_db.py. The course uses Langfuse Python SDK version 5, whose LangChain handler needs langchain-core 1 or later. Have paper or a notes app ready for the flashcards, the model answer and your notes/traces.md.
Summary: what is tracing, and what does Langfuse show me about my app?
When an answer is slow or wrong, a program that prints only its final reply gives you nothing to go on. A trace is the recording of one request, from the user’s question to the final reply, and each step inside it, such as a model call, a tool call or a search, is an observation with its own input, output and time. Langfuse collects these recordings through a callback handler you pass into your LangChain or LangGraph call; in the Copilot, make_config() in code/tracing.py builds that call’s config. Langfuse also tracks the usage and cost of each model call and groups the traces of one conversation into a session. In this session you trace one chain, then the whole Copilot, and find its slowest step and the token counts behind its cost.
Key concepts: what do the tracing terms mean?
| Term | Definition |
|---|---|
| Trace | The recording of one request or operation, such as one chatbot interaction from the user’s question to the final response. |
| Observation | One step inside a trace: a model call, a tool call, a retrieval step, and so on. Observations can be nested. |
| Session | A group of traces that belong to the same user interaction, such as the thread of a chat. |
| Callback handler | A LangChain object that is told about each step of a run; Langfuse’s handler records those steps as a trace. |
| Config | The dictionary of settings handed to a LangChain call, which carries the callback handler and the metadata. |
| Latency | How long a step takes; Langfuse stores it on each observation. |
| Token | A piece of text a model reads or writes; usage is counted in input and output tokens. |
| Cost | The token counts multiplied by the model’s price per token for each usage type. |
| Model definition | A record in Langfuse that holds a model’s name and its prices, so cost can be worked out. |
| Flush | Sending queued events to Langfuse now, so a short program does not exit with traces still waiting. |
Explanations: how does Langfuse tracing work?
What is a trace, and why do I need one?
Think of a flight recorder. After an incident you replay what each instrument said, in order. Or think of a parcel tracking page: each stop has a time, so you can see where the delay was. A trace does that for one question. Langfuse describes a trace as a single request or operation and observations as the individual steps inside it, such as model calls, tool calls and retrieval steps.
Now the failure first. An employee reports that the Copilot gave a wrong carry-over answer yesterday. Your program printed just the reply, and the reply is gone. You can add print statements, but they show only what you thought to print, as loose text that is not tied to a step or a run. With a trace saved, anyone who was not there can open yesterday’s run and see what went in and what came out at each step.
For the multi-agent Copilot of code/multi_agent.py, one question has this shape. The model chooses which agents to call, so the tree you see in Langfuse can differ from this sketch:
trace: the Wi-Fi question (user: priya, session: copilot-priya)
'-- supervisor
|-- model call
|-- ask_policy_agent
| |-- model call, search_policy, model call
| '-- model call (the typed report)
|-- ask_ticket_agent
| '-- model call, create_ticket (the MCP call)
'-- model call (the final reply)
Another picture is a receipt with line items: the total is one number, but the lines show where it came from. Langfuse also has sessions, which group traces that belong to the same user interaction; a thread in a chat is the usual example. In code/trace_copilot.py the session id is copilot- followed by the user’s name, so one person’s questions sit together; when a script passes no session id, as code/ask_copilot.py does, make_config() uses user- followed by the name.
How do I add Langfuse tracing to a LangChain or LangGraph app?
First the failure. You run the Copilot, open Langfuse and find nothing: no trace, no steps, no times. Either the handler was left out of the call, or the script ended before its queued events were sent.
Picture a note-taker sitting beside the program. If nobody invites the note-taker in, nothing is written down; and if the program leaves before the notes are handed over, the notes do not arrive. The handler is that note-taker, and the config is the envelope of settings you hand to the call, so the note-taker rides inside it.
Now the rule. In Langfuse Python SDK version 5 the LangChain callback handler is imported from langfuse.langchain. Langfuse’s documentation says instrumenting LangGraph follows the same pattern as LangChain: you pass the handler to the invocation. code/tracing.py wraps that in one function:
def make_config(user_id="demo-user", session_id=None):
if not tracing_enabled():
# prints one notice, then returns an empty config
return {}
connect()
from langfuse.langchain import CallbackHandler
return {
"callbacks": [CallbackHandler()],
"metadata": {"langfuse_user_id": user_id,
"langfuse_session_id": session_id or f"user-{user_id}"},
}
Three details matter. First, the handler travels in the config handed to the call. Second, trace attributes such as the user id and session id go in metadata fields in that config, using the key names langfuse_user_id and langfuse_session_id. Third, the Langfuse SDKs queue and batch events in the background, which works without extra effort in a long-running service; a short-lived program has to shut down Langfuse, or flush it, so the queued events are sent before the program exits. That is the job of flush() in code/tracing.py, which the three scripts that run questions call last.
The file reads its three settings, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST, with the terminal winning over any value typed into the file. It then sets LANGFUSE_BASE_URL to the same address, because that is the name the Langfuse library itself reads. When either key is missing, make_config() prints one notice and returns an empty config, so every script still runs, untraced. That is also why a missing key does not stop the Copilot: an empty config is the same as passing nothing.
How does the config reach the whole Copilot?
Picture a clipboard passed down a chain of staff: the supervisor holds it, and hands it to each specialist, who hand it to each model call. In code/multi_agent.py, session 33 changes one function:
def ask(question, config=None):
result = supervisor.invoke(
{"messages": [{"role": "user", "content": question}]},
config=config)
return text_of(result["messages"][-1])
With config=None the Copilot behaves as before. With the config from make_config, the handler placed in it travels down into the supervisor, both specialists and every model call, so one question becomes one trace. code/ask_copilot.py now reads --user NAME and calls multi_agent.ask(question, make_config(user)), then flush(). Try python code/ask_copilot.py --user marcus "What is the status of ticket T-1042?", then look for marcus’s trace in Langfuse. With keys set it is traced; without them it prints the notice and runs untraced. If the supervisor wants a new ticket, this script asks you for a yes or a no.
How do I trace one chain, and then the whole Copilot?
Start small. python code/trace_chain.py runs one chain, a prompt, a model and a text parser joined with |, once, passing make_config("priya", "trace-chain"). In Langfuse open Tracing and click the newest trace. Look for one model call, with its input, its output and its token counts.
Then run the Copilot: python code/trace_copilot.py --user priya. The script sends three questions through multi_agent.ask: a leave question, a question about ticket T-1043, and the Wi-Fi question that needs both agents. It sets multi_agent.approver to a function that always says no, so it never waits for you and never writes to tickets.db. Each question is one trace, and all three sit in one Langfuse session named copilot-priya. The script prints only the total time of each question; the time per step is in Langfuse.
How do I find the slowest step in a trace?
Open a trace and read its steps in order, with the time on each. A table of what to look at:
| What to open | What to read |
|---|---|
| The newest trace | The run’s total time at the top |
| The tree under the supervisor | Each step’s own time, in order |
| A model call | Its input, its output and its token counts |
| The MCP tool call | One step: the server is a separate process |
Compare the longest step with the whole run, and look there first. Speeding up a short step would change nothing you could notice. The tool call to the MCP server shows as one step, and what happens inside the server is not in the trace; the server’s own record is the audit trail you meet in session 36. This is the habit the trace gives you: find where the time went before you change anything, and notice that a printed total, like the one trace_copilot.py shows, cannot tell you which step was slow.
How does Langfuse work out what a question costs?
Think of a receipt again. Langfuse tracks the usage and cost of each model call, and for most users cost tracking works without extra configuration. When cost is not sent along with the call, Langfuse infers it: it matches the call’s model to a model definition that holds a price for each usage type and multiplies the prices by the usage. For a self-hosted or fine-tuned model that is not on Langfuse’s list, you can add your own model definition with prices.
The arithmetic is plain: the cost of a call is its input tokens divided by a million times the input price per million, plus its output tokens divided by a million times the output price per million. Read the token counts off the model calls in your trace, take the prices from the model definition, and write down the cost of the Wi-Fi question and of the leave question in notes/traces.md. A model name and a token count are not enough on their own: the model also needs a matching definition with prices before a trace turns into a bill.
What goes wrong when you cannot see inside an app?
Three situations show the gap.
- A question takes a long time and nobody knows why. No timing was recorded per step, so the slow call looks like the fast ones. Fix: trace each run and read the step times.
- The monthly bill is double the estimate and the team argues about why. Cost was known as a total and no finer. Fix: read cost per step and per user from the traces.
- A short test script ends and its traces do not appear. The SDK sends events in the background, and the program exited first. Fix: call
flush()before the script ends.
| What you learned | The design decision | Where it shows |
|---|---|---|
| A trace shows each step’s time | Trace each run, grouped by user and session | make_config() in code/tracing.py |
| Cost is tokens times price | Check the model has a price in Langfuse | Langfuse model definitions |
| Events are sent in the background | Flush in a short program | flush() in code/tracing.py |
| Slow steps hide in totals | Find the slowest step before editing | Your notes/traces.md |
Flashcards: test yourself on Langfuse tracing
Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.
1. What is a trace?
The recording of one request, from the user's question to the final reply. Like a flight recorder, it lets you replay what happened at each step after the fact.2. What is an observation?
One step inside a trace, such as a model call, a tool call or a retrieval step. Observations can be nested, so a run is a tree of steps.3. What is a session in Langfuse?
A group of traces that belong to the same user interaction, such as the thread of a chat. In `code/trace_copilot.py` the session id is `copilot-` plus the user's name, so one person's questions sit together; with no session id given, `make_config()` uses `user-` plus the name.4. Where does the Langfuse callback handler come from in SDK version 5?
From `langfuse.langchain`: `from langfuse.langchain import CallbackHandler`. `make_config()` puts it in the `callbacks` list of the call's config, the same way for LangChain and LangGraph.5. How do you attach a user and a session to a trace?
Put `langfuse_user_id` and `langfuse_session_id` in the `metadata` of the call's config, which `make_config(user, session)` does for you. That lets you find one person's conversation later.6. What does make_config return when the Langfuse keys are missing?
An empty config, after printing one notice. Passing an empty config is the same as passing none, so the script runs untraced instead of failing.7. Why does a short script sometimes leave no traces in Langfuse?
The SDK queues and sends events in the background. If the program exits first, queued events may not have been sent, so call `flush()` last, as the scripts that run questions in this session do.8. How do you find where a slow run lost its time?
Open the trace and compare each step's time with the whole run. The step with the biggest share is where to look first, before you change anything; the total printed by `trace_copilot.py` cannot name it.9. How does Langfuse work out cost?
It tracks usage on each model call, then matches the model to a model definition with a price per usage type and multiplies prices by usage. Cost is tokens divided by a million times the price per million.10. What do you do for a model Langfuse has no price for?
Add a custom model definition with your own prices, which is useful for self-hosted or fine-tuned models that are not on Langfuse's list of maintained models.11. Why are print statements not enough?
They show only what you thought to print, as loose text not tied to a step or run. A saved trace lets someone who was not there see what each step received and returned.Model answer: how do I explain LLM app tracing in an interview?
The core question this session answers: “A teammate asks: an employee says the Copilot was slow and gave a wrong answer yesterday. How do you find out what happened and what a question costs?”
Four beats a teammate listens for, in this order: record each run · read the steps in order · find the slowest and the costliest · find the right run. Skip any of these four and the answer is incomplete.
- Record each run: a trace, with the handler in the call’s config.
- Read the steps in order: inputs, outputs and times.
- Find the slowest and the costliest: step times, token counts and prices.
- Find the right run: session and user ids.
The full answer, said aloud:
(Record each run) “Each question is sent to Langfuse as a trace. I pass the callback handler in the call’s config, and I flush before a short program exits.
(Read the steps in order) To investigate, I open the trace and read each step’s input, output and time, in order, so I can see where the answer went wrong.
(Find the slowest and the costliest) I compare step times to find the slowest step, and I use token counts and the model’s prices to see what a question costs. I look before I change anything.
(Find the right run) Traces carry a session and a user id, so yesterday’s question is a search away instead of a guess.“
Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and you are asked how you would find that one employee’s run among thousands. Practise until the four come out without prompting, then stop talking and let the questions come.
Why this matters: why does tracing matter at work?
In a conversation with a teammate or manager: whoever is on call for an assistant is asked which step made an answer slow or wrong. Pointing at a trace instead of guessing is the difference between an answer and an opinion.
On the job: product owners need a cost per question to plan a budget, and support leads need to open a complaint’s exact run. Both come from the same traces.
In the rest of this course: session 34 scores the same kind of test questions automatically, and session 36 writes an audit record for each answer. The capstone’s monitoring work builds on this habit.
After this unit: what do I do after this lesson, and what comes next?
Assignment. Run python code/trace_chain.py and find its model call and token counts in the trace. Project step. Write code/tracing.py, code/trace_chain.py and code/trace_copilot.py (the build prompts are code/prompts/1-write-tracing.md, code/prompts/2-write-trace-chain.md and code/prompts/3-write-trace-copilot.md), run python code/trace_copilot.py --user priya, and write the slowest step of each question and the cost of the Wi-Fi and leave questions in notes/traces.md. The brief for the project work is the project file. Next session: testing an AI app with a model that grades the answers.
Want help building this with someone?
Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.
Ask about mentor-led trainingTechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.