Courses / Agentic AI from Scratch (Python)

Session 32 of 40

Frameworks compared, and Milestone 3: a multi-agent Copilot over MCP

Notes

LlamaIndex vs Agno vs CrewAI vs AutoGen: Which AI Agent Framework Should You Choose?

Agentic AI Foundations · Session 32

Before you start: what do I need before I compare AI agent frameworks?

You need the code folder for this session open in your terminal. It holds the Copilot as you left it after session 31 (the supervisor in code/multi_agent.py, and the MCP ticket server in code/ticket_server.py with its permission check in code/governance.py) plus two new files: code/llamaindex_policy.py, which answers a policy question with LlamaIndex, and code/check_milestone3.py, which runs the six Milestone 3 checks. Install the libraries with pip install mcp langchain langchain-openai langchain-chroma langchain-community langchain-text-splitters llama-index-core llama-index-llms-openai-like llama-index-embeddings-openai-like. Set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL and EMBED_MODEL as in earlier sessions, then build the policy index and the ticket database with python code/build_index.py and python code/make_tickets_db.py. Only LlamaIndex is run in this lab; Agno, CrewAI and AutoGen are read about in these notes. Have paper or a notes app ready for the flashcards and the model answer.

Summary: which AI agent framework fits which job, and where does Milestone 3 come in?

The Copilot so far uses LangChain and LangGraph. Several other frameworks solve overlapping problems with different choices about what you write by hand and what the framework decides. LlamaIndex centres on documents, Agno on an agent made from a model and tools, CrewAI on a crew of roles, and AutoGen, now archived, has a maintained successor in Microsoft Agent Framework. This session explains each in turn, puts them beside LangGraph in one comparison table, has you run one of them, LlamaIndex, on the same policy question the Copilot already answers by hand, and closes the module with Milestone 3: six checks that the multi-agent Copilot over MCP works. The rule to carry away is simple: choose by the job, check that the framework is still maintained, and test before you switch.

Key concepts: what do the framework terms mean?

Term Definition
Framework A library that decides part of how an agent is built, such as how steps are described and how tools are called.
LangGraph A low-level orchestration framework where you define workflows as graphs of shared state, nodes and edges.
LlamaIndex A Python toolkit for building agents over your own data, centred on indexing documents and querying them.
Index A structure built from documents so a question can find the right passages quickly.
Query engine The LlamaIndex object that takes a question, finds passages in an index and answers.
Agno A Python framework for building agents, combining them into teams and workflows, and running them as services.
CrewAI A framework that organises agents by role, with tasks and a crew that runs them.
Crew In CrewAI, a group of agents working together to complete a set of tasks.
AutoGen A Microsoft multi-agent framework whose repository is archived and no longer maintained.
Microsoft Agent Framework The direct successor to AutoGen, created by the same teams, with agents and workflows.
Milestone 3 The check that the multi-agent Copilot over MCP meets six stated conditions.

Explanations: how do LlamaIndex, Agno, CrewAI and AutoGen differ?

How do I choose between agent frameworks?

Picture a toolbox. A hammer and a screwdriver both fasten things, but a screw wants one of them. Or picture moving house: a van you drive yourself gives you control, a removals crew with set roles gives you speed, and a courier does one thing well. Frameworks differ in the same way: each makes a different choice about what you write by hand and what it handles for you. The cause of most mix-ups is picking one by familiarity instead of by the job.

Start with the baseline you already know. LangGraph is a low-level orchestration framework for long-running, stateful agents: you define workflows as graphs of shared state, nodes and edges, and can mix fixed steps with model-driven steps in the same graph. It focuses on durable execution, streaming and human-in-the-loop, and it does not prescribe prompts or agent architecture. That is why the Copilot’s routes, saved state and approval step live there.

What is LlamaIndex, and when should I use it?

Think of a librarian with a card index: you ask a question and the librarian finds the right shelf first. Or a research assistant given a folder of files. LlamaIndex describes itself as a Python toolkit for building LLM-powered agents over your data, and as the original open-source toolkit for retrieval and agents. Its strength for the Copilot is turning the policy folder into an index and querying it.

documents = SimpleDirectoryReader(
    str(POLICY_FOLDER), required_exts=[".md"]).load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

response = query_engine.query(question)
for found in response.source_nodes:
    print(found.node.metadata["file_name"])

This is the heart of code/llamaindex_policy.py. SimpleDirectoryReader loads the files in the folder, VectorStoreIndex.from_documents builds the index in memory (so there is no policy_index/ folder to build first), and as_query_engine turns it into something you can ask. The script then prints the answer and the files the answer came from. With no embedding model configured, the index falls back to OpenAI embeddings, so for an OpenAI-compatible endpoint you set both models yourself. The file does it first, before it builds the index:

Settings.llm = OpenAILike(
    model=os.environ.get("LLM_MODEL", LLM_MODEL),
    api_base=os.environ.get("LLM_BASE_URL", LLM_BASE_URL),
    api_key=os.environ.get("LLM_API_KEY", LLM_API_KEY),
    is_chat_model=True, temperature=0)
Settings.embed_model = OpenAILikeEmbedding(
    model_name=os.environ.get("EMBED_MODEL", EMBED_MODEL),
    api_base=..., api_key=...)

OpenAILike means any provider that speaks the OpenAI format. Run python code/llamaindex_policy.py, and then python code/llamaindex_policy.py "How many days a week can I work from home?". Look for the answer and the policy files listed as its sources. Then open the file, count the lines that do the work, and compare them with code/load_policies.py, code/build_index.py, code/search.py and code/helpdesk_rag.py. LlamaIndex decided how to cut the files into pieces, how to search them and how to word the prompt. What you gave up is the quote check, the “I don’t know” rule and the control of the search that your own files have.

LlamaIndex also has agent workflows: its documentation lists AgentWorkflow, where you declare a set of agents and it manages the hand-offs, and an orchestrator pattern where sub-agents are exposed to an orchestrator agent as tools. That is the same supervisor shape you built in session 31.

Choose it when the job is mostly answering from documents.

What is Agno, and when should I use it?

Think of a meal kit: the parts arrive in one box and you add your own. With LangGraph you design the kitchen, drawing each station and the path between them. Agno builds an agent from a model, instructions and tools in one object, and its documentation says any Python function can be a tool. The sketch below is for comparison only: the lab does not run Agno, and policy_text, model_name, api_key and base_url stand for helpers you would write. It describes itself as a platform to build agents, combine them into teams and workflows, and run them as services.

from agno.agent import Agent
from agno.models.openai.like import OpenAILike

def search_policy(question: str) -> str:
    """Search the company policies and return the matching text."""
    return policy_text(question)

agent = Agent(
    model=OpenAILike(id=model_name(), api_key=api_key, base_url=base_url),
    tools=[search_policy],
    instructions="You are HelpDesk Copilot for Fernhill Systems. Name the policy section.",
)
agent.print_response("How many unused leave days can I carry over?")

OpenAILike is imported from agno.models.openai.like and takes an id, an API key and a base URL, so the agent can use any provider that implements the Chat Completions format. print_response prints the answer in the terminal and is meant for development. Choose it when you want a tool-using agent or a small team quickly.

What is CrewAI, and when should I use it?

Think of a newsroom: a reporter gathers the facts and an editor writes the piece. Or a kitchen brigade where each cook has a station and the dish passes along. CrewAI is built around that idea. An agent has a role, a goal and a backstory. A task has a description and an expected output, which says what finished looks like. A crew is a group of agents working together on a set of tasks, and by default it runs them in sequence. CrewAI describes itself as combining the collaborative intelligence of crews with the control of flows.

analyst = Agent(role="Ticket analyst", goal="Group last week's tickets",
                backstory="You read help-desk data.", llm=llm)
writer = Agent(role="Summary writer", goal="Write a short summary for the IT manager",
               backstory="You write plain, short updates.", llm=llm)
count_task = Task(description=f"Group these tickets:\n{digest}",
                  expected_output="Counts by status and by assignee", agent=analyst)
write_task = Task(description="Write a five-line summary from the counts",
                  expected_output="Five short lines", agent=writer, context=[count_task])
crew = Crew(agents=[analyst, writer], tasks=[count_task, write_task],
            process=Process.sequential)
print(crew.kickoff().raw)

This is also a sketch, not a lab file. kickoff() runs the crew, and the text of its output is in .raw. In this example the counts in digest come from code that queries the database; the crew writes the words, because counting belongs in code. CrewAI can point at an OpenAI-compatible endpoint through base_url and api_key on its LLM class. Choose it when the work is a fixed pipeline of roles.

What happened to AutoGen, and what is Microsoft Agent Framework?

AutoGen helped pioneer multi-agent applications, and many tutorials still use it. Its repository now carries a notice that the project is archived and no longer maintained by Microsoft, and tells new users to start with Microsoft Agent Framework. Agent Framework is the direct successor, created by the same teams. It merges AutoGen’s simple single- and multi-agent building blocks with the enterprise features of Semantic Kernel, and it adds workflows that give developers explicit control over multi-agent execution paths. So an old AutoGen tutorial is history, not a starting point.

How do the frameworks compare side by side?

Framework You describe the work as Choose it when
LangGraph A graph of shared state, nodes and edges You need routes you can see, saved state and approval
LlamaIndex An index over your documents, plus agents The job is mostly answering from documents
Agno An agent made from a model, instructions and tools You want a tool-using agent or small team quickly
CrewAI Roles, tasks and a crew The work is a fixed pipeline of roles
Microsoft Agent Framework Agents plus workflows You are on the Microsoft stack and need a maintained successor to AutoGen

The column “Choose it when” is our advice for this course, not a vendor claim. It comes from what each tool describes itself as, applied to the Copilot.

What goes wrong when a team picks a framework badly?

  1. A new project starts on AutoGen because a tutorial did, and no fixes arrive. The project is archived, and Microsoft points new users to its successor.
  2. A working LangGraph Copilot is rebuilt in another framework and loses its saved state and approval step. The switch was made on a hunch, without building the same small task in both tools first.
  3. A three-line document question is built as a five-agent crew and costs far more than needed. A document index, or one agent with one tool, would have done the job.

The fix in each case is the same habit: pick by the job, check that the framework is maintained, and test before you switch. The same advice as in session 31 applies: if a plain function can do the job, use the function.

What is Milestone 3, and how do I check it?

Milestone 3 is the multi-agent Copilot over MCP. It is met when the six checks in code/check_milestone3.py all pass. Run python code/check_milestone3.py; each check prints PASS or FAIL.

Check What the script measures
1 The client lists lookup_ticket and create_ticket from the MCP server
2 A caller holding just tickets:read is refused on create_ticket, nothing is written, and the refusal is in audit.log
3 A mixed question, the Wi-Fi question, uses both the policy agent and the ticket agent
4 The hand-off gives the ticket agent a summary that mentions the laptop or the Wi-Fi
5 A ticket is written only after a yes, through the server
6 code/notes/frameworks.md has five or more lines

Each check passes only when the script saw the thing happen. Check 2 counts the rows of the tickets table and the lines of audit.log before and after the refused call. Check 5 swaps the person for a function and counts the tickets after a no and after a yes:

multi_agent.approver = lambda title: False
multi_agent.ask(MIXED_QUESTION)
after_no = count_tickets()

multi_agent.approver = lambda title: True
multi_agent.ask(MIXED_QUESTION)
after_yes = count_tickets()

It passes when the no leaves the count unchanged, the yes adds exactly one ticket, and audit.log shows create_ticket allowed for the caller ticket-agent. Checks 3, 4 and 5 call the model, so they are the slowest part of the run. Check 6 fails until you write the framework note: make a folder code/notes and a file code/notes/frameworks.md with five lines, one each for LangGraph, LlamaIndex, Agno, CrewAI and AutoGen (or its successor, the Microsoft Agent Framework), each saying in your own words when you would choose it. Then run the script again. The script leaves one new ticket and some lines in audit.log; run python code/make_tickets_db.py to reset the table before another run.

If a line says FAIL, look at the likely cause before you change anything:

What you see Likely cause Where to look
Check 1 fails The server did not start, or a tool name changed code/ticket_server.py; run python code/list_tools.py
Check 2 fails The lock is missing, or the refusal was not logged require() and audit() in code/ticket_server.py
Check 3 fails The supervisor answered by itself or skipped an agent code/model_prompts/supervisor.md
Check 4 fails The report did not set needs_ticket, or the summary is empty code/model_prompts/report.md, hand_off()
Check 5 fails A ticket appeared after the no, or none after the yes create_ticket and approver in code/multi_agent.py
Check 6 fails The note is missing, outside code/, or has fewer than five lines code/notes/frameworks.md

Fix one thing, then run the whole script again. A model’s choices vary, so a check that fails once is worth running a second time.

Flashcards: how well do I know the AI agent framework choices?

Say the answer aloud before revealing it. Speaking is a different skill from reading, so practise both.

1. How does LangGraph describe work? As a graph of shared state, nodes and edges. It is a low-level orchestration framework that focuses on durable execution, streaming and human-in-the-loop, which is why the Copilot's routes and approval step live there.
2. When would you choose LlamaIndex? When the job is mostly answering from documents. You point a reader at a folder, build an index and ask the query engine, like a librarian with a card index.
3. What happens if you build a LlamaIndex index with no embedding model set? It falls back to OpenAI embeddings, so for a compatible endpoint you configure an embedding model yourself, for example with the `OpenAILike` integration.
4. What does an Agno agent pack together? A model, instructions and a list of tools in one object, where any Python function can be a tool. It is like a meal kit: the parts arrive in one box.
5. How does check 5 of Milestone 3 know a ticket was written only after a yes? It replaces the person with a function that says no, counts the tickets, then one that says yes and counts again. It passes when the no adds nothing, the yes adds exactly one, and `audit.log` shows `create_ticket` allowed for `ticket-agent`.
6. What are the parts of a CrewAI crew? Agents with a role, a goal and a backstory; tasks with a description and an expected output; and a crew that runs the tasks, in sequence by default. Like a newsroom, each member has a job.
7. Why should the counts in the weekly summary come from code and not the crew? A model can miscount, so counting belongs in code that queries the database. The crew writes the words around the counts.
8. What is the status of AutoGen? Microsoft's repository is archived and no longer maintained, and new users are told to start with Microsoft Agent Framework. Old AutoGen tutorials are history, not a starting point.
9. What is Microsoft Agent Framework? The direct successor to AutoGen, created by the same teams. It combines AutoGen's agent abstractions with Semantic Kernel's enterprise features and adds workflows for explicit control over multi-agent execution paths.
10. How do you decide whether to switch a working app to another framework? Build the same small task in both tools and compare the code and results first. Switching has a cost, and a working app may lose features such as saved state or an approval step.

Model answer: how do I explain my choice of AI agent framework in an interview?

The core question this session answers: “A teammate asks: which framework would you choose for the Copilot, what about AutoGen, and how do you know before switching?”

Four beats a teammate listens for, in this order: choose by the job · say what each framework describes · AutoGen is archived · test before switching. Skip any of these four and the answer is incomplete.

  1. Choose by the job: the problem decides, not familiarity.
  2. Say what each framework describes: a graph, an index, an agent with tools, a crew of roles.
  3. AutoGen is archived: point new work at Microsoft Agent Framework.
  4. Test before switching: the same small task in two tools.

The full answer, said aloud:

(Choose by the job) “I start from the problem. For routes I can see, saved state and an approval step, I use LangGraph. For answering from documents, LlamaIndex. For a quick tool-using agent, Agno. For a fixed pipeline of roles, CrewAI.

(Say what each framework describes) The difference is what I write by hand: a graph of nodes and edges, an index over files, an agent with a list of tools, or a crew of roles and tasks.

(AutoGen is archived) I would not start new work on AutoGen. Microsoft archived it and points new users to Microsoft Agent Framework, its successor, so I read old AutoGen tutorials as history.

(Test before switching) Before moving a working app I build the same small task in both tools and compare the code and the results. Switching has a cost, and I may lose saved state or approval.“

Missing a beat is a failure state. The follow-up question lands where you went quiet: skip the last beat and you are asked why you did not just rewrite it in the newer tool. Practise until the four come out without prompting, then stop talking and let the questions come.

Why this matters: why does choosing a framework well matter at work?

In a conversation with a teammate or manager: a tech lead is asked which framework to adopt before a new project. A short, reasoned answer, tied to the job and to whether the tool is still maintained, shows judgment rather than fashion.

On the job: teams lose time to a switch made on a hunch, or to building on a tool that stops getting fixes. Comparing on one small task and checking maintenance status are cheap and catch both.

In the rest of this course: session 33 traces these agents in Langfuse, session 34 scores them, and sessions 35 and 36 secure and govern them. The framework note you write now feeds the capstone’s architecture diagram.

After this unit: what do I do after this lesson, and what comes next?

Assignment. Run python code/llamaindex_policy.py on a policy question, compare its code and its answer with the policy files you wrote by hand, and write down what LlamaIndex decided for you. Project step. Milestone 3: write code/llamaindex_policy.py and code/check_milestone3.py (the build prompts are code/prompts/1-write-llamaindex-policy.md and code/prompts/2-write-check-milestone3.md), run the six checks, and write five lines in code/notes/frameworks.md on when you would choose each framework. The brief for the project work is the project file. Next session: seeing inside the app with Langfuse.

Want help building this with someone?

Tell us what you are working on and we will write back about a mentor-led cohort, live sessions or 1:1 training. It is only an enquiry: you choose whether to go further.

Ask about mentor-led training

TechEazy Consulting training material — not affiliated with or endorsed by OpenAI, Anthropic, LangChain or any other company named in this course.