Advanced AI Agents: Multi-Agent Systems, Observability, Evals & Data Pipelines
How to build production-grade AI agent systems in Next.js — multi-agent orchestration over MCP, OpenTelemetry tracing, a native eval framework, the agent topics pattern, and AI agent data pipeline theory.
Get the Full Source Code on GitHub
This article is a summary. The full, runnable project —
five in-depth chapters with complete code, docs, and a live
demo UI — lives in the audoir/advanced-ai-tutorial repository.
Introduction
Prerequisites
This tutorial is a continuation of the AI Agent Tutorial, which covers building AI agents from scratch — from a simple streaming chatbot to a LangGraph-powered agent with database tools served over MCP. Complete that tutorial first before proceeding here.
Once you have the fundamentals of AI agents down — streaming chatbots, tool-calling, MCP, and LangGraph — the next frontier is building systems that are robust, observable, and scalable. The advanced-ai-tutorial repository picks up where the AI Agent Tutorial left off and introduces five advanced concepts that separate production-grade AI systems from prototypes.
This post summarizes a multi-agent content pipeline where an Orchestrator Agent coordinates a Researcher, Writer, and Editor — all via MCP — then adds OpenTelemetry tracing, a native eval framework, the "agent topics" pattern to eliminate context bloat, and the theory behind AI agent data pipelines. Head to the repo for the full, runnable code and docs.
Five Advanced Chapters
Each chapter builds on the last, layering in a new capability that production AI agent systems need.
Chapter 1: Multi-Agent Systems
An Orchestrator Agent dynamically coordinates a Researcher, Writer, and Editor — all exposed as tools via a stacked MCP server — instead of hardcoding the order of agent calls.
Chapter 2: Observability with OpenTelemetry
Adds distributed tracing with OpenTelemetry and Jaeger, propagating trace context across HTTP boundaries so every agent call shows up in one unified trace instead of orphaned spans.
Chapter 3: Evals for AI Agents
Builds a native eval framework — datasets, a model-agnostic runner, code-based and LLM-as-judge scorers — to measure agent quality systematically instead of by "vibes."
Chapter 4: Agent Topics
Refactors the pipeline so agents write to named, database-backed "topics" instead of passing full outputs through context — eliminating bloat and enabling resumable, inspectable pipelines.
Chapter 5: Data Pipelines for AI Agents
Explores how AI agent data pipelines differ from traditional ETL — cyclical instead of linear, real-time instead of batch, and built around a Retrieve-Reason-Act-Loop instead of a one-way flow into a data warehouse.
Key Concepts
Single-Agent vs. Multi-Agent
A single agent is simple, cheap, and predictable, but hits context window limits and degrades on complex tasks. A multi-agent system trades some latency and coordination overhead for specialization and parallelism.
Context Propagation Across MCP
Each MCP call is a separate HTTP request, so OTel trace context has to be manually injected into headers on the client and extracted on the server to keep spans from orphaning into disconnected traces.
Third-Party Evals vs. Building Your Own
Platforms like Braintrust or LangSmith offer faster time-to-value and rich UIs. A native framework offers full control, no vendor lock-in, and integrates directly with your existing CI pipeline.
Passing Context vs. Agent Topics
Passing full outputs as string arguments causes context bloat and leaves no record if a step fails. Writing to named database topics makes each step persisted, inspectable, and independently resumable.
Getting Started
Clone the repository, add your OpenAI API key, run npm install and npm run dev, then open http://localhost:3000. Requires Node.js v18+ and an OpenAI API key. See the repository README for the full setup instructions and each chapter's dedicated docs page.
Key Dependencies
| Package | Purpose |
|---|---|
ai / @ai-sdk/openai | Vercel AI SDK core — generateText, streamText — and the OpenAI provider |
@ai-sdk/mcp / mcp-handler | MCP client for the AI SDK and MCP server handler for Next.js routes |
@modelcontextprotocol/sdk | Official MCP TypeScript SDK |
better-sqlite3 | Synchronous SQLite driver used for chat history, agent topics, and eval logs |
zod | Schema validation for tool inputs and eval scorer outputs |
Conclusion
Building production-grade AI agent systems requires more than just wiring up an LLM with tools. You need observability to understand what's happening, evals to measure quality, and architectural patterns like agent topics to keep pipelines scalable and maintainable.
Each chapter in the repository implements a piece of the AI agent data pipeline — from retrieval and reasoning (Chapter 1) to storage and operational state (Chapter 4) — giving you a complete picture of what it takes to build AI agents that work reliably in production.
Learning Outcomes
By working through the tutorials, you will gain practical experience with:
- • Building multi-agent systems with an Orchestrator and specialist agents via stacked MCP servers
- • Adding distributed tracing to AI agents with OpenTelemetry and Jaeger
- • Propagating OTel trace context across HTTP boundaries (fixing orphaned spans)
- • Building a native eval framework with datasets, runners, scorers, and LLM-as-judge
- • Implementing the agent topics pattern to eliminate context bloat in multi-agent pipelines
- • Understanding the six stages of an AI agent data pipeline and how they differ from traditional ETL
- • Designing resumable, inspectable, and decoupled agent pipelines
Ready to Build It Yourself?
Clone the repository, add your OpenAI API key, and run all five chapters locally to see multi-agent orchestration, tracing, evals, and agent topics in action.
Clone advanced-ai-tutorial on GitHubAbout the Author
Wayne Cheng is the founder and AI app developer at Audoir, LLC. Prior to founding Audoir, he worked as a hardware design engineer for Silicon Valley startups and an audio engineer for creative organizations. He holds an MSEE from UC Davis and a Music Technology degree from Foothill College.
Further Exploration
Explore the advanced-ai-tutorial repository and experiment with extending the examples. Consider adding new specialist agents, connecting to external APIs, or implementing human-in-the-loop approval steps to deepen your understanding of advanced AI agent architectures.
New to AI agents? Start with the AI Agent Tutorial first, which covers building agents from scratch — from a simple streaming chatbot to a LangGraph-powered agent with MCP tools.
For more AI-powered development tools and tutorials, visit Audoir .