AI Agent Masterclass: Long-Term Memory, Swarms, Checkpointing & Human-in-the-Loop

Introduction

Once you have the advanced AI agent fundamentals down — multi-agent systems, observability, evals, and agent topics — the next frontier is building systems that are truly production-ready: systems that remember users across sessions, can be paused and resumed mid-pipeline, and require human oversight for high-risk actions.

This masterclass picks up where the Advanced AI Agent Tutorial left off and introduces four new capabilities. You'll add long-term episodic and semantic memory so the Orchestrator learns user preferences over time, explore a swarm architecture where agents hand off to each other directly, implement state checkpointing for time travel debugging, and build a human-in-the-loop pipeline for safe database mutations.

What's Included

Getting Started

Key Dependencies

Chapter 1: Multi-Agent System (Recap)

Architecture

User prompt
    ↓
POST /api/orchestrator/default { prompt, runId, userId }
    ↓
🤖 Orchestrator Agent  (gpt + tools from /api/mcp/agents/orchestrator/mcp)
    │
    ├── write_topic("research_topic_v0", <user's prompt>)
    │
    ├── researcher_agent(readTopics=["research_topic_v0"], writeTopic="research_v0")
    │       └── Researcher Agent queries /api/mcp/database/read/mcp (SQL tools)
    │           → writes research report to chat_sessions.topics JSON
    │           → returns short confirmation
    │
    ├── writer_agent(readTopics=["research_v0"], writeTopic="draft_v0")
    │       └── Writer Agent reads research from DB, writes blog post draft
    │           → returns short confirmation
    │
    └── editor_agent(readTopics=["draft_v0"], writeTopic="final_v0")
            └── Editor Agent reads draft from DB, writes polished final article
                → returns short confirmation
    ↓
Orchestrator streams narration to user

The Agent Topics Pattern

This project uses the agent topics pattern. Instead of passing large content strings between agents through the Orchestrator's context window, each agent reads its input from and writes its output to a named slot stored as JSON in the chat_sessions table.

Why Topics?

Observability

The project includes OpenTelemetry tracing. Every generateText and streamText call emits spans automatically via the AI SDK's experimental_telemetry option. The Orchestrator also propagates the OTel trace context to the Agent MCP server via W3C traceparent headers, so all four agents appear in a single unified trace in Jaeger.

docker run --rm --name jaeger \
  -p 16686:16686 -p 4317:4317 -p 4318:4318 \
  cr.jaegertracing.io/jaegertracing/jaeger:2.18.0

Open http://localhost:16686 and select the ai-agent-masterclass service.

Chapter 2: Long-Term Memory

In Chapter 1 we built an Orchestrator that drives a pipeline of specialist sub-agents. Each run is self-contained: the agents do their work, write topics to the database, and the session ends. The next time the user sends a prompt, the Orchestrator starts fresh — no memory of what the user asked for before, no knowledge of their preferences.

This chapter adds long-term memory: the ability for the system to remember what happened in past sessions and to learn the user's preferences over time, so agents can apply them automatically without the user having to repeat themselves.

Types of AI Agent Memory

The Memory Pipeline

Orchestrator responds to the user (onFinish fires)
    ↓
Episodic Memory Agent  [runs in background via after()]
    → reads session history from chat_sessions.messages JSON
    → appends a factual summary to chat_sessions.episodic_memories JSON array
    ↓
Semantic Memory Agent  [runs immediately after episodic agent]
    → reads the new episodic summary + previous semantic memory
    → appends an updated user preference fact-sheet to users.semantic_memories JSON array
    ↓
Next session starts
    → Orchestrator reads semantic memory (preferences) + recent episodic memories (context)
    → injects both into its system prompt
    → applies user preferences automatically

Step 1: Triggering Memory After a Session

Memory updates run after the Orchestrator responds to the user, using Next.js's after() function. The user gets their response immediately and the memory agents run in the background without adding latency.

// lib/agents/orchestrator/default.ts

import { after } from "next/server";
import { updateLongTermMemory } from "@/lib/memory";

onFinish: async ({ text }) => {
  await agentMcpClient.close();

  after(() => updateLongTermMemory({ userId, sessionId: runId, finalText: text }));
},

Step 2: The Episodic Memory Agent

The Episodic Memory Agent reads the session's chat history and appends a factual 2–4 sentence summary to chat_sessions.episodic_memories.

Step 3: The Semantic Memory Agent

The Semantic Memory Agent reads the new episodic summary and the user's previous semantic memory, then appends an updated preference fact-sheet to users.semantic_memories. It also removes stale preferences — if a new episodic memory contradicts a previously stored preference, the old one is updated or deleted.

Step 4: Injecting Memory into the Orchestrator

At the start of each new session, the Orchestrator reads both memory types and injects them into its system prompt. Semantic memory comes first because it contains the most actionable, durable preferences.

## User Preferences (Semantic Memory)

This is a distilled fact-sheet of what is consistently true about this user,
built up from all their past sessions. You must apply these preferences
automatically — do not ask the user to repeat them.

[Semantic Memory #3 · 2026-06-02 18:30:00]
## User Preferences
- The user prefers content under 300 words.
- The user wants bullet points instead of prose.

## Recent Session History (Episodic Memory)

Summaries of the user's most recent sessions. Use these to understand what
they have been working on and to avoid repeating work unnecessarily.

[Session abc123 · 2026-06-01 14:22:00]
The user asked for a blog post about their best-selling electronics...

The Full Memory Flow (Example)

Chapter 3: Swarm Architecture

In Chapters 1 and 2 we built an Orchestrator that drives a fixed pipeline of sub-agents and remembers user preferences across sessions. The Orchestrator is a hub-and-spoke model: one central agent holds all the context, decides what to do next, and delegates to specialists one at a time.

This chapter introduces a fundamentally different architecture: the Swarm. Instead of a central boss, you have a team of autonomous specialists that hand off control to each other directly — no middleman required.

Architecture

User prompt
    ↓
POST /api/swarm { prompt, runId, userId }
    ↓
Swarm Loop (app/api/swarm/route.ts)
    │
    ├── Start: researcher (first prompt) or last active agent (follow-up)
    │
    ├── 🔍 Researcher Agent
    │       ├── Queries business database via MCP (inventory, customers, sales)
    │       ├── Writes research findings to chat_sessions.topics (e.g. "research_v0")
    │       └── Calls handoff(writer, summary, instructions, readTopics)
    │
    ├── ✍️ Writer Agent
    │       ├── Reads research from topics via list_topics() / read_topic()
    │       ├── Writes blog post draft to topics (e.g. "draft_v0")
    │       └── Calls handoff(editor, summary, instructions, readTopics)
    │
    └── 📝 Editor Agent
            ├── Reads draft from topics via list_topics() / read_topic()
            ├── Writes polished article to topics (e.g. "final_v0")
            └── Responds with text (no handoff = done)
    ↓
Final response streamed to browser via SSE

The Handoff Tool

The core primitive of the swarm is the handoff tool. Every agent has access to it. When an agent calls handoff(), it saves its output to a named topic and passes control to the next agent with instructions and a list of topic names to read.

// lib/agents/swarm/tools.ts

export function buildHandoffTool({ agentName, onHandoff }) {
  const config = SWARM_AGENT_CONFIG[agentName];

  return tool({
    description: "Hand off control to another agent. Call this when your work is done.",
    inputSchema: z.object({
      agentName: z.enum(config.handoffs),
      summary: z.string().describe("A brief summary of what you did and why you are handing off."),
      instructions: z.string().describe("Clear instructions for the next agent."),
      readTopics: z.array(z.string()).describe("Named topic slots the next agent should read."),
    }),
    execute: async ({ agentName: nextAgent, summary, instructions, readTopics }) => {
      onHandoff({ nextAgent, summary, instructions, readTopics });
      return `Handing off to ${nextAgent}.`;
    },
  });
}

The Swarm Loop

// app/api/swarm/route.ts

// Determine starting agent:
//   - First prompt: start at "researcher"
//   - Follow-up: resume from the last agent that finished (from registry)
const lastFinished = getLastFinishedAgent(runId);
let agentName = lastFinished ? lastFinished : "researcher";

while (hops < MAX_HOPS) {
  hops++;
  const result = await runSwarmAgent({ db, runId, agentName, input });

  // After each agent turn, send the updated messages snapshot via SSE
  send({ type: "messages", messages: getMessages(runId) });

  if (result.type === "done") break;

  agentName = result.nextAgent;
  input = { instructions: result.instructions, readTopics: result.readTopics };
}

Orchestrator vs. Swarm: Analysis

Example: Multi-Turn Conversation

Chapter 4: State Checkpointing

In Chapters 1–3 we built an Orchestrator, added long-term memory, and explored a swarm architecture. Every run was a one-way trip: the agents did their work, wrote topics to the database, and the session ended. If you wanted to change something mid-run, you had to start over from scratch.

This chapter adds state checkpointing: the ability to snapshot the conversation state before every step, roll back to any snapshot, and re-run the pipeline from that point — optionally with a new prompt. This is sometimes called time travel debugging for AI agents.

Why Checkpointing?

Long-running agent pipelines are expensive. A full Researcher → Writer → Editor run can take 30–60 seconds and cost several cents in API calls. If the Writer produces a draft you don't like, you shouldn't have to re-run the Researcher. You should be able to roll back to just before the Writer ran and give it different instructions.

The key insight is that the entire state of an agent pipeline is just the messages JSON array and the topics JSON object stored in chat_sessions. If you can snapshot both before each step and restore them on demand, you get full time travel for free.

Architecture

User prompt
    ↓
POST /api/orchestrator/checkpoints/start { prompt, runId, userId }
    ↓
runOrchestratorAgent (lib/agents/orchestrator/checkpoints.ts)
    │
    ├── checkpointBeforeMessage()  ← snapshot before user message
    ├── initChatSession()          ← write user message to DB
    │
    └── runOrchestratorCore()
            │
            ├── streamText({ abortSignal: req.signal, ... })
            │
            ├── experimental_onToolCallStart:
            │       └── checkpointBeforeMessage()  ← snapshot before tool call
            │
            ├── onStepFinish (tool call step):
            │       ├── saveAssistantMessage()
            │       ├── saveToolCallMessage()
            │       └── saveToolMessage()
            │
            ├── onAbort:
            │       └── agentMcpClient.close()     ← clean up on stop
            │
            └── onFinish:
                    ├── checkpointBeforeMessage()  ← snapshot before assistant reply
                    └── saveAssistantMessage()

Schema

No new tables are added. Checkpoints are stored as a JSON array in a new checkpoints column on the existing chat_sessions table:

interface StoredCheckpoint {
  message_id: string;          // short UUID (first 8 chars) — also the id of the next message
  messages_snapshot: StoredMessage[];          // full copy of messages at this point
  topics_snapshot: Record<string, unknown>;   // full copy of topics at this point
  created_at: string;
}

Step 1: Saving Checkpoints

Checkpoints are saved at three points in the pipeline. The checkpoint id and the id of the next message written are always the same value — checkpointBeforeMessage generates the UUID, saves the checkpoint, and returns the UUID so the caller can pass it to the message writer.

Step 2: Why experimental_onToolCallStart?

The AI SDK fires experimental_onToolCallStart before the tool executes — which is exactly when we need to snapshot the state. onStepFinish fires after the tool has already run and returned its result, so it is too late to checkpoint the pre-tool state there.

Step 3: Restoring a Checkpoint

// lib/chat-session.ts

export function restoreCheckpoint(db, sessionId, messageId) {
  const checkpoint = checkpoints.find((cp) => cp.message_id === messageId);
  if (!checkpoint) return null;

  // Restore messages AND topics from the snapshot.
  db.prepare(
    "UPDATE chat_sessions SET messages = ?, topics = ?, updated_at = datetime('now') WHERE id = ?"
  ).run(
    JSON.stringify(checkpoint.messages_snapshot),
    JSON.stringify(checkpoint.topics_snapshot),
    sessionId,
  );

  return storedMessagesToModelMessages(checkpoint.messages_snapshot);
}

Step 4: Aborting a Run

The AI SDK's abortSignal parameter lets the client cancel a stream mid-run. When the user clicks the Stop button, useCompletion's stop() function cancels the HTTP request. The DB state stays consistent because onStepFinish only fires for fully completed steps. Any step that gets aborted mid-stream simply doesn't get written to the DB — so the last checkpoint is always valid.

Example: Stop and Rerun with a New Prompt

Further Exploration: Branching

The current implementation supports linear time travel: roll back to any checkpoint and re-run from that point. But because checkpoints are never deleted, the data model already supports branching — like a version control system for your agent runs. Checkpoints from the original run remain intact even after a restore, so you can always navigate back to any branch. No schema changes needed.

Chapter 5: Human-in-the-Loop (HITL)

In Chapters 1–4 we built an Orchestrator, added long-term memory, explored a swarm architecture, and added state checkpointing. Every pipeline ran autonomously from start to finish — the agent decided what to do, called the tools, and reported back.

This chapter adds Human-in-the-Loop (HITL): the ability for an agent to deliberately pause mid-pipeline and wait for a human to either approve a high-risk action or provide missing information before continuing.

Why HITL?

Architecture: The Two-Turn Approval Flow

[UPDATE/DELETE path]

Turn 0 — first pass:
  write_topic("database-mutation_v0", <user's request>)
  write_topic("user-approval_v0", "false")
  database_mutator_agent(readTopics=["database-mutation_v0", "user-approval_v0"],
                         writeTopic="mutation-result_v0")
      └── Agent finds records, writes STATUS: fail + description of what will change
  read_topic("mutation-result_v0")
  request_human_approval(action_summary, question_for_human)
      └── ⚠️ LOOP BREAKS HERE — streamText returns to the browser
          The user sees the question and types a reply

Turn 1 — user replies "yes" or "no":
  write_topic("database-mutation_v1", <original request>)
  write_topic("user-approval_v1", "true" or "false")
  database_mutator_agent(readTopics=["database-mutation_v1", "user-approval_v1"],
                         writeTopic="mutation-result_v1")
      └── user-approval_v1 = "true"  → executes, writes STATUS: success
          user-approval_v1 = "false" → writes STATUS: fail (cancelled)
  Orchestrator narrates final outcome

Step 1: The HITL Tool

The key insight is that HITL is implemented as a tool that intentionally breaks the execution loop. The request_human_approval tool executes, returns a JSON payload describing the pending action, and then the Orchestrator — following its system prompt instructions — stops streaming and presents the question to the user.

// lib/agents/orchestrator/mutator-tools.ts

export const mutatorTools = {
  request_human_approval: tool({
    description:
      "Stop execution and present a confirmation question to the user for a destructive " +
      "UPDATE or DELETE operation. After calling this tool, stop and wait for the user's reply.",
    inputSchema: z.object({
      action_summary: z.string().describe("A clear description of what records will be modified."),
      question_for_human: z.string().describe("The confirmation question to present to the user."),
    }),
    execute: async ({ action_summary, question_for_human }) => {
      return JSON.stringify({
        status: "awaiting_human_approval",
        action_summary,
        question_for_human,
        instructions: "Present the question_for_human to the user and stop.",
      });
    },
  }),
};

Step 2: The Database Mutator Agent

The database_mutator_agent is the specialist that actually reads and writes the database. Its system prompt encodes the approval logic:

The Stateless Advantage

Other frameworks (LangGraph, AutoGen, CrewAI) implement HITL by pausing a running process and waiting for a signal to resume. This requires a persistent state store, a background worker, and a mechanism to wake the process back up.

The Vercel AI SDK approach is different: the "pause" is just the HTTP stream ending, and the "resume" is the next HTTP request. There is no running process to keep alive, no state to serialize, no worker to wake up. The entire conversation state lives in the database as a JSON array of messages.

Example: The Four Suggestion Prompts

Conclusion

Across five chapters we built a complete AI agent system from the ground up — an Orchestrator driving specialist sub-agents, long-term memory, a swarm architecture, state checkpointing, and human-in-the-loop controls. Each chapter adds a capability that makes the system more production-ready.

Learning Outcomes

By working through this masterclass, you will have gained practical experience with:

• • Adding episodic and semantic long-term memory to an Orchestrator agent
• • Building a swarm architecture where agents hand off control to each other directly
• • Understanding when to use an Orchestrator vs. a swarm (and why to prefer the Orchestrator)
• • Implementing state checkpointing with full time travel debugging for agent pipelines
• • Building human-in-the-loop controls as a tool — not middleware — for both authorization and steering
• • Designing stateless, horizontally scalable agent pipelines that survive restarts and support indefinite pauses

About the Author

Wayne Cheng is the founder and AI app developer at Audoir, LLC. Prior to founding Audoir, he worked as a hardware design engineer for Silicon Valley startups and an audio engineer for creative organizations. He holds an MSEE from UC Davis and a Music Technology degree from Foothill College.

Further Exploration

Explore the complete masterclass repository and experiment with extending the examples. Consider adding new specialist agents, implementing branching checkpoints with a visual tree UI, or extending HITL with structured approval buttons instead of plain text replies.

New to advanced AI agents? Start with the Advanced AI Agent Tutorial first, which covers multi-agent systems, OpenTelemetry observability, evals, and data pipeline theory.

For more AI-powered development tools and tutorials, visit Audoir .