Video by MLflow via YouTube
In the tenth and final tutorial of the Mastering MLflow for GenAI series, Jules Damji (Databricks) builds a multi-agent supervisor pattern with LangGraph and MLflow, routing queries to specialized agents, synthesizing results, and evaluating the full workflow with granular traces and scorers.
This session simulates a Databricks Agent framework–style supervisor using LangGraph’s StateGraph for acyclic, state-machine-like execution. The use case is a FEMA disaster-response assistant over roughly simulated records.
What You’ll Learn:
🔹 Multi-agent supervisor pattern: a supervisor fans out queries to specialized agents, then returns one cohesive answer
🔹 Supervisor router (Node 1): classify natural-language queries and route to a Genie-like agent, a knowledge assistant, or both
🔹 Genie-like agent (structured data): Natural Language → SQL with an LLM chain, executed against FEMA data via Python Pandas + SQLite
🔹 Knowledge-based assistant (unstructured data): document retrievers, OpenAI embeddings, and cosine similarity over in-memory files like PDFs
🔹 Synthesizer (Node 3): combine one or both agent outputs into a single response
🔹 Granular MLflow tracing: define agent tools: query, tool, embedding, and retriever functions with MLflow span markers
🔹 Built-in scorers: use MLflow’s 60+ scorers for answer relevance, safety, and guideline adherence on an evaluation dataset
🔹 Custom routing accuracy: implement a customer MLflow @scorer metric that compares supervisor fan-out against ground truth
🔹 Extra practice: supplementary notebooks for Deep Agents and CrewAI multi-agent frameworks
Resources:
🔗 Notebook 1.10: https://github.com/dmatrix/mlflow-genai-tutorials/blob/main/10_multi_agent_supervisor.ipynb
🎥 Full Series Playlist: https://youtube.com/playlist?list=PLaoPu6xpLk9EI99TuOjSgy-UuDWowJ_mR