Video by MLflow via YouTube
Generic LLM judges work well for broad tasks. In domain-specific settings, they often miss the nuance that subject matter experts care about.
This is Part 2 of a two-part series on evaluating a RAG agent with MLflow. In Part 1, we built the evaluation pipeline: traces, ground truth, multi-framework scorers, and a custom ChildAppropriateness judge. In Part 2, we close the loop with the alignment lifecycle: aligning that judge to SME standards using MemAlign.
What this video covers:
• Use case: The same RAG agent from Part 1 (assumes you already registered the judge and ran an initial evaluation)
• Why generic LLM-as-a-judge scorers drift from expert standards in domain-specific evaluation
• The three-stage alignment lifecycle: judge assessment → SME feedback → MemAlign alignment
• Collecting SME feedback on traces in MLflow (UI and programmatically with mlflow.log_feedback)
• Aligning the ChildAppropriateness judge with human feedback using MemAlign (semantic + episodic memory)
• Re-running evaluation with the aligned judge to improve domain-specific scoring
• How the evaluation-driven development loop and alignment lifecycle work together
Watch Part 1 first:
🎤 Speaker: Joana Mesquita, MLflow Ambassador
🔗 Repo with the code: https://github.com/joanacmesquitaf/rag-agent-mlflow-evaluation
📖 Related blog (series context): https://medium.com/@joana.c.mesquita.f/from-generic-evaluators-to-domain-experts-judge-alignment-with-mlflow-memalign-8fc34d0ad717?postPublishedType=initial
📚 MemAlign docs: https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/memalign/