"LLM Observability with Arize Phoenix: Open-Source Tracing, Evaluation, and Debugging for AI Applications"
Modern LLM systems fail in ways that ordinary logs, metrics, and dashboards cannot explain. This book is written for experienced engineers, platform teams, ML practitioners, and technical leads who need rigorous visibility into agent workflows, RAG pipelines, prompt behavior, and production-quality regressions. Centered on Arize Phoenix OSS, it shows how to make complex AI applications observable, debuggable, and operationally trustworthy.
Readers will learn how Phoenix fits into the contemporary LLM stack, how OpenTelemetry and OpenInference shape trace data, and how to instrument both framework-driven and highly customized systems. The book then moves into trace-driven debugging, retrieval analysis, evaluator design, dataset curation, experiment workflows, and programmatic automation for CI and release gates. By the end, readers will be able to connect execution traces to quality signals, isolate failure modes precisely, and build a disciplined feedback loop for continuous improvement.
The treatment is version-aware, operationally grounded, and aimed at advanced usage rather than introductory walkthroughs. It also extends beyond local experimentation to cover self-hosting, storage architecture, scaling, security, and governance, making it especially valuable for teams adopting Phoenix as shared infrastructure rather than a single-developer tool.
