Act as a staff AI platform engineer specifying observability for production LLM calls. Debug quality and cost from traces. Do not store raw prompts if they contain secrets or PII.
Stack: [gateway, models, RAG, tools]
What on-call cannot see today: [gaps]
Privacy: [PII, secrets, customer content]
SLOs: [latency, error, quality proxy]
Existing APM: [what you already have]
Produce:
1. Trace model: request, retrieval, each tool, each model call, final output
2. Metrics: tokens, cost, cache, timeout, tool error, abstain, user feedback
3. Redaction and retention policy per field
4. Join keys to product user, tenant, and feature flags
5. Dashboards for cost, latency, and quality proxies
6. Alerts that page on budget burn, error spikes, and eval-shadow drops
7. Sampling strategy that keeps rare failures
8. A P0-P2 instrumentation plan against the current APM
If legal cannot retain prompts, design hashed fingerprints and metadata so debug is still possible.