ContextJet-ai/awesome-llm-observability

"owner": {     "name": "ContextJet.ai",     "url": "https://www.contextjetai.com"

"name": "ContextJet.ai",     "url": "https://www.contextjetai.com"   },

"url": "https://www.contextjetai.com"

"author": {     "name": "ContextJet.ai",     "url": "https://www.contextjetai.com"

"name": "ContextJet.ai",     "url": "https://www.contextjetai.com"   },

"url": "https://www.contextjetai.com"

},   "homepage": "https://github.com/ContextJet-ai/awesome-llm-observability",   "license": "CC0-1.0",

"homepage": "https://github.com/ContextJet-ai/awesome-llm-observability",

Standard distributed-systems resilience patterns applied to LLM calls: exponential backoff with jitter (AWS Architecture guidance), retry/timeout/circuit-breaker (Nygard, *Release It!*). Gateways implement provider fallback: [LiteLLM](https

1. **Pick a library** - [Guardrails AI](https://github.com/guardrails-ai/guardrails) (validators + structured output), [Presidio](https://github.com/data-privacy-stack/presidio) + [Purple Llama](https://github.com/meta-llama/PurpleLlama) (P

"Look at your data" and error analysis are the core of practitioner eval methodology: Hamel Husain, [*Your AI Product Needs Evals*](https://hamel.dev/blog/posts/evals/); human labels are the ground truth that automated LLM-as-a-judge is cal

Eval-first practice: Hamel Husain, [*Your AI Product Needs Evals*](https://hamel.dev/blog/posts/evals/). Multi-metric, stratified evaluation: HELM, Liang et al. ([arXiv:2211.09110](https://arxiv.org/abs/2211.09110)). Human-alignment of judg

Consistency-based hallucination detection: SelfCheckGPT, Manakul et al. 2023 ([arXiv:2303.08896](https://arxiv.org/abs/2303.08896)). Meaning-level variant: semantic entropy, Farquhar et al., Nature 2024.

Executable file

Executable file

Human feedback as the signal for LLM quality underpins RLHF and preference modeling (Ouyang et al., InstructGPT, [arXiv:2203.02155](https://arxiv.org/abs/2203.02155)) and human-aligned evaluation (Zheng et al. 2023, [arXiv:2306.05685](https

Human-preference evaluation: Chatbot Arena, Zheng et al. 2023 ([arXiv:2306.05685](https://arxiv.org/abs/2306.05685)). Multi-metric holistic evaluation: HELM, Liang et al. ([arXiv:2211.09110](https://arxiv.org/abs/2211.09110)). Benchmark har

Self-consistency: SelfCheckGPT, Manakul et al. 2023 ([arXiv:2303.08896](https://arxiv.org/abs/2303.08896)). Semantic entropy: Farquhar et al., *Detecting hallucinations in large language models using semantic entropy*, Nature 2024 ([paper](

- Faithfulness / RAG: [Ragas](https://github.com/vibrantlabsai/ragas), [DeepEval](https://github.com/confident-ai/deepeval) (both have faithfulness + hallucination metrics).

Executable file

Executable file

Reference: OpenTelemetry GenAI semantic conventions (moved out of the main semconv repo in 2026) - https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/README.md

Agentic evaluation emphasizes end-to-end task success and trajectory, as in agent benchmarks like SWE-bench (Jimenez et al., [arXiv:2310.06770](https://arxiv.org/abs/2310.06770)), WebArena ([arXiv:2307.13854](https://arxiv.org/abs/2307.1385

[Ragas](https://github.com/vibrantlabsai/ragas) implements all four; [DeepEval](https://github.com/confident-ai/deepeval) and most observability platforms (Langfuse, Phoenix, Opik) have RAG evaluators too.

- **Auto instruction search (APE-style):** have an LLM propose many instruction candidates, score each on the eval set, keep the best. - **Few-shot / demonstration optimization:** algorithmically select which examples to include (this is of

- **Frameworks:** [DSPy](https://github.com/stanfordnlp/dspy) compiles and optimizes prompt pipelines against a metric (bootstrap few-shot, MIPRO); [promptfoo](https://github.com/promptfoo/promptfoo) and several eval platforms support promp

Automatic instruction optimization: APE, Zhou et al. 2022 ([arXiv:2211.01910](https://arxiv.org/abs/2211.01910)). Programmatic prompt/pipeline optimization: DSPy, Khattab et al. ([arXiv:2310.03714](https://arxiv.org/abs/2310.03714)); MIPRO,

1. **Build an attack set** per class above (start from a public jailbreak/injection dataset, add app-specific ones). Version it in the repo. 2. **Automate it** - run the attack set as an eval suite. Tools: [promptfoo](https://github.com/pro

2. **Automate it** - run the attack set as an eval suite. Tools: [promptfoo](https://github.com/promptfoo/promptfoo) (built-in red-team/`redteam` mode), [DeepTeam](https://github.com/confident-ai/deepteam), [garak](https://github.com/NVIDIA

Attack taxonomy: [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) (prompt injection is LLM01). Automated attacks: *Automatic and Universal Prompt Injection Attacks against LLMs

1. **Self-host the backend.** For regulated data, prefer an OSS platform you host: [Langfuse](https://github.com/langfuse/langfuse) (MIT), [Arize Phoenix](https://github.com/Arize-ai/phoenix), [Comet Opik](https://github.com/comet-ml/opik),

1. **Detect** PII with a real detector - [Presidio](https://github.com/data-privacy-stack/presidio) or a domain ruleset (account/card/IBAN/SSN patterns for finance). Don't rely on ad-hoc regex alone.

_Authored by [ContextJet.ai](https://www.contextjetai.com) - we build secure, observable AI for finance and regulated industries._

Executable file

Executable file

Online controlled experimentation methodology: Kohavi, Tang & Xu, *Trustworthy Online Controlled Experiments* (2020), and Kohavi et al., *Controlled experiments on the web* (2009). Applied here with LLM-specific metrics (online LLM-as-judge

Drift/monitoring for ML has a long lineage (data & concept drift); the LLM-specific additions are online LLM-as-a-judge quality scoring (Zheng et al. 2023, [arXiv:2306.05685](https://arxiv.org/abs/2306.05685)) and token/cost/latency telemet

This operationalizes the observability-to-eval feedback loop: online quality signals (LLM-as-a-judge, Zheng et al. 2023, [arXiv:2306.05685](https://arxiv.org/abs/2306.05685)) and human feedback surface weak traces; those become evaluation c

Executable file

CC0 - ContextJet.ai. Based on the OTel GenAI semantic conventions (https://github.com/open-telemetry/semantic-conventions-genai).

(https://github.com/open-telemetry/semantic-conventions-genai).

Executable file