Individual skill

llm-eval-harness

Tests/benchmarks a third-party LLM endpoint (OpenAI- or Anthropic-compatible): availability, fidelity, speed, concurrency, protocol compliance, quality regression. Use before hand-rolling a curl loop, when onboarding a provider, or debugging "system prompt 不生效" / a tok/s claim — 测评/压测一个模型或渠道. Not for TTS/voice-clone supplier eval (use the audio skill).

llm-eval-harness/SKILL.md

Read pinned source

Source commit 7c4d326ddd94d504e00e241ba49498e95160afbe

What was checked

Matches the observed repository HEAD: Yes
Scanner rules are current: Yes
Repository scan complete: No
Skill scan complete: No
Observed behavior: Network or broad shell
Last checked: Nov 4, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings): 1,522
Repository last updated: Nov 6, 2026
License: MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

legal Derived category
testing Derived category

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files

SKILL.mdcore · 20,316 bytes
SKILL.mdcore · 20,316 bytes
SKILL.mdcore · 20,316 bytes
SKILL.mdcore · 20,316 bytes
SKILL.mdcore · 20,316 bytes
SKILL.mdcore · 20,316 bytes
SKILL.mdcore · 20,316 bytes
SKILL.mdcore · 20,316 bytes
assets/example_usecases.jsonasset · 1,798 bytes
references/evaluation_disciplines.mddoc · 26,057 bytes
references/machine-results.mddoc · 8,710 bytes
references/production_testing_patterns.mddoc · 9,776 bytes
references/quality_blind_judge.mddoc · 4,174 bytes
references/vendor_evidence_protocol.mddoc · 8,629 bytes
scripts/availability_probe.pyruns · 18,880 bytes
scripts/concurrency_probe.pyruns · 9,866 bytes
scripts/eval_result.pyruns · 33,904 bytes
scripts/fidelity_probe.pyruns · 17,785 bytes
scripts/protocol_probe.pyruns · 22,364 bytes
scripts/speed_probe.pyruns · 17,094 bytes
scripts/usecase_runner.pyruns · 8,599 bytes
tests/test_availability.pyruns · 15,234 bytes
tests/test_results.pyruns · 38,208 bytes

Read excerpt


# LLM Eval Harness

## Overview

Give this skill an endpoint (`base_url` + `model` + an API key in an env var) and it
measures whether the endpoint actually works and whether the model is fast, stable,
protocol-correct, and good enough — instead of trusting the vendor's headline numbers.
Choose from the following evaluation dimensions:

| Dimension | Script | Answers |
|---|---|---|
| **Availability** | `scripts/availability_probe.py` | which model IDs work here, with explicit error classification |
| **Request fidelity** | `scripts/fidelity_probe.py` | do system prompt / tools / history actually REACH the model? |
| **Speed** | `scripts/speed_probe.py` | TTFT + sustained decode tok/s, **thinking-aware** |