Individual skill
devtu-benchmark-harness
Continuous improvement system for ToolUniverse tools, skills, and plugin. Run benchmarks, diagnose failures, route fixes to devtu skills, retest. Use after skill optimization, tool additions, or as regression check.
skills/devtu-benchmark-harness/SKILL.md
Source commit 5598a9fc97e32964fe9b703818e7d0747142f179
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- Yes
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Network or broad shell
- Last checked
- Nov 9, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 1,780
- Repository last updated
- Nov 8, 2026
- License
- Apache-2.0
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- agents Repository topic
- reasoning-agent Repository topic
- reasoning-language-models Repository topic
- tool-use Repository topic
- ai-agents Repository topic
- ai-scientists Repository topic
- automated-science Repository topic
- co-pilot Repository topic
- co-scientist Repository topic
- ai-for-science Repository topic
- engineering Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesSKILL.mdcore · 10,703 bytesevals/evals.jsonasset · 702 bytesreferences/benchmark-guide.mddoc · 3,779 bytesscripts/analyze_failures.pyruns · 7,532 bytesscripts/analyze_results.pyruns · 18,866 bytesscripts/check_memorization.pyruns · 8,298 bytesscripts/compute_capsule_checksums.pyruns · 1,576 bytesscripts/generate_report.pyruns · 3,723 bytesscripts/grade_answers.pyruns · 14,546 bytesscripts/run_eval.pyruns · 49,466 bytesscripts/run_harness_loop.shruns · 4,717 bytesscripts/run_one_by_one.shruns · 2,913 bytesscripts/skill_routing_test.pyruns · 5,439 bytesscripts/skill_routing_test_v2.pyruns · 8,434 bytesscripts/solve_one.shruns · 3,678 bytes
Read excerpt
# Benchmark Harness — Continuous Improvement System
A 5-step feedback loop for improving ToolUniverse tools, skills, and plugin quality.
**Note**: This skill is dataset-agnostic. Per-benchmark score history, known-failing question IDs, and dataset-specific investigations belong in `temp_docs_and_tests/benchmark_tracking/` (gitignored workfolder), NOT in this skill directory.
## The Feedback Loop
```
1. RUN benchmark → 2. ANALYZE results → 3. DIAGNOSE failures → 4. FIX via devtu skill → 5. RETEST → repeat
```
## Orchestrated runner (preferred)
