#847 by GitHub stars
agentscope-ai/OpenJudge
OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards
History
Review weekly changes to SKILL.md before updating. Dates mark the weeks in which we observed each version; checks apply to the recorded commits.
View weekly changes
skills/academic-eval/00-academic-router/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Instructions only
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/academic-eval/01-paper-review/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/academic-eval/02-bib-verify/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/academic-eval/03-ref-hallucination-arena/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/arena-eval/00-arena-router/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Instructions only
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/arena-eval/01-auto-arena/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/arena-eval/02-ref-hallucination-arena/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/claude-authenticity/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/00-meta-eval/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Instructions only
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/01-eval-design/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/02-metric-design/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/03-align-human/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/04-eval-report/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/05-rag-eval/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/06-prompt-regression/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/07-redteam/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/eval_pipeline/08-bootstrap/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/find-skills-combo/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/mmx-cli/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/openjudge-core/01-graders-and-pipeline/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
skills/openjudge-core/02-rl-reward/SKILL.md
Week of Nov 2, 2026 · +0 / −0 lines
Before Not checked yet → After Network or broad shell
c2fdc38314cd → 6416629995e5
Diff text is unavailable under the recorded licenses. Line counts and the GitHub link remain available.
Week of Oct 26, 2026 · Line counts unavailable
Before Not checked yet → After Not checked yet
First observation → c2fdc38314cd
No earlier version was observed for comparison.
1.1kGitHub stars
63stars in 7 days
21skills in this repository
9commits in 30 days
Heat
8.2- Star growth, last 7 daysAhead of 97% of tracked repositories
- Star growth, last 24 hoursAhead of 97% of tracked repositories
- Rank movement, last 7 daysAhead of 94% of tracked repositories
- Commits, last 30 daysAhead of 76% of tracked repositories
- Accounts asking about it through MCP, last 7 daysNot available
- Hot votes this weekAhead of 0% of tracked repositories
Heat never changes the order by stars, and sponsorship never touches it. How heat works
Stars over time
Individual skills in this repository
- 00-academic-router
skills/academic-eval/00-academic-router - 01-paper-review
skills/academic-eval/01-paper-review - 02-bib-verify
skills/academic-eval/02-bib-verify - 03-ref-hallucination-arena
skills/academic-eval/03-ref-hallucination-arena - 00-arena-router
skills/arena-eval/00-arena-router - 01-auto-arena
skills/arena-eval/01-auto-arena - 02-ref-hallucination-arena
skills/arena-eval/02-ref-hallucination-arena - claude-authenticity
skills/claude-authenticity - meta-eval
skills/eval_pipeline/00-meta-eval - eval-design
skills/eval_pipeline/01-eval-design - metric-design
skills/eval_pipeline/02-metric-design - align-human
skills/eval_pipeline/03-align-human - eval-report
skills/eval_pipeline/04-eval-report - rag-eval
skills/eval_pipeline/05-rag-eval - prompt-regression
skills/eval_pipeline/06-prompt-regression - redteam
skills/eval_pipeline/07-redteam - bootstrap
skills/eval_pipeline/08-bootstrap - find-skills-combo
skills/find-skills-combo - mmx-cli
skills/mmx-cli - 01-graders-and-pipeline
skills/openjudge-core/01-graders-and-pipeline - 02-rl-reward
skills/openjudge-core/02-rl-reward
How it works
00-academic-router
When your agent uses it
Use when the user wants help with academic papers or citations but it's unclear which specific workflow fits — reviewing a paper, checking a BibTeX file for fake references, or benchmarking multiple LLMs on reference-recommendation accuracy. Also use when the user mentions paper review, peer review, BibTeX verification, citation checking, reference hallucination, or academic literature accuracy and hasn't specified which of those three tasks they mean. This skill is the entry router for the academic-eval suite: it asks one diagnostic question then routes to the right sub-skill.
What it loads into context
Frontmatter
136 tokensAlways available to the agent
SKILL.md
940 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Diagnostic Question
- Triage Table
- Key distinctions
- Output
Files
SKILL.mdcore3,921 bytes
Read SKILL.md
# Academic Eval Router
Entry router for the `academic-eval` suite. You diagnose what the user actually
wants and route them to one of three sub-skills. You don't review papers, verify
BibTeX files, or run arena benchmarks yourself — you're the triage desk.
Each sub-skill is self-contained: it carries inline everything it needs, so it can
be installed and used on its own.
## Diagnostic Question
Ask (unless the user's request already makes the answer obvious):
```01-paper-review
When your agent uses it
Review academic papers for correctness, quality, and novelty using OpenJudge's multi-stage pipeline. Supports PDF files and LaTeX source packages (.tar.gz/.zip). Covers 10 disciplines: cs, medicine, physics, chemistry, biology, economics, psychology, environmental_science, mathematics, social_sciences. Use when the user asks to review, evaluate, critique, or assess a research paper, check references, or verify a BibTeX file.
What it loads into context
Frontmatter
108 tokensAlways available to the agent
SKILL.md
2,498 tokensWhen the skill is activated
Reference documents
2,150 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Prerequisites
- Gather from user before running
- Quick start
- All options
- Interpreting results
- Model selection
- Additional resources
- Troubleshooting API errors
Files
SKILL.mdcore9,576 bytesSKILL.mdcore9,576 bytesSKILL.mdcore9,576 bytesSKILL.mdcore9,576 bytesSKILL.mdcore9,576 bytesSKILL.mdcore9,576 bytesreference.mddoc7,707 bytes
Read SKILL.md
# Paper Review Skill
Multi-stage academic paper review using the OpenJudge `PaperReviewPipeline`:
1. **Safety check** — jailbreak detection + format validation
2. **Correctness** — objective errors (math, logic, data inconsistencies)
3. **Review** — quality, novelty, significance (score 1–6)
4. **Criticality** — severity of correctness issues
5. **BibTeX verification** — cross-checks references against CrossRef/arXiv/DBLP
## Prerequisites
```bash
# Install OpenJudge02-bib-verify
When your agent uses it
Verify a BibTeX file for hallucinated or fabricated references by cross-checking every entry against CrossRef, arXiv, and DBLP. Reports each reference as verified, suspect, or not found, with field-level mismatch details (title, authors, year, DOI). Use when the user wants to check a .bib file for fake citations, validate references in a paper, or audit bibliography entries for accuracy.
What it loads into context
Frontmatter
106 tokensAlways available to the agent
SKILL.md
688 tokensWhen the skill is activated
Reference documents
4,648 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Prerequisites
- Gather from user before running
- Quick start
- Relevant options
- Interpreting results
- Additional resources
Files
../01-paper-review/SKILL.mddoc9,576 bytes../01-paper-review/reference.mddoc7,707 bytesSKILL.mdcore2,750 bytesSKILL.mdcore2,750 bytesSKILL.mdcore2,750 bytes
Read SKILL.md
# BibTeX Verification Skill
Check every entry in a `.bib` file against real academic databases using the
OpenJudge `PaperReviewPipeline` in BibTeX-only mode:
1. **Parse** — extract all entries from the `.bib` file
2. **Lookup** — query CrossRef, arXiv, and DBLP for each reference
3. **Match** — compare title, authors, year, and DOI
4. **Report** — flag each entry as `verified`, `suspect`, or `not_found`
## Prerequisites
```bash
pip install py-openjudge litellm03-ref-hallucination-arena
When your agent uses it
Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.
What it loads into context
Frontmatter
116 tokensAlways available to the agent
SKILL.md
2,471 tokensWhen the skill is activated
Reference documents
5,318 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Prerequisites
- Gather from user before running
- Quick start
- CLI options
- Minimal config file
- Full config reference
- Dataset format
- Interpreting results
- Output files
- API key by model
- Additional resources
Files
../../../docs/validating_graders/ref_hallucination_arena.mddoc23,535 bytesSKILL.mdcore9,815 bytesSKILL.mdcore9,815 bytesSKILL.mdcore9,815 bytesSKILL.mdcore9,815 bytes
Read SKILL.md
# Reference Hallucination Arena Skill
Evaluate how accurately LLMs recommend real academic references using the
OpenJudge `RefArenaPipeline`:
1. **Load queries** — from JSON/JSONL dataset
2. **Collect responses** — BibTeX-formatted references from target models
3. **Extract references** — parse BibTeX entries from model output
4. **Verify references** — cross-check against Crossref / PubMed / arXiv / DBLP
5. **Score & rank** — compute verification rate, per-field accuracy, discipline breakdown
6. **Generate report** — Markdown report + visualization charts
## Prerequisites
00-arena-router
When your agent uses it
Use when the user wants to compare or benchmark multiple LLMs/agents arena-style but it's unclear which specific workflow fits — a general-purpose win-rate comparison on a custom task, or a benchmark specifically about reference/citation hallucination rate. Also use when the user mentions model arena, agent arena, pairwise model comparison, win-rate ranking, or comparing models on a task and hasn't specified whether that task is generic or about citation accuracy. This skill is the entry router for the arena-eval suite: it asks one diagnostic question when needed, then recommends the workflow or workflows needed to cover the request.
What it loads into context
Frontmatter
148 tokensAlways available to the agent
SKILL.md
918 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Diagnostic Question
- Triage Table
- Key distinction
- Output
Files
SKILL.mdcore4,141 bytes
Read SKILL.md
# Arena Eval Router
Entry router for the `arena-eval` suite. You diagnose what the user wants to
compare models on and route them to the appropriate sub-skill or both when
the request spans both evaluation goals. You don't run comparisons yourself
— you're the triage desk.
Each sub-skill is self-contained: it carries inline everything it needs, so it
can be installed and used on its own.
## Diagnostic Question
Ask (unless the user's request already makes the answer obvious):
01-auto-arena
When your agent uses it
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation.
What it loads into context
Frontmatter
122 tokensAlways available to the agent
SKILL.md
2,474 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Prerequisites
- Gather from user before running
- Quick start
- CLI options
- Minimal config file
- Full config reference
- Interpreting results
- Checkpoint & resume
- Output files
- API key by model
- Additional resources
Files
SKILL.mdcore9,965 bytesSKILL.mdcore9,965 bytesSKILL.mdcore9,965 bytesSKILL.mdcore9,965 bytesSKILL.mdcore9,965 bytes
Read SKILL.md
# Auto Arena Skill
End-to-end automated model comparison using the OpenJudge `AutoArenaPipeline`:
1. **Generate queries** — LLM creates diverse test queries from task description
2. **Collect responses** — query all target endpoints concurrently
3. **Generate rubrics** — LLM produces evaluation criteria from task + sample queries
4. **Pairwise evaluation** — judge model compares every model pair (with position-bias swap)
5. **Analyze & rank** — compute win rates, win matrix, and rankings
6. **Report & charts** — Markdown report + win-rate bar chart + optional matrix heatmap
## Prerequisites
```bash02-ref-hallucination-arena
When your agent uses it
Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.
What it loads into context
Frontmatter
116 tokensAlways available to the agent
SKILL.md
2,471 tokensWhen the skill is activated
Reference documents
5,318 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Prerequisites
- Gather from user before running
- Quick start
- CLI options
- Minimal config file
- Full config reference
- Dataset format
- Interpreting results
- Output files
- API key by model
- Additional resources
Files
../../../docs/validating_graders/ref_hallucination_arena.mddoc23,535 bytesSKILL.mdcore9,815 bytesSKILL.mdcore9,815 bytesSKILL.mdcore9,815 bytesSKILL.mdcore9,815 bytes
Read SKILL.md
# Reference Hallucination Arena Skill
Evaluate how accurately LLMs recommend real academic references using the
OpenJudge `RefArenaPipeline`:
1. **Load queries** — from JSON/JSONL dataset
2. **Collect responses** — BibTeX-formatted references from target models
3. **Extract references** — parse BibTeX entries from model output
4. **Verify references** — cross-check against Crossref / PubMed / arXiv / DBLP
5. **Score & rank** — compute verification rate, per-field accuracy, discipline breakdown
6. **Generate report** — Markdown report + visualization charts
## Prerequisites
claude-authenticity
When your agent uses it
Detect whether an API endpoint is backed by genuine Claude (not a wrapper, proxy, or impersonator) using 9 weighted rule-based checks that mirror the claude-verify project. Also extracts injected system prompts from providers that override Claude's identity. Fully self-contained — copy the code below and run, no extra packages beyond httpx. Use when the user wants to verify a Claude API key or endpoint, check if a third-party Claude service is authentic, audit API providers for Claude authenticity, test multiple models in parallel, or discover what system prompt a provider has injected.
What it loads into context
Frontmatter
138 tokensAlways available to the agent
SKILL.md
5,266 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- The 9 checks (mirrors [claude-verify](https://github.com/molloryn/claude-verify))
- Gather from user before running
- Self-contained script
- Interpreting results
- Extracting injected system prompts
- Troubleshooting
Files
SKILL.mdcore22,536 bytesSKILL.mdcore22,536 bytesSKILL.mdcore22,536 bytesSKILL.mdcore22,536 bytesSKILL.mdcore22,536 bytesSKILL.mdcore22,536 bytes
Read SKILL.md
# Claude Authenticity Skill
Verify whether an API endpoint serves genuine Claude and optionally extract any
injected system prompt.
**No installation required beyond `httpx`.** Copy the code blocks below directly
into a single `.py` file and run — no openjudge, no cookbooks, no other setup.
```bash
pip install httpx
```
## The 9 checks (mirrors [claude-verify](https://github.com/molloryn/claude-verify))
meta-eval
When your agent uses it
Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.
What it loads into context
Frontmatter
116 tokensAlways available to the agent
SKILL.md
2,291 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Checklist
- Diagnostic Questions
- Triage Table
- Output
- Canonical Workflow (the standard lifecycle)
- Red Flags — STOP and Re-evaluate
- Rationalization Defense
- Common Mistakes
- What This Skill Doesn't Cover
Files
SKILL.mdcore10,147 bytes
Read SKILL.md
<HARD-GATE>
NO sub-skill recommendation WITHOUT identifying data_form + label_status (these two pick the entry workflow).
ALWAYS give a provisional recommendation once data_form + label_status are known, even if stakes/user_prior are still unknown — then ask the remaining questions to refine the downstream path. Do not withhold the route while waiting on stakes.
</HARD-GATE>
# Meta Eval
Entry router for the eval skill collection. You diagnose what the user has and route
them to the right sub-skill. You don't do evaluation yourself — you're the triage desk.
Each sub-skill is self-contained: it carries inline the data shapes, statistics, and data
principles it needs, so it can be installed and used on its own.
## Checklisteval-design
When your agent uses it
Use when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.
What it loads into context
Frontmatter
92 tokensAlways available to the agent
SKILL.md
2,570 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Activate
- Checklist
- Coverage check: run the bundled script
- Step 1: Extract Evaluation Dimensions
- Step 2: Design Stratified Sampling
- Step 3: Generate Test Data
- Step 4: Output OpenJudge Dataset
- Output Files
- Common Mistakes
- Next Skills
Scripts executed separately
scripts/coverage_check.py
Files
SKILL.mdcore11,173 bytesSKILL.mdcore11,173 bytesSKILL.mdcore11,173 bytesscripts/coverage_check.pyruns5,670 bytes
Read SKILL.md
# Eval Design
Design high-quality evaluation datasets that measure what actually matters for your
application. You extract evaluation dimensions from business context, structure them
into stratified test cases, and output datasets ready for OpenJudge `GradingRunner`.
## When to Activate
- User has agent traces / production logs and wants to build an eval set from them
- User has evaluation principles but needs properly stratified test data
- User wants to generate adversarial examples that stress-test their system
- User needs coverage analysis — are they testing all the right things?
- User wants a labeling guide for human annotators
metric-design
When your agent uses it
Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.
What it loads into context
Frontmatter
102 tokensAlways available to the agent
SKILL.md
4,864 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Activate
- Checklist
- Step 1: Select Grader Type Per Dimension
- Step 2: Create Custom Graders
- Step 3: Auto-Generate Graders (Cold Start)
- Step 4: Anti-Pattern Scan
- Step 5: Build Pipeline Code
- Common Mistakes
- Next Skills
Files
SKILL.mdcore20,606 bytesSKILL.mdcore20,606 bytesSKILL.mdcore20,606 bytesSKILL.mdcore20,606 bytesSKILL.mdcore20,606 bytesSKILL.mdcore20,606 bytesSKILL.mdcore20,606 bytes
Read SKILL.md
# Metric Design
Select, configure, and combine evaluation graders into a working pipeline. You choose
the right tool for each evaluation dimension — from zero-cost code checks to LLM judges
— and produce executable `GradingRunner` code that runs on OpenJudge.
> **Requires OpenJudge** (`pip install py-openjudge`). This skill is intentionally
> SDK-centric — grader selection, `GradingRunner`, and aggregators are OpenJudge APIs. The
> design/decision logic still applies if you use another harness; only the code does not.
## When to Activate
- User has eval dimensions/principles but doesn't know which grader type to use
- User wants to write an LLM-as-judge prompt for a specific failure modealign-human
When your agent uses it
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
What it loads into context
Frontmatter
114 tokensAlways available to the agent
SKILL.md
3,305 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Load and Pair Data
- Step 2: Measure TPR/TNR
- Step 3: Calculate Agreement
- Step 4: Five Bias Detection Checks
- Step 5: Disagreement Pattern Analysis
- Step 6: Human-Reduction Roadmap
- Step 7: Confirmation and Output
- Common Mistakes
- Next Skills
Scripts executed separately
scripts/calibration.py
Files
SKILL.mdcore12,319 bytesSKILL.mdcore12,319 bytesSKILL.mdcore12,319 bytesSKILL.mdcore12,319 bytesSKILL.mdcore12,319 bytesSKILL.mdcore12,319 bytesscripts/calibration.pyruns15,883 bytes
Read SKILL.md
<HARD-GATE>
NO calibrated:true WITHOUT TPR >= 0.8 AND TNR >= 0.8 AND boundary stratum TPR >= 0.6 AND TNR >= 0.6 AND n_dev >= 10 per class AND test-set drop < 10%.
NO human_reduction_phase >= 2 WITHOUT kappa >= 0.6 AND boundary kappa >= 0.6.
NO alignment conclusion WITHOUT all 5 bias checks completed.
</HARD-GATE>
# Align Human
Measure whether your automatic judge agrees with human judgment, detect where and
why they disagree, and build a roadmap to reduce human review over time.
## When to Activate
- You have a working judge/grader and 50+ human-labeled exampleseval-report
When your agent uses it
Use when the user has run multiple evaluation skills and wants a comprehensive analysis — maturity assessment, cross-skill signals, trends, prioritized actions, and an executive summary. Also use when the user mentions eval health check, evaluation audit, ship readiness, evaluation maturity, or "how good is my evaluation system itself." This is a read-only analysis skill.
What it loads into context
Frontmatter
85 tokensAlways available to the agent
SKILL.md
2,436 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Activate
- Checklist
- Step 1: Inventory Scan
- Step 2: Maturity Assessment
- Step 3: Cross-Skill Signal Synthesis
- Step 4: Weakness Diagnosis
- Step 5: Root Cause Classification
- Step 6: Prioritized Recommendations
- Step 7: Executive Summary
- Output
- Common Mistakes
- Next Skills
Files
SKILL.mdcore9,811 bytesSKILL.mdcore9,811 bytes
Read SKILL.md
<HARD-GATE>
NO recommendation WITHOUT statistical evidence backing it.
NO "system ready" declaration WITHOUT all calibrated judges passing AND all production gates green.
NO trend analysis WITHOUT at least 2 data points in history.
</HARD-GATE>
# Eval Report
Synthesize everything from your evaluation journey into a comprehensive report.
This skill is read-only — it analyzes what exists, doesn't create new graders or datasets.
## When to Activate
- You've run 2+ evaluation skills and want the big picturerag-eval
When your agent uses it
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.
What it loads into context
Frontmatter
99 tokensAlways available to the agent
SKILL.md
2,320 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Load RAG Traces
- Step 2: Separate Retrieval vs Generation
- Step 3: Faithfulness Evaluation
- Step 4: Retrieval Evaluation
- Step 5: Diagnostic Matrix
- Step 6: Output
- Common Mistakes
- Next Skills
Scripts executed separately
scripts/rag_diagnostic.py
Files
SKILL.mdcore9,684 bytesSKILL.mdcore9,684 bytesSKILL.mdcore9,684 bytesSKILL.mdcore9,684 bytesSKILL.mdcore9,684 bytesSKILL.mdcore9,684 bytesSKILL.mdcore9,684 bytesscripts/rag_diagnostic.pyruns7,366 bytes
Read SKILL.md
# RAG Eval
Evaluate RAG systems by diagnosing retrieval and generation separately. A single
"RAG accuracy" number hides whether the problem is finding the right documents or
using them correctly. This skill separates them so you know what to fix.
## When to Activate
- User has a RAG pipeline (retriever + generator) with traces
- User wants to know if their RAG system hallucinates
- User is optimizing chunking strategy and needs before/after comparison
- User wants to build a RAG evaluation dataset
## Checklistprompt-regression
When your agent uses it
Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.
What it loads into context
Frontmatter
99 tokensAlways available to the agent
SKILL.md
2,733 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Load and Analyze Prompts
- Step 2: Derive Comparison Dimensions
- Step 3: Select Graders
- Step 4: Run Position-Debiased Comparison
- Step 5: Compute Statistics
- Step 6: Present Results
- Common Mistakes
- Next Skills
Scripts executed separately
scripts/pairwise.py
Files
SKILL.mdcore11,330 bytesSKILL.mdcore11,330 bytesSKILL.mdcore11,330 bytesSKILL.mdcore11,330 bytesscripts/pairwise.pyruns7,853 bytes
Read SKILL.md
<HARD-GATE>
NO conclusion about which prompt is better WITHOUT bootstrap 95% CI reported.
NO candidate declared "better" WITHOUT position-debiased (swap-aggregate) comparison.
NO comparison with fewer than 10 samples per axis — CI is too wide to be meaningful.
</HARD-GATE>
# Prompt Regression
Compare two prompts head-to-head and determine, with statistical rigor, whether
the candidate is better, worse, or tied on each evaluation dimension.
## When to Activate
- You changed the system prompt and want to verify it's actually betterredteam
When your agent uses it
Use when the user wants to test their LLM/agent application for safety and security vulnerabilities — jailbreaks, prompt injection, PII extraction, harmful content generation, or evaluator gaming. Also use when the user mentions security testing, adversarial testing, red teaming, safety evaluation, ASR (Attack Success Rate), or "is my app safe to deploy." Outputs ASR paired with over-refusal rate and an audit document.
What it loads into context
Frontmatter
103 tokensAlways available to the agent
SKILL.md
3,306 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Establish Policy
- Step 2: Derive Attack Vectors from Policy
- Step 3: Generate Attack and Over-Refusal Sets
- Step 4: Run Attacks
- Step 5: Compute Metrics
- Step 6: Safety Scorecard
- Output Files
- Common Mistakes
- Next Skills
Scripts executed separately
scripts/asr_report.py
Files
SKILL.mdcore14,669 bytesSKILL.mdcore14,669 bytesSKILL.mdcore14,669 bytesSKILL.mdcore14,669 bytesSKILL.mdcore14,669 bytesSKILL.mdcore14,669 bytesSKILL.mdcore14,669 bytesSKILL.mdcore14,669 bytesscripts/asr_report.pyruns7,944 bytes
Read SKILL.md
<HARD-GATE>
NO ASR report WITHOUT paired over-refusal rate measurement.
NO attack vector distribution WITHOUT reading a policy document first.
NO regulated-stakes redteam WITHOUT a sign-off block in the audit document.
</HARD-GATE>
# Redteam
Test your application's safety boundaries systematically. This skill generates
attack prompts from a policy document, measures what gets through, and pairs
the Attack Success Rate (ASR) with the Over-Refusal Rate so you don't reward
models that simply refuse everything.
## When to Activatebootstrap
When your agent uses it
Use when the user has nothing — no traces, no labels, no eval set — and needs to build a v0 evaluation from scratch. Also use when the user says "I need to start evaluating my app but don't know where to begin," "I want to set up eval for a new product," or has just identified failure modes and needs to turn them into principles. Outputs a v0 grader in 30 minutes using OpenJudge SimpleRubricsGenerator, plus a roadmap to reach calibrated evaluation.
What it loads into context
Frontmatter
117 tokensAlways available to the agent
SKILL.md
1,981 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Checklist
- Step 1: Product Interview (One Shot)
- Step 2: Generate v0 Grader
- Step 3: Synthesize Eval Inputs
- Step 4: Run v0 Evaluation
- Step 5: Output Roadmap
- Quick Mode vs Deep Mode
- Red Flags — STOP and Re-evaluate
- Common Mistakes
- Next Skills
Files
SKILL.mdcore8,098 bytesSKILL.mdcore8,098 bytesSKILL.mdcore8,098 bytesSKILL.mdcore8,098 bytesSKILL.mdcore8,098 bytesSKILL.mdcore8,098 bytes
Read SKILL.md
<HARD-GATE>
NO v0 grader deployed WITHOUT explicitly marking it as uncalibrated.
NO synthetic labels — LLM can generate eval inputs, but labels MUST come from real system output + human judgment.
NO principle without a source label documenting where it came from.
</HARD-GATE>
# Bootstrap
Cold-start an evaluation system when you have nothing. In 30 minutes you get a working
v0 grader and a clear path to a calibrated, trustworthy evaluation.
> **Requires OpenJudge** (`pip install py-openjudge`) for the grader generators
> (`SimpleRubricsGenerator` / `IterativeRubricsGenerator`). The interview, stratification,
> and calibration-roadmap methodology is SDK-independent.find-skills-combo
When your agent uses it
Discover and recommend **combinations** of agent skills to complete complex, multi-faceted tasks. Provides two recommendation strategies — **Maximum Quality** (best skill per subtask) and **Minimum Dependencies** (fewest installs). Use this skill whenever the user wants to find skills, asks "how do I do X", "find a skill for X", or describes a task that likely requires multiple capabilities working together. Also use when the user mentions composing workflows, building pipelines, or needs help across several domains at once — even if they only say "find me a skill". This skill supersedes simple single-skill search by decomposing the task into subtasks and assembling an optimal skill portfolio.
What it loads into context
Frontmatter
149 tokensAlways available to the agent
SKILL.md
4,189 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Use This Skill
- What is the Skills CLI?
- The 5-Phase Pipeline
- Fallback: Simple Single-Skill Search
- Common Skill Categories
- Tips
- When No Skills Are Found
Files
SKILL.mdcore18,471 bytesSKILL.mdcore18,471 bytesSKILL.mdcore18,471 bytesSKILL.mdcore18,471 bytesSKILL.mdcore18,471 bytesSKILL.mdcore18,471 bytes
Read SKILL.md
# Find Skills Combo
Discover and install **skill combinations** from the open agent skills ecosystem. Unlike single-skill search, this skill decomposes complex tasks into subtasks, searches for candidates per subtask, evaluates coverage, and recommends two strategies: **Maximum Quality** (best skill per subtask, highest output quality) and **Minimum Dependencies** (fewest installs, lean setup). Users pick the strategy that fits their priorities.
## When to Use This Skill
Use this skill when the user:
- Asks "how do I do X" where X involves multiple capabilities or domains
- Says "find a skill for X" or "is there a skill for X"
- Describes a task that spans several concerns (e.g., "build a quarterly report with charts, risk analysis, and executive summary")
- Wants to compose a workflow from multiple skills
- Asks "can you do X" where X is a complex, multi-step task
- Expresses interest in extending agent capabilities for a non-trivial projectmmx-cli
When your agent uses it
Generate text, images, video, speech, and music via the MiniMax AI platform. Covers text generation (MiniMax-M3 model), image generation (image-01), video generation (Hailuo-2.3), speech synthesis (speech-2.8-hd, 300+ voices), music generation (music-2.6 with lyrics, cover, and instrumental), and web search. Use when the user needs to create AI-generated multimedia content, produce narrated audio from text, compose music, or search the web through MiniMax AI services.
What it loads into context
Frontmatter
129 tokensAlways available to the agent
SKILL.md
697 tokensWhen the skill is activated
Reference documents
0 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- Install
- Capabilities
- Quick Examples
- Gather from user before running
- Tips
Files
SKILL.mdcore2,620 bytesSKILL.mdcore2,620 bytesSKILL.mdcore2,620 bytesSKILL.mdcore2,620 bytesSKILL.mdcore2,620 bytes
Read SKILL.md
# mmx-cli Skill
Generate multimedia content (text, images, video, speech, music) via the
[MiniMax AI](https://github.com/MiniMax-AI/cli) platform.
## Install
```bash
npx skills add MiniMax-AI/cli@skill -g -y
```
Or manually:
```bash01-graders-and-pipeline
When your agent uses it
Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.
What it loads into context
Frontmatter
119 tokensAlways available to the agent
SKILL.md
1,289 tokensWhen the skill is activated
Reference documents
10,247 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Use This Skill
- Sub-documents — Read When Relevant
- Install
- Architecture Overview
- 5-Minute Quick Start
- Key Data Types
- Result Handling Pattern
- Model Configuration
Files
SKILL.mdcore5,277 bytesSKILL.mdcore5,277 bytesSKILL.mdcore5,277 bytesSKILL.mdcore5,277 bytesSKILL.mdcore5,277 bytesanalyzer.mddoc9,171 bytesgenerator.mddoc8,053 bytesgraders.mddoc16,324 bytespipeline.mddoc8,913 bytes
Read SKILL.md
# OpenJudge Skill
Build evaluation pipelines for LLM applications using the `openjudge` library.
## When to Use This Skill
- User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.)
- User wants to compare two or more models and rank them
- User wants to design a scoring rubric and automate evaluation
- User wants to analyze evaluation results statistically
- User wants to build a reward model or quality filter
## Sub-documents — Read When Relevant
02-rl-reward
When your agent uses it
Build RL reward signals using the OpenJudge framework. Covers choosing between pointwise and pairwise reward strategies based on RL algorithm, task type, and cost; aggregating multi-dimensional pointwise scores into a scalar reward; pairwise tournament reward for GRPO on subjective tasks (net win rate across group rollouts); generating preference pairs for DPO/RLAIF; and normalizing scores for training stability. Use when building reward models, scoring rollouts for GRPO/REINFORCE, generating preference data for DPO, or doing Best-of-N selection.
What it loads into context
Frontmatter
136 tokensAlways available to the agent
SKILL.md
1,678 tokensWhen the skill is activated
Reference documents
5,824 tokensLoaded on request
Token counts are estimates. Scripts run separately; they are not counted as reference context.
The playbook
- When to Use This Skill
- Step 1 — Choose Your Reward Strategy
- Sub-documents — Read When Relevant
- Install
- Strategy Comparison
- Score Normalization
- Evaluation Strategies
Files
SKILL.mdcore6,692 bytesSKILL.mdcore6,692 bytesSKILL.mdcore6,692 bytesSKILL.mdcore6,692 bytesSKILL.mdcore6,692 bytespairwise.mddoc13,367 bytespointwise.mddoc9,803 bytes
Read SKILL.md
# RL Reward Construction with OpenJudge
Build reward signals for reinforcement learning from human feedback (RLHF) and
reinforcement learning from AI feedback (RLAIF) using the `openjudge` library.
## When to Use This Skill
- Building scalar rewards for GRPO / REINFORCE rollout scoring
- Generating (chosen, rejected) preference pairs for DPO / IPO
- Best-of-N candidate selection
- Multi-dimensional reward shaping (correctness + safety + format)
- Replacing or bootstrapping a reward model with LLM-as-judge
## Step 1 — Choose Your Reward StrategySkills included
- 00-academic-router
- 01-paper-review
- 02-bib-verify
- 03-ref-hallucination-arena
- 00-arena-router
- 01-auto-arena
- 02-ref-hallucination-arena
- claude-authenticity
- meta-eval
- eval-design
- metric-design
- align-human
- eval-report
- rag-eval
- prompt-regression
- redteam
- bootstrap
- find-skills-combo
- mmx-cli
- 01-graders-and-pipeline
- 02-rl-reward
Repository facts
- License
- Apache-2.0
- Created
- Jul 8, 2025
- Last pushed
- Nov 5, 2026
- Forks
- 101
- Open issues
- 15
- Project activity
- Active
Rank over the last 12 weeks
Now #847 · best #645 · down 87 in 30 days
- #858 on Nov 1
- #859 on Nov 2
- #857 on Nov 3
- #855 on Nov 4
- #851 on Nov 5
- #847 since Nov 6
71 earlier periods
- #744 on Aug 1
- #644 on Aug 7
- #642 on Aug 8
- #647 on Aug 9
- #670 on Aug 10
- #677 on Aug 11
- #669 on Aug 13
- #663 on Aug 14
- #658 on Aug 16
- #671 on Aug 17
- #689 on Aug 18
- #669 on Aug 19
- #672 from Aug 20 to Aug 21
- #645 on Aug 22
- #697 on Aug 24
- #703 on Aug 25
- #683 on Aug 26
- #691 on Aug 27
- #716 on Aug 28
- #661 on Aug 29
- #669 on Aug 30
- #704 on Aug 31
- #703 on Sep 2
- #709 on Sep 3
- #683 on Sep 4
- #674 on Sep 5
- #712 on Sep 7
- #734 on Sep 8
- #719 on Sep 9
- #726 on Sep 10
- #704 on Sep 11
- #703 on Sep 12
- #702 on Sep 13
- #726 on Sep 14
- #721 on Sep 15
- #730 on Sep 16
- #734 on Sep 17
- #715 on Sep 18
- #689 on Sep 19
- #705 on Sep 20
- #725 on Sep 21
- #740 on Sep 23
- #712 on Sep 24
- #696 on Sep 25
- #693 on Sep 26
- #724 on Sep 27
- #736 on Sep 28
- #753 on Sep 30
- #702 on Oct 1
- #703 on Oct 3
- #732 on Oct 4
- #741 on Oct 5
- #749 on Oct 6
- #760 on Oct 7
- #805 on Oct 8
- #771 on Oct 9
- #778 on Oct 12
- #785 on Oct 13
- #797 on Oct 14
- #785 on Oct 15
- #794 on Oct 16
- #755 on Oct 18
- #803 on Oct 19
- #810 on Oct 20
- #819 on Oct 21
- #798 on Oct 22
- #803 on Oct 23
- #932 on Oct 25
- #897 on Oct 28
- #875 on Oct 29
- #864 from Oct 30 to Oct 31
Before you install
Network or broad shellAutomated signals, not a guarantee
Checked Nov 4, 2026
Not fully analyzed: some files were too long, unreadable or missing. This never lowers the level.
Show 74 notes
skills/academic-eval/01-paper-review/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/01-paper-review/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/01-paper-review/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/01-paper-review/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/01-paper-review/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/02-bib-verify/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/02-bib-verify/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/03-ref-hallucination-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/03-ref-hallucination-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/academic-eval/03-ref-hallucination-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/arena-eval/01-auto-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/arena-eval/01-auto-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/arena-eval/01-auto-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/arena-eval/01-auto-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/arena-eval/02-ref-hallucination-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/arena-eval/02-ref-hallucination-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/arena-eval/02-ref-hallucination-arena/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/claude-authenticity/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/claude-authenticity/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzedskills/claude-authenticity/SKILL.md:1Analysis incomplete: Referenced executable target unavailable; not analyzed
- No broad tool permissions
- No dangerous shell commands
- 156 network referencesin 18 files: docs/validating_graders/ref_hallucination_arena.md:12 01-auto-arena/SKILL.md:96 01-paper-review/SKILL.md:131 and 15 more
- 4 obfuscation signalsin 1 file: find-skills-combo/SKILL.md:251
- No instruction overrides
- 5 executable scriptsin 5 files: 01-eval-design/scripts/coverage_check.py:1 03-align-human/scripts/calibration.py:1 05-rag-eval/scripts/rag_diagnostic.py:1 and 2 more
- 121 references to secretsin 17 files: docs/validating_graders/ref_hallucination_arena.md:127 01-paper-review/SKILL.md:38 01-paper-review/reference.md:8 and 14 more
- No requests to vote in Hot
Show all 286 signals by file
docs/validating_graders/ref_hallucination_arena.md
- Network accessline 12
The official evaluation dataset is available on HuggingFace: [**OpenJudge/ref-hallucination-arena**](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena). - Network accessline 12
The official evaluation dataset is available on HuggingFace: [**OpenJudge/ref-hallucination-arena**](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena). - Network accessline 41
The evaluation dataset is hosted on HuggingFace: [**OpenJudge/ref-hallucination-arena**](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena). - Network accessline 41
The evaluation dataset is hosted on HuggingFace: [**OpenJudge/ref-hallucination-arena**](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena). - Network accessline 107
Or download directly from [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena). - Network accessline 107
Or download directly from [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena). - Network accessline 124
# Bare mode (default): direct LLM call model_a: base_url: "https://api.example.com/v1" - Network accessline 124
# Bare mode (default): direct LLM call model_a: base_url: "https://api.example.com/v1" - Network accessline 125
[Credential-related source omitted] - Network accessline 125
[Credential-related source omitted] - Network accessline 126
base_url: "https://api.example.com/v1" - Network accessline 126
base_url: "https://api.example.com/v1" - Access to secretsline 127
[Credential-related source omitted] - Access to secretsline 127
[Credential-related source omitted] - Network accessline 134
model_b: base_url: "https://api.example.com/v1" - Network accessline 134
model_b: base_url: "https://api.example.com/v1" - Network accessline 135
base_url: "https://api.example.com/v1" - Network accessline 135
base_url: "https://api.example.com/v1" - Access to secretsline 136
[Credential-related source omitted] - Access to secretsline 136
[Credential-related source omitted] - Network accessline 143
# Tool-augmented mode: ReAct agent with Tavily web search model_b_with_tools: base_url: "https://api.example.com/v1" - Network accessline 143
# Tool-augmented mode: ReAct agent with Tavily web search model_b_with_tools: base_url: "https://api.example.com/v1" - Network accessline 144
[Credential-related source omitted] - Network accessline 144
[Credential-related source omitted] - Network accessline 145
base_url: "https://api.example.com/v1" - Network accessline 145
base_url: "https://api.example.com/v1" - Access to secretsline 146
[Credential-related source omitted] - Access to secretsline 146
[Credential-related source omitted] - Access to secretsline 152
[Credential-related source omitted] - Access to secretsline 152
[Credential-related source omitted] - Access to secretsline 159
[Credential-related source omitted] - Access to secretsline 159
[Credential-related source omitted] - Network accessline 233
!!! info "Benchmark Leaderboard" For real-world evaluation results on mainstream LLMs, visit the [**OpenJudge Leaderboard**](https://openjudge.me/leaderboard). - Network accessline 233
!!! info "Benchmark Leaderboard" For real-world evaluation results on mainstream LLMs, visit the [**OpenJudge Leaderboard**](https://openjudge.me/leaderboard). - Network accessline 234
For real-world evaluation results on mainstream LLMs, visit the [**OpenJudge Leaderboard**](https://openjudge.me/leaderboard). - Network accessline 234
For real-world evaluation results on mainstream LLMs, visit the [**OpenJudge Leaderboard**](https://openjudge.me/leaderboard). - Access to secretsline 387
[Credential-related source omitted] - Access to secretsline 387
[Credential-related source omitted] - Network accessline 413
# Same model, bare mode (no tools) model_a_bare: base_url: "https://api.example.com/v1" - Network accessline 413
# Same model, bare mode (no tools) model_a_bare: base_url: "https://api.example.com/v1" - Network accessline 414
[Credential-related source omitted] - Network accessline 414
[Credential-related source omitted] - Network accessline 415
base_url: "https://api.example.com/v1" - Network accessline 415
base_url: "https://api.example.com/v1" - Access to secretsline 416
[Credential-related source omitted] - Access to secretsline 416
[Credential-related source omitted] - Network accessline 421
# Same model, tool-augmented mode model_a_with_tools: base_url: "https://api.example.com/v1" - Network accessline 421
# Same model, tool-augmented mode model_a_with_tools: base_url: "https://api.example.com/v1" - Network accessline 422
[Credential-related source omitted] - Network accessline 422
[Credential-related source omitted] - Network accessline 423
base_url: "https://api.example.com/v1" - Network accessline 423
base_url: "https://api.example.com/v1" - Access to secretsline 424
[Credential-related source omitted] - Access to secretsline 424
[Credential-related source omitted] - Access to secretsline 430
[Credential-related source omitted] - Access to secretsline 430
[Credential-related source omitted] - Access to secretsline 438
[Credential-related source omitted] - Access to secretsline 438
[Credential-related source omitted] - Network accessline 452
target_endpoints: my_model: base_url: "https://api.example.com/v1" - Network accessline 452
target_endpoints: my_model: base_url: "https://api.example.com/v1" - Network accessline 453
[Credential-related source omitted] - Network accessline 453
[Credential-related source omitted] - Network accessline 454
base_url: "https://api.example.com/v1" - Network accessline 454
base_url: "https://api.example.com/v1" - Access to secretsline 455
[Credential-related source omitted] - Access to secretsline 455
[Credential-related source omitted]
skills/academic-eval/01-paper-review/SKILL.md
- Access to secretsline 38
[Credential-related source omitted] - Access to secretsline 38
[Credential-related source omitted] - Access to secretsline 93
[Credential-related source omitted] - Access to secretsline 93
[Credential-related source omitted] - Network accessline 131
This pipeline uses [litellm](https://docs.litellm.ai/docs/providers) for model calls. - Network accessline 131
This pipeline uses [litellm](https://docs.litellm.ai/docs/providers) for model calls. - Network accessline 141
| Provider | `--model` value | Env var | Notes | |----------|----------------|---------|-------| | OpenAI | `gpt-5.2`, `gpt-5-mini`, … | `OPENAI_API_KEY` | No prefix needed; `gpt-5.2` is the current flagship vision model; check [OpenAI mode - Network accessline 141
| Provider | `--model` value | Env var | Notes | |----------|----------------|---------|-------| | OpenAI | `gpt-5.2`, `gpt-5-mini`, … | `OPENAI_API_KEY` | No prefix needed; `gpt-5.2` is the current flagship vision model; check [OpenAI mode - Network accessline 142
|----------|----------------|---------|-------| | OpenAI | `gpt-5.2`, `gpt-5-mini`, … | `OPENAI_API_KEY` | No prefix needed; `gpt-5.2` is the current flagship vision model; check [OpenAI models](https://platform.openai.com/docs/models) for - Network accessline 142
|----------|----------------|---------|-------| | OpenAI | `gpt-5.2`, `gpt-5-mini`, … | `OPENAI_API_KEY` | No prefix needed; `gpt-5.2` is the current flagship vision model; check [OpenAI models](https://platform.openai.com/docs/models) for - Network accessline 143
[Credential-related source omitted] - Network accessline 143
[Credential-related source omitted] - Access to secretsline 143
[Credential-related source omitted] - Access to secretsline 143
[Credential-related source omitted] - Network accessline 144
[Credential-related source omitted] - Network accessline 144
[Credential-related source omitted] - Access to secretsline 144
[Credential-related source omitted] - Access to secretsline 144
[Credential-related source omitted] - Access to secretsline 145
[Credential-related source omitted] - Access to secretsline 145
[Credential-related source omitted] - Access to secretsline 146
[Credential-related source omitted] - Access to secretsline 146
[Credential-related source omitted] - Access to secretsline 153
[Credential-related source omitted] - Access to secretsline 153
[Credential-related source omitted] - Access to secretsline 154
[Credential-related source omitted] - Access to secretsline 154
[Credential-related source omitted] - Access to secretsline 155
[Credential-related source omitted] - Access to secretsline 155
[Credential-related source omitted] - Access to secretsline 177
[Credential-related source omitted] - Access to secretsline 177
[Credential-related source omitted] - Network accessline 193
[Credential-related source omitted] - Network accessline 193
[Credential-related source omitted] - Network accessline 194
[Credential-related source omitted] - Network accessline 194
[Credential-related source omitted] - Network accessline 195
[Credential-related source omitted] - Network accessline 195
[Credential-related source omitted]
skills/academic-eval/01-paper-review/reference.md
- Access to secretsline 8
[Credential-related source omitted] - Access to secretsline 8
[Credential-related source omitted] - Access to secretsline 45
[Credential-related source omitted] - Access to secretsline 45
[Credential-related source omitted] - Access to secretsline 46
[Credential-related source omitted] - Access to secretsline 46
[Credential-related source omitted] - Network accessline 47
[Credential-related source omitted] - Network accessline 47
[Credential-related source omitted] - Access to secretsline 47
[Credential-related source omitted] - Access to secretsline 47
[Credential-related source omitted] - Access to secretsline 48
[Credential-related source omitted] - Access to secretsline 48
[Credential-related source omitted] - Network accessline 49
| Local (Ollama) | `ollama/llama3.1` | — (use `--base-url http://localhost:11434`) | - Network accessline 49
| Local (Ollama) | `ollama/llama3.1` | — (use `--base-url http://localhost:11434`) | - Access to secretsline 64
[Credential-related source omitted] - Access to secretsline 64
[Credential-related source omitted] - Access to secretsline 149
[Credential-related source omitted] - Access to secretsline 149
[Credential-related source omitted]
skills/academic-eval/03-ref-hallucination-arena/SKILL.md
- Access to secretsline 40
[Credential-related source omitted] - Network accessline 100
target_endpoints: model_a: base_url: "https://api.openai.com/v1" - Network accessline 101
[Credential-related source omitted] - Network accessline 102
base_url: "https://api.openai.com/v1" - Access to secretsline 103
[Credential-related source omitted] - Network accessline 107
model_b: base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1" - Network accessline 108
base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1" - Access to secretsline 109
[Credential-related source omitted] - Access to secretsline 136
[Credential-related source omitted] - Access to secretsline 142
[Credential-related source omitted] - Access to secretsline 151
[Credential-related source omitted] - Network accessline 206
Official dataset: [OpenJudge/ref-hallucination-arena](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena) - Access to secretsline 249
[Credential-related source omitted] - Access to secretsline 250
[Credential-related source omitted] - Access to secretsline 251
[Credential-related source omitted] - Access to secretsline 252
[Credential-related source omitted] - Access to secretsline 253
[Credential-related source omitted] - Network accessline 257
- Full config examples: [cookbooks/ref_hallucination_arena/examples/](../../../cookbooks/ref_hallucination_arena/examples/) - Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../../docs/validating_graders/ref_hallucina - Network accessline 258
- Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../../docs/validating_graders/ref_hallucination_arena.md) - Official dataset: [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena) - Leader - Network accessline 259
- Official dataset: [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena) - Network accessline 260
- Leaderboard: [openjudge.me/leaderboard](https://openjudge.me/leaderboard)
skills/arena-eval/01-auto-arena/SKILL.md
- Access to secretsline 42
[Credential-related source omitted] - Network accessline 96
target_endpoints={ "gpt4": OpenAIEndpoint( base_url="https://api.openai.com/v1", - Network accessline 97
[Credential-related source omitted] - Network accessline 98
base_url="https://api.openai.com/v1", - Access to secretsline 99
[Credential-related source omitted] - Network accessline 101
), "qwen": OpenAIEndpoint( base_url="https://dashscope.aliyuncs.com/compatible-mode/v1", - Network accessline 102
[Credential-related source omitted] - Network accessline 103
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1", - Access to secretsline 104
[Credential-related source omitted] - Network accessline 107
}, judge_endpoint=OpenAIEndpoint( base_url="https://api.openai.com/v1", - Network accessline 108
[Credential-related source omitted] - Network accessline 109
base_url="https://api.openai.com/v1", - Access to secretsline 110
[Credential-related source omitted] - Network accessline 138
target_endpoints: model_v1: base_url: "https://api.openai.com/v1" - Network accessline 139
[Credential-related source omitted] - Network accessline 140
base_url: "https://api.openai.com/v1" - Access to secretsline 141
[Credential-related source omitted] - Network accessline 142
model: "gpt-4" model_v2: base_url: "https://api.openai.com/v1" - Network accessline 143
[Credential-related source omitted] - Network accessline 144
base_url: "https://api.openai.com/v1" - Access to secretsline 145
[Credential-related source omitted] - Network accessline 148
judge_endpoint: base_url: "https://api.openai.com/v1" - Network accessline 149
base_url: "https://api.openai.com/v1" - Access to secretsline 150
[Credential-related source omitted] - Access to secretsline 168
[Credential-related source omitted] - Access to secretsline 265
[Credential-related source omitted] - Access to secretsline 266
[Credential-related source omitted] - Access to secretsline 267
[Credential-related source omitted] - Access to secretsline 268
[Credential-related source omitted] - Access to secretsline 269
[Credential-related source omitted] - Network accessline 273
- Full config examples: [cookbooks/auto_arena/examples/](../../../cookbooks/auto_arena/examples/) - Documentation: [Auto Arena Guide](https://agentscope-ai.github.io/OpenJudge/applications/auto_arena/) - Network accessline 274
- Documentation: [Auto Arena Guide](https://agentscope-ai.github.io/OpenJudge/applications/auto_arena/)
skills/arena-eval/02-ref-hallucination-arena/SKILL.md
- Access to secretsline 40
[Credential-related source omitted] - Network accessline 100
target_endpoints: model_a: base_url: "https://api.openai.com/v1" - Network accessline 101
[Credential-related source omitted] - Network accessline 102
base_url: "https://api.openai.com/v1" - Access to secretsline 103
[Credential-related source omitted] - Network accessline 107
model_b: base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1" - Network accessline 108
base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1" - Access to secretsline 109
[Credential-related source omitted] - Access to secretsline 136
[Credential-related source omitted] - Access to secretsline 142
[Credential-related source omitted] - Access to secretsline 151
[Credential-related source omitted] - Network accessline 206
Official dataset: [OpenJudge/ref-hallucination-arena](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena) - Access to secretsline 249
[Credential-related source omitted] - Access to secretsline 250
[Credential-related source omitted] - Access to secretsline 251
[Credential-related source omitted] - Access to secretsline 252
[Credential-related source omitted] - Access to secretsline 253
[Credential-related source omitted] - Network accessline 257
- Full config examples: [cookbooks/ref_hallucination_arena/examples/](../../../cookbooks/ref_hallucination_arena/examples/) - Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../../docs/validating_graders/ref_hallucina - Network accessline 258
- Documentation: [docs/validating_graders/ref_hallucination_arena.md](../../../docs/validating_graders/ref_hallucination_arena.md) - Official dataset: [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena) - Leader - Network accessline 259
- Official dataset: [HuggingFace](https://huggingface.co/datasets/OpenJudge/ref-hallucination-arena) - Network accessline 260
- Leaderboard: [openjudge.me/leaderboard](https://openjudge.me/leaderboard)
skills/claude-authenticity/SKILL.md
- Network accessline 26
## The 9 checks (mirrors [claude-verify](https://github.com/molloryn/claude-verify)) - Network accessline 44
| Info | Required? | Notes | |------|-----------|-------| | API endpoint | Yes | Native: `https://xxx/v1/messages` OpenAI-compat: `https://xxx/v1/chat/completions` | - Network accessline 45
|------|-----------|-------| | API endpoint | Yes | Native: `https://xxx/v1/messages` OpenAI-compat: `https://xxx/v1/chat/completions` | | API key | Yes | The key to test | - Network accessline 46
| API endpoint | Yes | Native: `https://xxx/v1/messages` OpenAI-compat: `https://xxx/v1/chat/completions` | - Network accessline 81
# CONFIG — edit here # ============================================================ ENDPOINT = "https://your-provider.com/v1/messages" - Network accessline 82
[Credential-related source omitted] - Network accessline 83
ENDPOINT = "https://your-provider.com/v1/messages" - Access to secretsline 84
[Credential-related source omitted] - Access to secretsline 265
[Credential-related source omitted] - Access to secretsline 270
[Credential-related source omitted] - Access to secretsline 275
[Credential-related source omitted] - Network accessline 279
"thinking": {"budget_tokens": budget, "type": "enabled"}, "messages": [{"role": "user", "content": prompt}]} async with httpx.AsyncClient(timeout=90.0) as client: - Network accessline 280
"messages": [{"role": "user", "content": prompt}]} async with httpx.AsyncClient(timeout=90.0) as client: resp = await client.post(endpoint, headers=headers, json=body) - Network accessline 281
async with httpx.AsyncClient(timeout=90.0) as client: - Access to secretsline 310
[Credential-related source omitted] - Access to secretsline 314
[Credential-related source omitted] - Access to secretsline 351
[Credential-related source omitted] - Access to secretsline 356
[Credential-related source omitted] - Access to secretsline 402
[Credential-related source omitted] - Access to secretsline 423
[Credential-related source omitted]
skills/eval_pipeline/01-eval-design/scripts/coverage_check.py
- Executable scriptline 1
#!/usr/bin/env python3
skills/eval_pipeline/02-metric-design/SKILL.md
- Access to secretsline 346
[Credential-related source omitted] - Network accessline 348
[Credential-related source omitted] - Access to secretsline 348
[Credential-related source omitted] - Network accessline 349
[Credential-related source omitted] - Access to secretsline 349
[Credential-related source omitted] - Network accessline 350
# OPENAI_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
skills/eval_pipeline/03-align-human/scripts/calibration.py
- Executable scriptline 1
#!/usr/bin/env python3
skills/eval_pipeline/05-rag-eval/scripts/rag_diagnostic.py
- Executable scriptline 1
#!/usr/bin/env python3
skills/eval_pipeline/06-prompt-regression/scripts/pairwise.py
- Executable scriptline 1
#!/usr/bin/env python3
skills/eval_pipeline/07-redteam/scripts/asr_report.py
- Executable scriptline 1
#!/usr/bin/env python3
skills/eval_pipeline/08-bootstrap/SKILL.md
- Network accessline 68
[Credential-related source omitted] - Access to secretsline 68
[Credential-related source omitted] - Network accessline 69
# For Aliyun DashScope (Bailian): set OPENAI_BASE_URL to # https://dashscope.aliyuncs.com/compatible-mode/v1 and OPENAI_API_KEY to your key. model = OpenAIChatModel(model="qwen-plus") # or "gpt-4o", etc. - Network accessline 70
[Credential-related source omitted] - Access to secretsline 70
[Credential-related source omitted]
skills/find-skills-combo/SKILL.md
- Network accessline 35
**Browse skills at:** https://skills.sh/ - Possible obfuscationline 251
<U+200B>```bash - Possible obfuscationline 255
<U+200B>``` - Possible obfuscationline 270
<U+200B>```bash - Possible obfuscationline 272
<U+200B>``` - Network accessline 323
3. **Parallel search matters**: Running all keyword groups simultaneously saves significant time. Use subagents when available. 4. **Don't over-decompose**: 3–5 subtasks is the sweet spot for most tasks. More than that creates noise. 5. **S - Network accessline 324
4. **Don't over-decompose**: 3–5 subtasks is the sweet spot for most tasks. More than that creates noise. 5. **Skills.sh is your friend**: When evaluating candidates, quickly check `https://skills.sh/<owner>/<repo>/<skill-name>` for descrip - Network accessline 325
5. **Skills.sh is your friend**: When evaluating candidates, quickly check `https://skills.sh/<owner>/<repo>/<skill-name>` for descriptions.
skills/mmx-cli/SKILL.md
- Network accessline 14
Generate multimedia content (text, images, video, speech, music) via the [MiniMax AI](https://github.com/MiniMax-AI/cli) platform. - Network accessline 15
[MiniMax AI](https://github.com/MiniMax-AI/cli) platform. - Access to secretsline 32
[Credential-related source omitted] - Network accessline 70
| Info | Required? | Notes | |------|-----------|-------| | MINIMAX_API_KEY | Yes | From [MiniMax platform](https://www.minimaxi.com/) | - Network accessline 71
|------|-----------|-------| | MINIMAX_API_KEY | Yes | From [MiniMax platform](https://www.minimaxi.com/) | | Prompt or text | Yes | Describe what to generate | - Network accessline 72
[Credential-related source omitted] - Access to secretsline 72
[Credential-related source omitted] - Network accessline 80
- Use `--non-interactive` and `--quiet` flags in automated / agent workflows. - Run `mmx --help` or `mmx <subcommand> --help` to see all options. - The [SKILL.md](https://github.com/MiniMax-AI/cli/blob/main/skill/SKILL.md) - Network accessline 81
- Run `mmx --help` or `mmx <subcommand> --help` to see all options. - The [SKILL.md](https://github.com/MiniMax-AI/cli/blob/main/skill/SKILL.md) in the upstream repo is always up-to-date with the latest flags and models. - Network accessline 82
- The [SKILL.md](https://github.com/MiniMax-AI/cli/blob/main/skill/SKILL.md)
skills/openjudge-core/01-graders-and-pipeline/SKILL.md
- Network accessline 74
[Credential-related source omitted] - Network accessline 75
[Credential-related source omitted] - Access to secretsline 75
[Credential-related source omitted] - Network accessline 76
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1", - Access to secretsline 147
[Credential-related source omitted] - Access to secretsline 150
[Credential-related source omitted] - Network accessline 155
[Credential-related source omitted] - Network accessline 156
[Credential-related source omitted] - Access to secretsline 156
[Credential-related source omitted] - Network accessline 157
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
skills/openjudge-core/01-graders-and-pipeline/analyzer.md
skills/openjudge-core/01-graders-and-pipeline/generator.md
skills/openjudge-core/01-graders-and-pipeline/graders.md
- Access to secretsline 202
[Credential-related source omitted] - Network accessline 204
result = await grader.aevaluate( query="A red apple on a wooden table", response=MLLMImage(url="https://example.com/image.jpg"), - Network accessline 205
query="A red apple on a wooden table", response=MLLMImage(url="https://example.com/image.jpg"), ) - Network accessline 206
response=MLLMImage(url="https://example.com/image.jpg"), - Access to secretsline 362
[Credential-related source omitted]
skills/openjudge-core/01-graders-and-pipeline/pipeline.md
skills/openjudge-core/02-rl-reward/pairwise.md
- Network accessline 45
[Credential-related source omitted] - Network accessline 46
[Credential-related source omitted] - Access to secretsline 46
[Credential-related source omitted] - Network accessline 47
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1", - Network accessline 139
[Credential-related source omitted] - Network accessline 140
[Credential-related source omitted] - Access to secretsline 140
[Credential-related source omitted] - Network accessline 141
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
skills/openjudge-core/02-rl-reward/pointwise.md
- Network accessline 48
[Credential-related source omitted] - Access to secretsline 48
[Credential-related source omitted] - Network accessline 49
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1") - Network accessline 88
[Credential-related source omitted] - Access to secretsline 88
[Credential-related source omitted] - Network accessline 89
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1")
How the signals fit together
Facts from a static read of the files at the pinned commit. They describe what the code does, not why.
Some files could not be read; the facts below still hold.
Signal confidence: 286 facts
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:12Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:12Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:41Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:41Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:107Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:107Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:124Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:124Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:125Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:125Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:126Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:126Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:127Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:127Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:134Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:134Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:135Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:135Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:136Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:136Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:143Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:143Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:144Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:144Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:145Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:145Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:146Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:146Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:152Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:152Low confidence
- Access to secrets in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:159Low confidence
- Access to secrets in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:159Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:233Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:233Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:234Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:234Low confidence
- Access to secrets in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:387Low confidence
- Access to secrets in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:387Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:413Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:413Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:414Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:414Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:415Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:415Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:416Low confidence
- Access to secrets in prose of documentation: a variable, placeholder or call, not a stored valuedocs/validating_graders/ref_hallucination_arena.md:416Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:421Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:421Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:422Low confidence
- Network access in prose of documentationdocs/validating_graders/ref_hallucination_arena.md:422Low confidence
and 236 more
Works with
| Agent | Status | Evidence | Install to |
|---|---|---|---|
| Claude Code | Compatible | Standard SKILL.md skills/academic-eval/00-academic-router/SKILL.md | .claude/skills/<skill-name>/ |
| Codex | Compatible | Standard SKILL.md skills/academic-eval/00-academic-router/SKILL.md | .agents/skills/<skill-name>/ |
| Cursor | Compatible | Standard SKILL.md skills/academic-eval/00-academic-router/SKILL.md | .cursor/skills/<skill-name>/ |
| Gemini CLI | Compatible | Standard SKILL.md skills/academic-eval/00-academic-router/SKILL.md | .gemini/skills/<skill-name>/ |
| GitHub Copilot | Compatible | Standard SKILL.md skills/academic-eval/00-academic-router/SKILL.md | .github/skills/<skill-name>/ |
| OpenCode | Compatible | Standard SKILL.md skills/academic-eval/00-academic-router/SKILL.md | .opencode/skills/<skill-name>/ |
- Native The repository ships a manifest, folder or frontmatter field for this agent.
- Compatible A standard SKILL.md that this agent can read.
- Unknown No evidence either way.
Derived automatically from paths, manifests and frontmatter. Not tested on each agent.
Explore alternatives and comparisons
Add this badge to your README
It updates every night and links to this page.
[](https://hotskills.dev/skill/agentscope-ai-openjudge/)