Individual skill
colm-experiments
Use when designing or auditing the empirical core of a COLM paper — contamination analysis for evaluation data, fair baselines under matched prompting and compute, pinned model versions and decoding parameters, uncertainty over runs and samples, scaling coverage, and honest reporting of API-model comparisons.
COLM-Skills/skills/colm-experiments/SKILL.md
Source commit 932eb23331721b16e967c59f093781be1928b498
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- Yes
- Repository scan complete
- No
- Skill scan complete
- Yes
- Observed behavior
- Network or broad shell
- Last checked
- Nov 9, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 1,400
- Repository last updated
- Sep 27, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- academic-writing Repository topic
- agent-skills Repository topic
- journal Repository topic
- academic-research Repository topic
- ai-agents Repository topic
- anthropic Repository topic
- awesome-list Repository topic
- claude Repository topic
- claude-code Repository topic
- economics Repository topic
- data Derived category
- research Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 6,581 bytes
Read excerpt
# COLM Experiments
COLM's reviewer pool works on language models daily, which changes what "rigorous"
means: the failure modes they hunt are not missing error bars alone but the
LM-specific ones — leaked test sets, un-tuned baselines, drifting API models, and
decoding settings that quietly decide the result. Build the evaluation so those four
attacks fail.
## The four LM-specific attacks on your results
**1. Contamination.** Any public benchmark plausibly overlaps the pre-training data
of any recent model. For each evaluation set, either (a) run and report an overlap
analysis against known corpora, (b) use dated or freshly generated items and say so,
or (c) state explicitly that contamination could not be assessed and bound what it