Individual skill
nemo-evaluator-sdk
Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
plugins/KalarisLabs/research-agent-skills/skills/nemo-evaluator-sdk/SKILL.md
Source commit d780783eba4ecc7c8b9429f95c9362df95f07c61
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- Yes
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Network or broad shell
- Last checked
- Nov 4, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 1,537
- Repository last updated
- Nov 7, 2026
- License
- Apache-2.0
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- ai-coding Repository topic
- awesome Repository topic
- awesome-list Repository topic
- codex Repository topic
- codex-plugins Repository topic
- openai Repository topic
- codex-skill Repository topic
- codex-skills Repository topic
- marketplace Repository topic
- codex-marketplace Repository topic
- engineering Derived category
- devops Derived category
- devops-containers Derived category
Declared tags are partial or unavailable.
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Browse files
SKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesSKILL.mdcore · 13,668 bytesreferences/adapter-system.mddoc · 9,468 bytesreferences/configuration.mddoc · 9,567 bytesreferences/custom-benchmarks.mddoc · 7,101 bytesreferences/execution-backends.mddoc · 9,351 bytesreferences/python-api-usage.mddoc · 887 bytes
Read excerpt
# NeMo Evaluator SDK - Enterprise LLM Benchmarking
## Quick Start
NeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud).
**Installation**:
```bash
pip install nemo-evaluator-launcher
```
**Set API key and run evaluation**:
```bash
[Credential-related source omitted]