Not in the top 1,000 by GitHub stars

strands-agents/evals

A comprehensive evaluation framework for AI agents and LLM applications.

History

Review weekly changes to SKILL.md before updating. Dates mark the weeks in which we observed each version; checks apply to the recorded commits.

View weekly changes

`SKILL.md`

Week of Nov 2, 2026 · +0 / −0 lines

Before Network or broad shell → After Network or broad shell

32e72bb0639f → dacef5a7d07a

Compare on GitHub

Week of Oct 26, 2026 · Line counts unavailable

Before Not checked yet → After Network or broad shell

First observation → 32e72bb0639f

No earlier version was observed for comparison.

Compare on GitHub

View on GitHub Add to my skills

Verified commit dacef5a7d07a48b064981d1b9339942c5c22f44d

Version-pinned installation · Codex

git clone --filter=blob:none --no-checkout https://github.com/strands-agents/evals.git .hotskills/strands-agents-evals && git -C .hotskills/strands-agents-evals checkout --detach dacef5a7d07a48b064981d1b9339942c5c22f44d
# Copy each selected skill directory from .hotskills/strands-agents-evals into .agents/skills/<skill-name>/

251GitHub stars

1star in 7 days

1skill in this repository

17commits in 30 days

Heat

2.9

Star growth, last 7 daysNot available
Star growth, last 24 hoursNot available
Rank movement, last 7 daysNot available
Commits, last 30 daysNot available
Accounts asking about it through MCP, last 7 daysNot available
Hot votes this weekNot available

Heat never changes the order by stars, and sponsorship never touches it. How heat works

Stars over time

30-day change not yet available: not enough history.

Individual skills in this repository

strands-evals /

How it works

strands-evals

When your agent uses it

Use when authoring evaluations with strands-agents-evals. Activates on tasks involving Case/Experiment construction, picking evaluators (Output, Trajectory, Helpfulness, Faithfulness, Coherence, Conciseness, ResponseRelevance, Harmfulness, Refusal, Stereotyping, InstructionFollowing, GoalSuccessRate, ToolSelection/ParameterAccuracy, Multimodal*), trace-based evaluation with mappers (CloudWatch, OpenSearch, OpenInference, LangChain OTel, Strands in-memory), simulators (ActorSimulator, ToolSimulator), failure detection and root-cause analysis (detect_failures, analyze_root_cause, diagnose_session), chaos / fault-injection testing (ChaosCase, ChaosExperiment, ChaosPlugin), red-team evaluation (RedTeamExperiment, AdversarialCaseGenerator, AttackSuccessEvaluator, attack strategies), or auto test-case generation (ExperimentGenerator). Trigger phrases include "evaluate this agent", "score the trajectory", "simulate a user", "diagnose the session", "generate test cases", "LLM-as-a-Judge", "inject tool failures", "chaos test", "red team this agent", "jailbreak / adversarial". Skip for general LLM eval theory unrelated to this package.

What it loads into context

Frontmatter

266 tokens

Always available to the agent

SKILL.md

5,995 tokens

When the skill is activated

Reference documents

26,916 tokens

Loaded on request

Token counts are estimates. Scripts run separately; they are not counted as reference context.

The playbook

Routing Map
Core Building Blocks
OutputEvaluator (rubric-based LLM-as-judge)
TrajectoryEvaluator
Trace-Based Evaluators (need a Session)
Multimodal (image-to-text)
Deterministic Evaluators (no LLM)
ActorSimulator (multi-turn user simulation)
ToolSimulator (LLM-powered fake tools)
Detectors (failure detection + RCA)
Chaos Testing (deterministic fault injection)
Red Team (adversarial evaluation)
ExperimentGenerator (auto test-case generation)
Custom Evaluator
Async Execution
Result Caching
Authoring Best Practices
Type Annotations
Pointers

Scripts executed separately

.github/scripts/group-release-notes.py
src/__init__.py
src/strands_evals/__init__.py
src/strands_evals/__main__.py
src/strands_evals/_async.py
src/strands_evals/case.py
src/strands_evals/chaos/__init__.py
src/strands_evals/chaos/_context.py
src/strands_evals/chaos/case.py
src/strands_evals/chaos/effects.py
src/strands_evals/chaos/experiment.py
src/strands_evals/chaos/plugin.py
src/strands_evals/cli/__init__.py
src/strands_evals/cli/__main__.py
src/strands_evals/cli/_agent_task.py
src/strands_evals/cli/_common.py
src/strands_evals/cli/_entrypoint.py
src/strands_evals/cli/_io.py
src/strands_evals/cli/commands/__init__.py
src/strands_evals/cli/commands/diagnose.py
src/strands_evals/cli/commands/fetch.py
src/strands_evals/cli/commands/generate.py
src/strands_evals/cli/commands/report.py
src/strands_evals/cli/commands/run.py
src/strands_evals/cli/commands/validate.py
src/strands_evals/cli/main.py
src/strands_evals/detectors/__init__.py
src/strands_evals/detectors/chunking.py
src/strands_evals/detectors/constants.py
src/strands_evals/detectors/diagnosis.py
src/strands_evals/detectors/failure_detector.py
src/strands_evals/detectors/prompt_templates/__init__.py
src/strands_evals/detectors/prompt_templates/failure_detection/__init__.py
src/strands_evals/detectors/prompt_templates/failure_detection/failure_detection_v0.py
src/strands_evals/detectors/prompt_templates/root_cause/__init__.py
src/strands_evals/detectors/prompt_templates/root_cause/root_cause_merge_v0.py
src/strands_evals/detectors/prompt_templates/root_cause/root_cause_v0.py
src/strands_evals/detectors/root_cause_analyzer.py
src/strands_evals/detectors/utils.py
src/strands_evals/display/display_console.py
src/strands_evals/eval_task_handler.py
src/strands_evals/evaluation_data_store.py
src/strands_evals/evaluators/__init__.py
src/strands_evals/evaluators/chaos/__init__.py
src/strands_evals/evaluators/chaos/failure_communication_evaluator.py
src/strands_evals/evaluators/chaos/partial_completion_evaluator.py
src/strands_evals/evaluators/chaos/prompt_templates/__init__.py
src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/__init__.py
src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/failure_communication_v0.py
src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/__init__.py
src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/partial_completion_v0.py
src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/__init__.py
src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/recovery_strategy_v0.py
src/strands_evals/evaluators/chaos/recovery_strategy_evaluator.py
src/strands_evals/evaluators/coherence_evaluator.py
src/strands_evals/evaluators/conciseness_evaluator.py
src/strands_evals/evaluators/correctness_evaluator.py
src/strands_evals/evaluators/deterministic/__init__.py
src/strands_evals/evaluators/deterministic/environment_state.py
src/strands_evals/evaluators/deterministic/output.py
src/strands_evals/evaluators/deterministic/skill_invoked.py
src/strands_evals/evaluators/deterministic/structured_output.py
src/strands_evals/evaluators/deterministic/trajectory.py
src/strands_evals/evaluators/evaluator.py
src/strands_evals/evaluators/faithfulness_evaluator.py
src/strands_evals/evaluators/goal_success_rate_evaluator.py
src/strands_evals/evaluators/harmfulness_evaluator.py
src/strands_evals/evaluators/helpfulness_evaluator.py
src/strands_evals/evaluators/instruction_following_evaluator.py
src/strands_evals/evaluators/interactions_evaluator.py
src/strands_evals/evaluators/multimodal_correctness_evaluator.py
src/strands_evals/evaluators/multimodal_faithfulness_evaluator.py
src/strands_evals/evaluators/multimodal_instruction_following_evaluator.py
src/strands_evals/evaluators/multimodal_output_evaluator.py
src/strands_evals/evaluators/multimodal_overall_quality_evaluator.py
src/strands_evals/evaluators/output_evaluator.py
src/strands_evals/evaluators/prompt_templates/case_prompt_template.py
src/strands_evals/evaluators/prompt_templates/coherence/__init__.py
src/strands_evals/evaluators/prompt_templates/coherence/coherence_v0.py
src/strands_evals/evaluators/prompt_templates/conciseness/__init__.py
src/strands_evals/evaluators/prompt_templates/conciseness/conciseness_v0.py
src/strands_evals/evaluators/prompt_templates/correctness/__init__.py
src/strands_evals/evaluators/prompt_templates/correctness/correctness_v0.py
src/strands_evals/evaluators/prompt_templates/correctness/correctness_with_reference_v0.py
src/strands_evals/evaluators/prompt_templates/faithfulness/__init__.py
src/strands_evals/evaluators/prompt_templates/faithfulness/faithfulness_v0.py
src/strands_evals/evaluators/prompt_templates/goal_success_rate/__init__.py
src/strands_evals/evaluators/prompt_templates/goal_success_rate/goal_success_rate_v0.py
src/strands_evals/evaluators/prompt_templates/goal_success_rate/goal_success_rate_with_assertions_v0.py
src/strands_evals/evaluators/prompt_templates/harmfulness/__init__.py
src/strands_evals/evaluators/prompt_templates/harmfulness/harmfulness_v0.py
src/strands_evals/evaluators/prompt_templates/helpfulness/__init__.py
src/strands_evals/evaluators/prompt_templates/helpfulness/helpfulness_v0.py
src/strands_evals/evaluators/prompt_templates/instruction_following/__init__.py
src/strands_evals/evaluators/prompt_templates/instruction_following/instruction_following_v0.py
src/strands_evals/evaluators/prompt_templates/multimodal/__init__.py
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/__init__.py
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/correctness_v0.py
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/faithfulness_v0.py
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/instruction_following_v0.py
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/overall_quality_v0.py
src/strands_evals/evaluators/prompt_templates/multimodal_case_prompt_template.py
src/strands_evals/evaluators/prompt_templates/multimodal_judge_system_prompt.py
src/strands_evals/evaluators/prompt_templates/prompt_templates.py
src/strands_evals/evaluators/prompt_templates/refusal/__init__.py
src/strands_evals/evaluators/prompt_templates/refusal/refusal_v0.py
src/strands_evals/evaluators/prompt_templates/response_relevance/__init__.py
src/strands_evals/evaluators/prompt_templates/response_relevance/response_relevance_v0.py
src/strands_evals/evaluators/prompt_templates/skill_instruction_following/__init__.py
src/strands_evals/evaluators/prompt_templates/skill_instruction_following/skill_instruction_following_v0.py
src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/__init__.py
src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/skill_selection_accuracy_v0.py
src/strands_evals/evaluators/prompt_templates/stereotyping/__init__.py
src/strands_evals/evaluators/prompt_templates/stereotyping/stereotyping_v0.py
src/strands_evals/evaluators/prompt_templates/tool_parameter_accuracy/__init__.py
src/strands_evals/evaluators/prompt_templates/tool_parameter_accuracy/tool_parameter_accuracy_v0.py
src/strands_evals/evaluators/prompt_templates/tool_selection_accuracy/__init__.py
src/strands_evals/evaluators/prompt_templates/tool_selection_accuracy/tool_selection_accuracy_v0.py
src/strands_evals/evaluators/prompt_templates/trajectory_prompt_template.py
src/strands_evals/evaluators/redteam/__init__.py
src/strands_evals/evaluators/redteam/attack_success_evaluator.py
src/strands_evals/evaluators/redteam/prompt_templates/__init__.py
src/strands_evals/evaluators/redteam/prompt_templates/attack_success/__init__.py
src/strands_evals/evaluators/redteam/prompt_templates/attack_success/attack_success_v0.py
src/strands_evals/evaluators/refusal_evaluator.py
src/strands_evals/evaluators/response_relevance_evaluator.py
src/strands_evals/evaluators/skill_instruction_following_evaluator.py
src/strands_evals/evaluators/skill_selection_accuracy_evaluator.py
src/strands_evals/evaluators/stereotyping_evaluator.py
src/strands_evals/evaluators/tool_parameter_accuracy_evaluator.py
src/strands_evals/evaluators/tool_selection_accuracy_evaluator.py
src/strands_evals/evaluators/trajectory_evaluator.py
src/strands_evals/experiment.py
src/strands_evals/experimental/__init__.py
src/strands_evals/extractors/__init__.py
src/strands_evals/extractors/graph_extractor.py
src/strands_evals/extractors/skills/__init__.py
src/strands_evals/extractors/skills/_normalize.py
src/strands_evals/extractors/skills/_patterns.py
src/strands_evals/extractors/skills/adapters/__init__.py
src/strands_evals/extractors/skills/adapters/_common.py
src/strands_evals/extractors/skills/adapters/claude.py
src/strands_evals/extractors/skills/adapters/codex.py
src/strands_evals/extractors/skills/adapters/gemini.py
src/strands_evals/extractors/skills/adapters/openhands.py
src/strands_evals/extractors/skills/adapters/registry.py
src/strands_evals/extractors/skills/adapters/strands.py
src/strands_evals/extractors/skills/extractor.py
src/strands_evals/extractors/skills/models.py
src/strands_evals/extractors/swarm_extractor.py
src/strands_evals/extractors/tools_use_extractor.py
src/strands_evals/extractors/trace_extractor.py
src/strands_evals/generators/__init__.py
src/strands_evals/generators/experiment_generator.py
src/strands_evals/generators/prompt_template/prompt_templates.py
src/strands_evals/generators/topic_planner.py
src/strands_evals/local_file_task_result_store.py
src/strands_evals/mappers/__init__.py
src/strands_evals/mappers/adk_otel_session_mapper.py
src/strands_evals/mappers/cloudwatch_parser.py
src/strands_evals/mappers/cloudwatch_session_mapper.py
src/strands_evals/mappers/constants.py
src/strands_evals/mappers/generic_gen_ai_session_mapper.py
src/strands_evals/mappers/langchain_otel_session_mapper.py
src/strands_evals/mappers/openai_agents_gen_ai_session_mapper.py
src/strands_evals/mappers/openinference_session_mapper.py
src/strands_evals/mappers/opensearch_session_mapper.py
src/strands_evals/mappers/session_mapper.py
src/strands_evals/mappers/strands_in_memory_session_mapper.py
src/strands_evals/mappers/utils.py
src/strands_evals/providers/__init__.py
src/strands_evals/providers/cloudwatch_provider.py
src/strands_evals/providers/exceptions.py
src/strands_evals/providers/langfuse_provider.py
src/strands_evals/providers/opensearch_provider.py
src/strands_evals/providers/trace_provider.py
src/strands_evals/redteam/__init__.py
src/strands_evals/redteam/case.py
src/strands_evals/redteam/experiment.py
src/strands_evals/redteam/generators/__init__.py
src/strands_evals/redteam/generators/adversarial.py
src/strands_evals/redteam/generators/prompt_templates/__init__.py
src/strands_evals/redteam/generators/prompt_templates/adversarial_v0.py
src/strands_evals/redteam/report.py
src/strands_evals/redteam/strategies/__init__.py
src/strands_evals/redteam/strategies/_common.py
src/strands_evals/redteam/strategies/bad_likert_judge/__init__.py
src/strands_evals/redteam/strategies/bad_likert_judge/bad_likert_judge_v0.py
src/strands_evals/redteam/strategies/base.py
src/strands_evals/redteam/strategies/crescendo/__init__.py
src/strands_evals/redteam/strategies/crescendo/crescendo_v0.py
src/strands_evals/redteam/strategies/goat/__init__.py
src/strands_evals/redteam/strategies/goat/goat_v0.py
src/strands_evals/redteam/strategies/pair/__init__.py
src/strands_evals/redteam/strategies/pair/pair_v0.py
src/strands_evals/redteam/strategies/prompt_strategy/__init__.py
src/strands_evals/redteam/strategies/prompt_strategy/gradual_escalation/__init__.py
src/strands_evals/redteam/strategies/prompt_strategy/gradual_escalation/gradual_escalation_v0.py
src/strands_evals/redteam/strategies/sequentialbreak/__init__.py
src/strands_evals/redteam/strategies/sequentialbreak/sequentialbreak_v0.py
src/strands_evals/redteam/strategies/target_session.py
src/strands_evals/redteam/task.py
src/strands_evals/redteam/utils.py
src/strands_evals/simulation/__init__.py
src/strands_evals/simulation/actor_simulator.py
src/strands_evals/simulation/profiles/__init__.py
src/strands_evals/simulation/profiles/actor_profile.py
src/strands_evals/simulation/prompt_templates/__init__.py
src/strands_evals/simulation/prompt_templates/actor_profile_extraction.py
src/strands_evals/simulation/prompt_templates/actor_system_prompt.py
src/strands_evals/simulation/prompt_templates/goal_completion.py
src/strands_evals/simulation/prompt_templates/tool_response_generation.py
src/strands_evals/simulation/tool_simulator.py
src/strands_evals/simulation/tools/__init__.py
src/strands_evals/simulation/tools/goal_completion.py
src/strands_evals/telemetry/__init__.py
src/strands_evals/telemetry/_cloudwatch_logger.py
src/strands_evals/telemetry/config.py
src/strands_evals/telemetry/tracer.py
src/strands_evals/tools/evaluation_tools.py
src/strands_evals/types/__init__.py
src/strands_evals/types/detector.py
src/strands_evals/types/evaluation.py
src/strands_evals/types/evaluation_report.py
src/strands_evals/types/multimodal.py
src/strands_evals/types/redteam/__init__.py
src/strands_evals/types/redteam/config.py
src/strands_evals/types/redteam/risk_category.py
src/strands_evals/types/redteam/run_results.py
src/strands_evals/types/simulation/__init__.py
src/strands_evals/types/simulation/actor.py
src/strands_evals/types/simulation/tool.py
src/strands_evals/types/trace.py
src/strands_evals/utils.py
tests/__init__.py
tests/strands_evals/__init__.py
tests/strands_evals/chaos/__init__.py
tests/strands_evals/chaos/test_case.py
tests/strands_evals/chaos/test_context.py
tests/strands_evals/chaos/test_effects.py
tests/strands_evals/chaos/test_experiment.py
tests/strands_evals/chaos/test_failure_communication_evaluator.py
tests/strands_evals/chaos/test_partial_completion_evaluator.py
tests/strands_evals/chaos/test_plugin.py
tests/strands_evals/chaos/test_recovery_strategy_evaluator.py
tests/strands_evals/cli/__init__.py
tests/strands_evals/cli/conftest.py
tests/strands_evals/cli/fixtures/__init__.py
tests/strands_evals/cli/fixtures/agents.py
tests/strands_evals/cli/fixtures/evaluators.py
tests/strands_evals/cli/fixtures/tasks.py
tests/strands_evals/cli/test_agent_task.py
tests/strands_evals/cli/test_diagnose.py
tests/strands_evals/cli/test_entrypoint.py
tests/strands_evals/cli/test_fetch.py
tests/strands_evals/cli/test_generate.py
tests/strands_evals/cli/test_report.py
tests/strands_evals/cli/test_run.py
tests/strands_evals/cli/test_validate.py
tests/strands_evals/cloudwatch_helpers.py
tests/strands_evals/detectors/__init__.py
tests/strands_evals/detectors/test_chunking.py
tests/strands_evals/detectors/test_diagnosis.py
tests/strands_evals/detectors/test_failure_detector.py
tests/strands_evals/detectors/test_root_cause_analyzer.py
tests/strands_evals/detectors/test_types.py
tests/strands_evals/evaluators/deterministic/__init__.py
tests/strands_evals/evaluators/deterministic/test_environment_state.py
tests/strands_evals/evaluators/deterministic/test_output.py
tests/strands_evals/evaluators/deterministic/test_skill_invoked.py
tests/strands_evals/evaluators/deterministic/test_structured_output.py
tests/strands_evals/evaluators/deterministic/test_trajectory.py
tests/strands_evals/evaluators/redteam/test_attack_success_evaluator.py
tests/strands_evals/evaluators/test_coherence_evaluator.py
tests/strands_evals/evaluators/test_conciseness_evaluator.py
tests/strands_evals/evaluators/test_correctness_evaluator.py
tests/strands_evals/evaluators/test_evaluator.py
tests/strands_evals/evaluators/test_faithfulness_evaluator.py
tests/strands_evals/evaluators/test_goal_success_rate_evaluator.py
tests/strands_evals/evaluators/test_harmfulness_evaluator.py
tests/strands_evals/evaluators/test_helpfulness_evaluator.py
tests/strands_evals/evaluators/test_instruction_following_evaluator.py
tests/strands_evals/evaluators/test_interactions_evaluator.py
tests/strands_evals/evaluators/test_multimodal_case_prompt_template.py
tests/strands_evals/evaluators/test_multimodal_output_evaluator.py
tests/strands_evals/evaluators/test_multimodal_specialized_evaluators.py
tests/strands_evals/evaluators/test_output_evaluator.py
tests/strands_evals/evaluators/test_refusal_evaluator.py
tests/strands_evals/evaluators/test_response_relevance_evaluator.py
tests/strands_evals/evaluators/test_skill_instruction_following_evaluator.py
tests/strands_evals/evaluators/test_skill_selection_accuracy_evaluator.py
tests/strands_evals/evaluators/test_stereotyping_evaluator.py
tests/strands_evals/evaluators/test_tool_parameter_accuracy_evaluator.py
tests/strands_evals/evaluators/test_tool_selection_accuracy_evaluator.py
tests/strands_evals/evaluators/test_trajectory_evaluator.py
tests/strands_evals/evaluators/test_trajectory_prompt_template.py
tests/strands_evals/experimental/__init__.py
tests/strands_evals/extractors/__init__.py
tests/strands_evals/extractors/fixtures/capture_skill_fixtures.py
tests/strands_evals/extractors/skill_fixtures.py
tests/strands_evals/extractors/test_graph_extractor.py
tests/strands_evals/extractors/test_skills.py
tests/strands_evals/extractors/test_swarm_extractor.py
tests/strands_evals/extractors/test_tools_use_extractor.py
tests/strands_evals/extractors/test_trace_extractor.py
tests/strands_evals/generators/test_experiment_generator.py
tests/strands_evals/generators/test_topic_planner.py
tests/strands_evals/mappers/__init__.py
tests/strands_evals/mappers/test_adk_otel_session_mapper.py
tests/strands_evals/mappers/test_cloudwatch_parser.py
tests/strands_evals/mappers/test_cloudwatch_session_mapper.py
tests/strands_evals/mappers/test_generic_gen_ai_session_mapper.py
tests/strands_evals/mappers/test_langchain_mapper_integration.py
tests/strands_evals/mappers/test_langchain_otel_session_mapper.py
tests/strands_evals/mappers/test_openai_agents_gen_ai_session_mapper.py
tests/strands_evals/mappers/test_openinference_session_mapper.py
tests/strands_evals/mappers/test_opensearch_session_mapper.py
tests/strands_evals/mappers/test_session_mapper.py
tests/strands_evals/mappers/test_strands_in_memory_mapper.py
tests/strands_evals/mappers/test_utils.py
tests/strands_evals/opensearch_helpers.py
tests/strands_evals/providers/__init__.py
tests/strands_evals/providers/test_cloudwatch_provider.py
tests/strands_evals/providers/test_langfuse_provider.py
tests/strands_evals/providers/test_opensearch_provider.py
tests/strands_evals/providers/test_trace_provider.py
tests/strands_evals/redteam/__init__.py
tests/strands_evals/redteam/test_bad_likert_judge.py
tests/strands_evals/redteam/test_case.py
tests/strands_evals/redteam/test_crescendo.py
tests/strands_evals/redteam/test_experiment.py
tests/strands_evals/redteam/test_generator.py
tests/strands_evals/redteam/test_goat.py
tests/strands_evals/redteam/test_multi_agent_session.py
tests/strands_evals/redteam/test_pair.py
tests/strands_evals/redteam/test_redteam_e2e.py
tests/strands_evals/redteam/test_report.py
tests/strands_evals/redteam/test_sequentialbreak.py
tests/strands_evals/redteam/test_strategies.py
tests/strands_evals/redteam/test_target_session.py
tests/strands_evals/redteam/test_task.py
tests/strands_evals/simulation/__init__.py
tests/strands_evals/simulation/test_actor_simulator.py
tests/strands_evals/simulation/test_goal_completion.py
tests/strands_evals/simulation/test_tool_simulator.py
tests/strands_evals/telemetry/test_config.py
tests/strands_evals/telemetry/test_tracer.py
tests/strands_evals/test_cases.py
tests/strands_evals/test_eval_task.py
tests/strands_evals/test_experiment.py
tests/strands_evals/test_local_file_task_result_store.py
tests/strands_evals/tools/test_evaluation_tools.py
tests/strands_evals/types/redteam/test_risk_category.py
tests/strands_evals/types/test_evaluation_report.py
tests/strands_evals/types/test_multimodal.py
tests/strands_evals/types/test_trace.py
tests/test_integration.py
tests_integ/conftest.py
tests_integ/test_adk_eval.py
tests_integ/test_claude_openinference_eval.py
tests_integ/test_cloudwatch_provider.py
tests_integ/test_harmfulness_evaluator.py
tests_integ/test_instruction_following_evaluator.py
tests_integ/test_langchain_openinference_eval.py
tests_integ/test_langchain_traceloop_eval.py
tests_integ/test_langfuse_provider.py
tests_integ/test_multimodal_output_evaluator.py
tests_integ/test_output_evaluator.py
tests_integ/test_redteam.py
tests_integ/test_refusal_evaluator.py
tests_integ/test_stereotyping_evaluator.py

Files

.agents/skills/session-converter/SKILL.mddoc9,228 bytes
.agents/skills/session-converter/assets/segmenter.mddoc5,384 bytes
.github/CODEOWNERSasset30 bytes
.github/ISSUE_TEMPLATE/bug_report.ymlasset2,999 bytes
.github/ISSUE_TEMPLATE/config.ymlasset347 bytes
.github/ISSUE_TEMPLATE/feature_request.ymlasset1,447 bytes
.github/PULL_REQUEST_TEMPLATE.mddoc1,351 bytes
.github/dependabot.ymlasset2,941 bytes
.github/labelers/area.ymlasset5,487 bytes
.github/labelers/type.ymlasset2,015 bytes
.github/scripts/group-release-notes.pyruns3,021 bytes
.github/workflows/auto-strands-review.ymlasset1,966 bytes
.github/workflows/integration-test.ymlasset5,962 bytes
.github/workflows/issue-labeler.ymlasset1,062 bytes
.github/workflows/pr-and-push.ymlasset1,485 bytes
.github/workflows/pr-title.ymlasset921 bytes
.github/workflows/release.ymlasset16,301 bytes
.github/workflows/strands-command.ymlasset4,491 bytes
.github/workflows/test-lint.ymlasset2,638 bytes
.github/workflows/test-package-build.ymlasset4,060 bytes
.gitignoreasset183 bytes
.pre-commit-config.yamlasset786 bytes
AGENTS.mddoc34,231 bytes
CODE_OF_CONDUCT.mddoc308 bytes
CONTRIBUTING.mddoc9,201 bytes
LICENSEasset10,141 bytes
NOTICEasset66 bytes
README.mddoc35,911 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
SKILL.mdcore25,804 bytes
STYLE_GUIDE.mddoc2,377 bytes
pyproject.tomlasset5,886 bytes
src/__init__.pyruns0 bytes
src/strands_evals/__init__.pyruns1,154 bytes
src/strands_evals/__main__.pyruns142 bytes
src/strands_evals/_async.pyruns1,082 bytes
src/strands_evals/case.pyruns2,543 bytes
src/strands_evals/chaos/__init__.pyruns1,167 bytes
src/strands_evals/chaos/_context.pyruns654 bytes
src/strands_evals/chaos/case.pyruns8,161 bytes
src/strands_evals/chaos/effects.pyruns17,046 bytes
src/strands_evals/chaos/experiment.pyruns6,116 bytes
src/strands_evals/chaos/plugin.pyruns11,808 bytes
src/strands_evals/cli/__init__.pyruns232 bytes
src/strands_evals/cli/__main__.pyruns257 bytes
src/strands_evals/cli/_agent_task.pyruns4,607 bytes
src/strands_evals/cli/_common.pyruns4,240 bytes
src/strands_evals/cli/_entrypoint.pyruns16,003 bytes
src/strands_evals/cli/_io.pyruns861 bytes
src/strands_evals/cli/commands/__init__.pyruns52 bytes
src/strands_evals/cli/commands/diagnose.pyruns6,818 bytes
src/strands_evals/cli/commands/fetch.pyruns9,468 bytes
src/strands_evals/cli/commands/generate.pyruns7,780 bytes
src/strands_evals/cli/commands/report.pyruns2,606 bytes
src/strands_evals/cli/commands/run.pyruns24,344 bytes
src/strands_evals/cli/commands/validate.pyruns2,225 bytes
src/strands_evals/cli/main.pyruns1,584 bytes
src/strands_evals/detectors/__init__.pyruns989 bytes
src/strands_evals/detectors/chunking.pyruns10,377 bytes
src/strands_evals/detectors/constants.pyruns644 bytes
src/strands_evals/detectors/diagnosis.pyruns1,419 bytes
src/strands_evals/detectors/failure_detector.pyruns13,004 bytes
src/strands_evals/detectors/prompt_templates/__init__.pyruns152 bytes
src/strands_evals/detectors/prompt_templates/failure_detection/__init__.pyruns312 bytes
src/strands_evals/detectors/prompt_templates/failure_detection/failure_detection_v0.pyruns20,163 bytes
src/strands_evals/detectors/prompt_templates/root_cause/__init__.pyruns601 bytes
src/strands_evals/detectors/prompt_templates/root_cause/root_cause_merge_v0.pyruns3,842 bytes
src/strands_evals/detectors/prompt_templates/root_cause/root_cause_v0.pyruns20,590 bytes
src/strands_evals/detectors/root_cause_analyzer.pyruns10,938 bytes
src/strands_evals/detectors/utils.pyruns3,556 bytes
src/strands_evals/display/display_console.pyruns5,964 bytes
src/strands_evals/eval_task_handler.pyruns4,032 bytes
src/strands_evals/evaluation_data_store.pyruns922 bytes
src/strands_evals/evaluators/__init__.pyruns2,708 bytes
src/strands_evals/evaluators/chaos/__init__.pyruns381 bytes
src/strands_evals/evaluators/chaos/failure_communication_evaluator.pyruns3,311 bytes
src/strands_evals/evaluators/chaos/partial_completion_evaluator.pyruns2,775 bytes
src/strands_evals/evaluators/chaos/prompt_templates/__init__.pyruns0 bytes
src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/__init__.pyruns196 bytes
src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/failure_communication_v0.pyruns5,443 bytes
src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/__init__.pyruns190 bytes
src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/partial_completion_v0.pyruns6,350 bytes
src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/__init__.pyruns188 bytes
src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/recovery_strategy_v0.pyruns12,119 bytes
src/strands_evals/evaluators/chaos/recovery_strategy_evaluator.pyruns3,246 bytes
src/strands_evals/evaluators/coherence_evaluator.pyruns4,661 bytes
src/strands_evals/evaluators/conciseness_evaluator.pyruns3,029 bytes
src/strands_evals/evaluators/correctness_evaluator.pyruns6,520 bytes
src/strands_evals/evaluators/deterministic/__init__.pyruns433 bytes
src/strands_evals/evaluators/deterministic/environment_state.pyruns2,943 bytes
src/strands_evals/evaluators/deterministic/output.pyruns3,136 bytes
src/strands_evals/evaluators/deterministic/skill_invoked.pyruns2,760 bytes
src/strands_evals/evaluators/deterministic/structured_output.pyruns23,937 bytes
src/strands_evals/evaluators/deterministic/trajectory.pyruns1,841 bytes
src/strands_evals/evaluators/evaluator.pyruns16,816 bytes
src/strands_evals/evaluators/faithfulness_evaluator.pyruns3,327 bytes
src/strands_evals/evaluators/goal_success_rate_evaluator.pyruns6,434 bytes
src/strands_evals/evaluators/harmfulness_evaluator.pyruns3,060 bytes
src/strands_evals/evaluators/helpfulness_evaluator.pyruns3,598 bytes
src/strands_evals/evaluators/instruction_following_evaluator.pyruns2,350 bytes
src/strands_evals/evaluators/interactions_evaluator.pyruns10,774 bytes
src/strands_evals/evaluators/multimodal_correctness_evaluator.pyruns1,407 bytes
src/strands_evals/evaluators/multimodal_faithfulness_evaluator.pyruns1,372 bytes
src/strands_evals/evaluators/multimodal_instruction_following_evaluator.pyruns1,411 bytes
src/strands_evals/evaluators/multimodal_output_evaluator.pyruns3,724 bytes
src/strands_evals/evaluators/multimodal_overall_quality_evaluator.pyruns2,011 bytes
src/strands_evals/evaluators/output_evaluator.pyruns6,785 bytes
src/strands_evals/evaluators/prompt_templates/case_prompt_template.pyruns3,433 bytes
src/strands_evals/evaluators/prompt_templates/coherence/__init__.pyruns172 bytes
src/strands_evals/evaluators/prompt_templates/coherence/coherence_v0.pyruns1,343 bytes
src/strands_evals/evaluators/prompt_templates/conciseness/__init__.pyruns176 bytes
src/strands_evals/evaluators/prompt_templates/conciseness/conciseness_v0.pyruns583 bytes
src/strands_evals/evaluators/prompt_templates/correctness/__init__.pyruns375 bytes
src/strands_evals/evaluators/prompt_templates/correctness/correctness_v0.pyruns520 bytes
src/strands_evals/evaluators/prompt_templates/correctness/correctness_with_reference_v0.pyruns1,753 bytes
src/strands_evals/evaluators/prompt_templates/faithfulness/__init__.pyruns178 bytes
src/strands_evals/evaluators/prompt_templates/faithfulness/faithfulness_v0.pyruns1,410 bytes
src/strands_evals/evaluators/prompt_templates/goal_success_rate/__init__.pyruns401 bytes
src/strands_evals/evaluators/prompt_templates/goal_success_rate/goal_success_rate_v0.pyruns1,950 bytes
src/strands_evals/evaluators/prompt_templates/goal_success_rate/goal_success_rate_with_assertions_v0.pyruns948 bytes
src/strands_evals/evaluators/prompt_templates/harmfulness/__init__.pyruns176 bytes
src/strands_evals/evaluators/prompt_templates/harmfulness/harmfulness_v0.pyruns1,108 bytes
src/strands_evals/evaluators/prompt_templates/helpfulness/__init__.pyruns176 bytes
src/strands_evals/evaluators/prompt_templates/helpfulness/helpfulness_v0.pyruns1,898 bytes
src/strands_evals/evaluators/prompt_templates/instruction_following/__init__.pyruns196 bytes
src/strands_evals/evaluators/prompt_templates/instruction_following/instruction_following_v0.pyruns2,651 bytes
src/strands_evals/evaluators/prompt_templates/multimodal/__init__.pyruns431 bytes
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/__init__.pyruns426 bytes
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/correctness_v0.pyruns1,305 bytes
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/faithfulness_v0.pyruns1,372 bytes
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/instruction_following_v0.pyruns1,417 bytes
src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/overall_quality_v0.pyruns4,740 bytes
src/strands_evals/evaluators/prompt_templates/multimodal_case_prompt_template.pyruns5,793 bytes
src/strands_evals/evaluators/prompt_templates/multimodal_judge_system_prompt.pyruns2,652 bytes
src/strands_evals/evaluators/prompt_templates/prompt_templates.pyruns15,639 bytes
src/strands_evals/evaluators/prompt_templates/refusal/__init__.pyruns168 bytes
src/strands_evals/evaluators/prompt_templates/refusal/refusal_v0.pyruns1,312 bytes
src/strands_evals/evaluators/prompt_templates/response_relevance/__init__.pyruns190 bytes
src/strands_evals/evaluators/prompt_templates/response_relevance/response_relevance_v0.pyruns1,663 bytes
src/strands_evals/evaluators/prompt_templates/skill_instruction_following/__init__.pyruns208 bytes
src/strands_evals/evaluators/prompt_templates/skill_instruction_following/skill_instruction_following_v0.pyruns2,357 bytes
src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/__init__.pyruns202 bytes
src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/skill_selection_accuracy_v0.pyruns1,322 bytes
src/strands_evals/evaluators/prompt_templates/stereotyping/__init__.pyruns178 bytes
src/strands_evals/evaluators/prompt_templates/stereotyping/stereotyping_v0.pyruns767 bytes
src/strands_evals/evaluators/prompt_templates/tool_parameter_accuracy/__init__.pyruns200 bytes
src/strands_evals/evaluators/prompt_templates/tool_parameter_accuracy/tool_parameter_accuracy_v0.pyruns2,259 bytes
src/strands_evals/evaluators/prompt_templates/tool_selection_accuracy/__init__.pyruns200 bytes
src/strands_evals/evaluators/prompt_templates/tool_selection_accuracy/tool_selection_accuracy_v0.pyruns1,483 bytes
src/strands_evals/evaluators/prompt_templates/trajectory_prompt_template.pyruns2,085 bytes
src/strands_evals/evaluators/redteam/__init__.pyruns99 bytes
src/strands_evals/evaluators/redteam/attack_success_evaluator.pyruns6,245 bytes
src/strands_evals/evaluators/redteam/prompt_templates/__init__.pyruns0 bytes
src/strands_evals/evaluators/redteam/prompt_templates/attack_success/__init__.pyruns182 bytes
src/strands_evals/evaluators/redteam/prompt_templates/attack_success/attack_success_v0.pyruns4,243 bytes
src/strands_evals/evaluators/refusal_evaluator.pyruns2,194 bytes
src/strands_evals/evaluators/response_relevance_evaluator.pyruns3,344 bytes
src/strands_evals/evaluators/skill_instruction_following_evaluator.pyruns11,307 bytes
src/strands_evals/evaluators/skill_selection_accuracy_evaluator.pyruns9,856 bytes
src/strands_evals/evaluators/stereotyping_evaluator.pyruns2,342 bytes
src/strands_evals/evaluators/tool_parameter_accuracy_evaluator.pyruns3,464 bytes
src/strands_evals/evaluators/tool_selection_accuracy_evaluator.pyruns3,381 bytes
src/strands_evals/evaluators/trajectory_evaluator.pyruns4,323 bytes
src/strands_evals/experiment.pyruns40,105 bytes
src/strands_evals/experimental/__init__.pyruns0 bytes
src/strands_evals/extractors/__init__.pyruns463 bytes
src/strands_evals/extractors/graph_extractor.pyruns1,105 bytes
src/strands_evals/extractors/skills/__init__.pyruns1,448 bytes
src/strands_evals/extractors/skills/_normalize.pyruns10,095 bytes
src/strands_evals/extractors/skills/_patterns.pyruns5,953 bytes
src/strands_evals/extractors/skills/adapters/__init__.pyruns900 bytes
src/strands_evals/extractors/skills/adapters/_common.pyruns3,639 bytes
src/strands_evals/extractors/skills/adapters/claude.pyruns1,288 bytes
src/strands_evals/extractors/skills/adapters/codex.pyruns3,291 bytes
src/strands_evals/extractors/skills/adapters/gemini.pyruns1,964 bytes
src/strands_evals/extractors/skills/adapters/openhands.pyruns1,178 bytes
src/strands_evals/extractors/skills/adapters/registry.pyruns5,687 bytes
src/strands_evals/extractors/skills/adapters/strands.pyruns1,665 bytes
src/strands_evals/extractors/skills/extractor.pyruns27,366 bytes
src/strands_evals/extractors/skills/models.pyruns2,939 bytes
src/strands_evals/extractors/swarm_extractor.pyruns2,682 bytes
src/strands_evals/extractors/tools_use_extractor.pyruns6,894 bytes
src/strands_evals/extractors/trace_extractor.pyruns9,055 bytes
src/strands_evals/generators/__init__.pyruns89 bytes
src/strands_evals/generators/experiment_generator.pyruns22,606 bytes
src/strands_evals/generators/prompt_template/prompt_templates.pyruns3,969 bytes
src/strands_evals/generators/topic_planner.pyruns2,436 bytes
src/strands_evals/local_file_task_result_store.pyruns1,260 bytes
src/strands_evals/mappers/__init__.pyruns1,252 bytes
src/strands_evals/mappers/adk_otel_session_mapper.pyruns25,496 bytes
src/strands_evals/mappers/cloudwatch_parser.pyruns9,491 bytes
src/strands_evals/mappers/cloudwatch_session_mapper.pyruns15,653 bytes
src/strands_evals/mappers/constants.pyruns1,711 bytes
src/strands_evals/mappers/generic_gen_ai_session_mapper.pyruns30,899 bytes
src/strands_evals/mappers/langchain_otel_session_mapper.pyruns34,075 bytes
src/strands_evals/mappers/openai_agents_gen_ai_session_mapper.pyruns6,849 bytes
src/strands_evals/mappers/openinference_session_mapper.pyruns58,685 bytes
src/strands_evals/mappers/opensearch_session_mapper.pyruns6,362 bytes
src/strands_evals/mappers/session_mapper.pyruns4,634 bytes
src/strands_evals/mappers/strands_in_memory_session_mapper.pyruns23,658 bytes
src/strands_evals/mappers/utils.pyruns12,374 bytes
src/strands_evals/providers/README.mddoc3,745 bytes
src/strands_evals/providers/__init__.pyruns923 bytes
src/strands_evals/providers/cloudwatch_provider.pyruns10,977 bytes
src/strands_evals/providers/exceptions.pyruns330 bytes
src/strands_evals/providers/langfuse_provider.pyruns34,421 bytes
src/strands_evals/providers/opensearch_provider.pyruns3,862 bytes
src/strands_evals/providers/trace_provider.pyruns1,598 bytes
src/strands_evals/redteam/README.mddoc15,508 bytes
src/strands_evals/redteam/__init__.pyruns1,318 bytes
src/strands_evals/redteam/case.pyruns2,649 bytes
src/strands_evals/redteam/experiment.pyruns15,897 bytes
src/strands_evals/redteam/generators/__init__.pyruns116 bytes
src/strands_evals/redteam/generators/adversarial.pyruns8,125 bytes
src/strands_evals/redteam/generators/prompt_templates/__init__.pyruns176 bytes
src/strands_evals/redteam/generators/prompt_templates/adversarial_v0.pyruns1,739 bytes
src/strands_evals/redteam/report.pyruns18,481 bytes
src/strands_evals/redteam/strategies/__init__.pyruns1,428 bytes
src/strands_evals/redteam/strategies/_common.pyruns2,674 bytes
src/strands_evals/redteam/strategies/bad_likert_judge/__init__.pyruns6,321 bytes
src/strands_evals/redteam/strategies/bad_likert_judge/bad_likert_judge_v0.pyruns5,248 bytes
src/strands_evals/redteam/strategies/base.pyruns10,476 bytes
src/strands_evals/redteam/strategies/crescendo/__init__.pyruns10,752 bytes
src/strands_evals/redteam/strategies/crescendo/crescendo_v0.pyruns2,078 bytes
src/strands_evals/redteam/strategies/goat/__init__.pyruns9,178 bytes
src/strands_evals/redteam/strategies/goat/goat_v0.pyruns2,713 bytes
src/strands_evals/redteam/strategies/pair/__init__.pyruns9,571 bytes
src/strands_evals/redteam/strategies/pair/pair_v0.pyruns3,609 bytes
src/strands_evals/redteam/strategies/prompt_strategy/__init__.pyruns4,256 bytes
src/strands_evals/redteam/strategies/prompt_strategy/gradual_escalation/__init__.pyruns190 bytes
src/strands_evals/redteam/strategies/prompt_strategy/gradual_escalation/gradual_escalation_v0.pyruns1,913 bytes
src/strands_evals/redteam/strategies/sequentialbreak/__init__.pyruns8,080 bytes
src/strands_evals/redteam/strategies/sequentialbreak/sequentialbreak_v0.pyruns5,064 bytes
src/strands_evals/redteam/strategies/target_session.pyruns17,647 bytes
src/strands_evals/redteam/task.pyruns7,100 bytes
src/strands_evals/redteam/utils.pyruns1,494 bytes
src/strands_evals/simulation/__init__.pyruns367 bytes
src/strands_evals/simulation/actor_simulator.pyruns14,180 bytes
src/strands_evals/simulation/profiles/__init__.pyruns145 bytes
src/strands_evals/simulation/profiles/actor_profile.pyruns948 bytes
src/strands_evals/simulation/prompt_templates/__init__.pyruns476 bytes
src/strands_evals/simulation/prompt_templates/actor_profile_extraction.pyruns907 bytes
src/strands_evals/simulation/prompt_templates/actor_system_prompt.pyruns3,000 bytes
src/strands_evals/simulation/prompt_templates/goal_completion.pyruns670 bytes
src/strands_evals/simulation/prompt_templates/tool_response_generation.pyruns1,312 bytes
src/strands_evals/simulation/tool_simulator.pyruns16,088 bytes
src/strands_evals/simulation/tools/__init__.pyruns145 bytes
src/strands_evals/simulation/tools/goal_completion.pyruns3,252 bytes
src/strands_evals/telemetry/__init__.pyruns382 bytes
src/strands_evals/telemetry/_cloudwatch_logger.pyruns7,757 bytes
src/strands_evals/telemetry/config.pyruns10,201 bytes
src/strands_evals/telemetry/tracer.pyruns997 bytes
src/strands_evals/tools/evaluation_tools.pyruns1,864 bytes
src/strands_evals/types/__init__.pyruns999 bytes
src/strands_evals/types/detector.pyruns6,463 bytes
src/strands_evals/types/evaluation.pyruns5,973 bytes
src/strands_evals/types/evaluation_report.pyruns16,660 bytes
src/strands_evals/types/multimodal.pyruns9,048 bytes
src/strands_evals/types/redteam/__init__.pyruns430 bytes
src/strands_evals/types/redteam/config.pyruns989 bytes
src/strands_evals/types/redteam/risk_category.pyruns3,542 bytes
src/strands_evals/types/redteam/run_results.pyruns1,919 bytes
src/strands_evals/types/simulation/__init__.pyruns144 bytes
src/strands_evals/types/simulation/actor.pyruns3,131 bytes
src/strands_evals/types/simulation/tool.pyruns1,357 bytes
src/strands_evals/types/trace.pyruns8,059 bytes
src/strands_evals/utils.pyruns1,458 bytes
tests/__init__.pyruns0 bytes
tests/strands_evals/__init__.pyruns0 bytes
tests/strands_evals/chaos/__init__.pyruns1 bytes
tests/strands_evals/chaos/test_case.pyruns14,671 bytes
tests/strands_evals/chaos/test_context.pyruns1,470 bytes
tests/strands_evals/chaos/test_effects.pyruns8,492 bytes
tests/strands_evals/chaos/test_experiment.pyruns7,206 bytes
tests/strands_evals/chaos/test_failure_communication_evaluator.pyruns4,922 bytes
tests/strands_evals/chaos/test_partial_completion_evaluator.pyruns4,978 bytes
tests/strands_evals/chaos/test_plugin.pyruns28,787 bytes
tests/strands_evals/chaos/test_recovery_strategy_evaluator.pyruns5,208 bytes
tests/strands_evals/cli/__init__.pyruns0 bytes
tests/strands_evals/cli/conftest.pyruns3,232 bytes
tests/strands_evals/cli/fixtures/__init__.pyruns0 bytes
tests/strands_evals/cli/fixtures/agents.pyruns2,165 bytes
tests/strands_evals/cli/fixtures/evaluators.pyruns763 bytes
tests/strands_evals/cli/fixtures/tasks.pyruns585 bytes
tests/strands_evals/cli/test_agent_task.pyruns5,519 bytes
tests/strands_evals/cli/test_diagnose.pyruns5,422 bytes
tests/strands_evals/cli/test_entrypoint.pyruns8,269 bytes
tests/strands_evals/cli/test_fetch.pyruns8,497 bytes
tests/strands_evals/cli/test_generate.pyruns8,020 bytes
tests/strands_evals/cli/test_report.pyruns3,402 bytes
tests/strands_evals/cli/test_run.pyruns29,113 bytes
tests/strands_evals/cli/test_validate.pyruns2,859 bytes
tests/strands_evals/cloudwatch_helpers.pyruns1,120 bytes
tests/strands_evals/detectors/__init__.pyruns0 bytes
tests/strands_evals/detectors/test_chunking.pyruns13,170 bytes
tests/strands_evals/detectors/test_diagnosis.pyruns4,777 bytes
tests/strands_evals/detectors/test_failure_detector.pyruns16,291 bytes
tests/strands_evals/detectors/test_root_cause_analyzer.pyruns13,955 bytes
tests/strands_evals/detectors/test_types.pyruns7,299 bytes
tests/strands_evals/evaluators/deterministic/__init__.pyruns0 bytes
tests/strands_evals/evaluators/deterministic/test_environment_state.pyruns7,463 bytes
tests/strands_evals/evaluators/deterministic/test_output.pyruns8,936 bytes
tests/strands_evals/evaluators/deterministic/test_skill_invoked.pyruns5,706 bytes
tests/strands_evals/evaluators/deterministic/test_structured_output.pyruns68,246 bytes
tests/strands_evals/evaluators/deterministic/test_trajectory.pyruns4,279 bytes
tests/strands_evals/evaluators/redteam/test_attack_success_evaluator.pyruns9,148 bytes
tests/strands_evals/evaluators/test_coherence_evaluator.pyruns10,152 bytes
tests/strands_evals/evaluators/test_conciseness_evaluator.pyruns6,063 bytes
tests/strands_evals/evaluators/test_correctness_evaluator.pyruns15,577 bytes
tests/strands_evals/evaluators/test_evaluator.pyruns18,323 bytes
tests/strands_evals/evaluators/test_faithfulness_evaluator.pyruns6,063 bytes
tests/strands_evals/evaluators/test_goal_success_rate_evaluator.pyruns7,338 bytes
tests/strands_evals/evaluators/test_harmfulness_evaluator.pyruns5,893 bytes
tests/strands_evals/evaluators/test_helpfulness_evaluator.pyruns6,266 bytes
tests/strands_evals/evaluators/test_instruction_following_evaluator.pyruns6,056 bytes
tests/strands_evals/evaluators/test_interactions_evaluator.pyruns16,233 bytes
tests/strands_evals/evaluators/test_multimodal_case_prompt_template.pyruns11,604 bytes
tests/strands_evals/evaluators/test_multimodal_output_evaluator.pyruns12,099 bytes
tests/strands_evals/evaluators/test_multimodal_specialized_evaluators.pyruns10,120 bytes
tests/strands_evals/evaluators/test_output_evaluator.pyruns15,556 bytes
tests/strands_evals/evaluators/test_refusal_evaluator.pyruns5,541 bytes
tests/strands_evals/evaluators/test_response_relevance_evaluator.pyruns6,511 bytes
tests/strands_evals/evaluators/test_skill_instruction_following_evaluator.pyruns15,326 bytes
tests/strands_evals/evaluators/test_skill_selection_accuracy_evaluator.pyruns14,462 bytes
tests/strands_evals/evaluators/test_stereotyping_evaluator.pyruns5,840 bytes
tests/strands_evals/evaluators/test_tool_parameter_accuracy_evaluator.pyruns4,984 bytes
tests/strands_evals/evaluators/test_tool_selection_accuracy_evaluator.pyruns4,726 bytes
tests/strands_evals/evaluators/test_trajectory_evaluator.pyruns11,415 bytes
tests/strands_evals/evaluators/test_trajectory_prompt_template.pyruns812 bytes
tests/strands_evals/experimental/__init__.pyruns0 bytes
tests/strands_evals/extractors/__init__.pyruns0 bytes
tests/strands_evals/extractors/fixtures/capture_skill_fixtures.pyruns11,259 bytes
tests/strands_evals/extractors/fixtures/claude_code_skill_tool.jsonasset1,265 bytes
tests/strands_evals/extractors/fixtures/codex_exec_json.jsonasset1,453 bytes
tests/strands_evals/extractors/fixtures/codex_session_rollout.jsonasset1,593 bytes
tests/strands_evals/extractors/fixtures/google_adk_load_skill.jsonasset2,518 bytes
tests/strands_evals/extractors/fixtures/strands_agent_skills.jsonasset12,594 bytes
tests/strands_evals/extractors/skill_fixtures.pyruns6,957 bytes
tests/strands_evals/extractors/test_graph_extractor.pyruns4,037 bytes
tests/strands_evals/extractors/test_skills.pyruns53,816 bytes
tests/strands_evals/extractors/test_swarm_extractor.pyruns5,535 bytes
tests/strands_evals/extractors/test_tools_use_extractor.pyruns14,049 bytes
tests/strands_evals/extractors/test_trace_extractor.pyruns20,563 bytes
tests/strands_evals/generators/test_experiment_generator.pyruns16,198 bytes
tests/strands_evals/generators/test_topic_planner.pyruns921 bytes
tests/strands_evals/mappers/__init__.pyruns0 bytes
tests/strands_evals/mappers/fixtures/adk_live_spans.jsonasset9,638 bytes
tests/strands_evals/mappers/fixtures/autogen_live_spans.jsonasset1,942 bytes
tests/strands_evals/mappers/fixtures/claude_adot_spans.jsonasset16,008 bytes
tests/strands_evals/mappers/fixtures/claude_live_spans.jsonasset12,289 bytes
tests/strands_evals/mappers/fixtures/openai_agents_genai_adot_spans.jsonasset24,240 bytes
tests/strands_evals/mappers/fixtures/openai_agents_genai_live_spans.jsonasset32,922 bytes
tests/strands_evals/mappers/fixtures/openai_agents_openinference_adot_spans.jsonasset41,533 bytes
tests/strands_evals/mappers/fixtures/openai_agents_openinference_live_spans.jsonasset53,797 bytes
tests/strands_evals/mappers/fixtures/openinference_adot_spans.jsonasset7,810 bytes
tests/strands_evals/mappers/fixtures/openinference_live_spans.jsonasset4,912 bytes
tests/strands_evals/mappers/fixtures/otel_genai_convention_spans.jsonasset6,315 bytes
tests/strands_evals/mappers/fixtures/pydantic_ai_live_spans.jsonasset9,639 bytes
tests/strands_evals/mappers/fixtures/smolagents_live_spans.jsonasset73,839 bytes
tests/strands_evals/mappers/test_adk_otel_session_mapper.pyruns45,389 bytes
tests/strands_evals/mappers/test_cloudwatch_parser.pyruns10,182 bytes
tests/strands_evals/mappers/test_cloudwatch_session_mapper.pyruns12,754 bytes
tests/strands_evals/mappers/test_generic_gen_ai_session_mapper.pyruns67,367 bytes
tests/strands_evals/mappers/test_langchain_mapper_integration.pyruns20,097 bytes
tests/strands_evals/mappers/test_langchain_otel_session_mapper.pyruns11,738 bytes
tests/strands_evals/mappers/test_openai_agents_gen_ai_session_mapper.pyruns20,208 bytes
tests/strands_evals/mappers/test_openinference_session_mapper.pyruns119,978 bytes
tests/strands_evals/mappers/test_opensearch_session_mapper.pyruns16,817 bytes
tests/strands_evals/mappers/test_session_mapper.pyruns4,451 bytes
tests/strands_evals/mappers/test_strands_in_memory_mapper.pyruns27,819 bytes
tests/strands_evals/mappers/test_utils.pyruns16,227 bytes
tests/strands_evals/opensearch_helpers.pyruns2,697 bytes
tests/strands_evals/providers/__init__.pyruns0 bytes
tests/strands_evals/providers/test_cloudwatch_provider.pyruns23,730 bytes
tests/strands_evals/providers/test_langfuse_provider.pyruns39,940 bytes
tests/strands_evals/providers/test_opensearch_provider.pyruns5,044 bytes
tests/strands_evals/providers/test_trace_provider.pyruns3,275 bytes
tests/strands_evals/redteam/__init__.pyruns0 bytes
tests/strands_evals/redteam/test_bad_likert_judge.pyruns18,380 bytes
tests/strands_evals/redteam/test_case.pyruns2,050 bytes
tests/strands_evals/redteam/test_crescendo.pyruns28,075 bytes
tests/strands_evals/redteam/test_experiment.pyruns27,883 bytes
tests/strands_evals/redteam/test_generator.pyruns8,604 bytes
tests/strands_evals/redteam/test_goat.pyruns18,752 bytes
tests/strands_evals/redteam/test_multi_agent_session.pyruns27,426 bytes
tests/strands_evals/redteam/test_pair.pyruns25,483 bytes
tests/strands_evals/redteam/test_redteam_e2e.pyruns5,674 bytes
tests/strands_evals/redteam/test_report.pyruns32,201 bytes
tests/strands_evals/redteam/test_sequentialbreak.pyruns25,021 bytes
tests/strands_evals/redteam/test_strategies.pyruns16,122 bytes
tests/strands_evals/redteam/test_target_session.pyruns15,157 bytes
tests/strands_evals/redteam/test_task.pyruns12,986 bytes
tests/strands_evals/simulation/__init__.pyruns41 bytes
tests/strands_evals/simulation/test_actor_simulator.pyruns18,796 bytes
tests/strands_evals/simulation/test_goal_completion.pyruns6,662 bytes
tests/strands_evals/simulation/test_tool_simulator.pyruns17,040 bytes
tests/strands_evals/telemetry/test_config.pyruns14,587 bytes
tests/strands_evals/telemetry/test_tracer.pyruns3,934 bytes
tests/strands_evals/test_cases.pyruns2,456 bytes
tests/strands_evals/test_eval_task.pyruns8,391 bytes
tests/strands_evals/test_experiment.pyruns95,713 bytes
tests/strands_evals/test_local_file_task_result_store.pyruns1,726 bytes
tests/strands_evals/tools/test_evaluation_tools.pyruns3,879 bytes
tests/strands_evals/types/redteam/test_risk_category.pyruns2,076 bytes
tests/strands_evals/types/test_evaluation_report.pyruns18,121 bytes
tests/strands_evals/types/test_multimodal.pyruns10,969 bytes
tests/strands_evals/types/test_trace.pyruns16,191 bytes
tests/test_integration.pyruns13,073 bytes
tests_integ/conftest.pyruns2,312 bytes
tests_integ/test_adk_eval.pyruns14,996 bytes
tests_integ/test_claude_openinference_eval.pyruns10,957 bytes
tests_integ/test_cloudwatch_provider.pyruns7,625 bytes
tests_integ/test_harmfulness_evaluator.pyruns2,164 bytes
tests_integ/test_instruction_following_evaluator.pyruns2,264 bytes
tests_integ/test_langchain_openinference_eval.pyruns7,673 bytes
tests_integ/test_langchain_traceloop_eval.pyruns8,411 bytes
tests_integ/test_langfuse_provider.pyruns7,107 bytes
tests_integ/test_multimodal_output_evaluator.pyruns13,425 bytes
tests_integ/test_output_evaluator.pyruns7,629 bytes
tests_integ/test_redteam.pyruns3,995 bytes
tests_integ/test_refusal_evaluator.pyruns2,446 bytes
tests_integ/test_stereotyping_evaluator.pyruns2,385 bytes

Read SKILL.md


# strands-evals — Authoring Evaluations

Patterns for using the `strands-agents-evals` package to evaluate AI agents and LLM applications. For repo conventions, contribution rules, prompt versioning, and review checklists, see `AGENTS.md` at the repo root. This skill stays focused on authoring.

## Routing Map

| Goal | Section |
| --- | --- |
| Score a one-shot response | OutputEvaluator |
| Score tool/action sequences | TrajectoryEvaluator |
| Score a multi-turn trace | Trace-based evaluators |
| Multi-turn conversation testing | ActorSimulator |
| Replace real tools during eval | ToolSimulator |
| Diagnose a failing session | Detectors |

View the complete source on GitHub

Skills included

strands-evals

Repository facts

License: Apache-2.0
Created: Jul 31, 2025
Last pushed: Nov 6, 2026
Forks: 80
Open issues: 91
Project activity: Active

Before you install

Network or broad shell

Automated signals, not a guarantee

Checked Nov 7, 2026

Not fully analyzed: some files were too long, unreadable or missing. This never lowers the level.

Show 51 notes

SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed
SKILL.md:1 Analysis incomplete: Referenced executable target unavailable; not analyzed

No broad tool permissions
2 dangerous shell commandsin 2 files: .github/workflows/test-package-build.yml:68 tests/strands_evals/extractors/test_skills.py:574
212 network referencesin 26 files: README.md:1 src/strands_evals/cli/commands/fetch.py:138 tests/strands_evals/providers/test_langfuse_provider.py:88 and 23 more
18 obfuscation signalsin 5 files: tests/strands_evals/evaluators/test_multimodal_case_prompt_template.py:20 tests/strands_evals/evaluators/test_multimodal_output_evaluator.py:20 tests/strands_evals/evaluators/test_multimodal_specialized_evaluators.py:27 and 2 more
3 instruction override signalsin 1 file: .github/workflows/release.yml:23
371 executable scriptsin 371 files: .github/scripts/group-release-notes.py:1 src/__init__.py:1 src/strands_evals/__init__.py:1 and 368 more
52 references to secretsin 18 files: tests/strands_evals/providers/test_langfuse_provider.py:92 tests/strands_evals/providers/test_cloudwatch_provider.py:102 tests_integ/conftest.py:24 and 15 more
No requests to vote in Hot

Show all 658 signals by file

`.agents/skills/session-converter/SKILL.md`

Network accessline 24

**Claude Code** ([docs](https://code.claude.com/docs/en/claude-directory#application-data))

Network accessline 29

**Strands CLI** ([docs](https://github.com/strands-agents/harness-sdk/blob/main/strands-cli/README.md))

Network accessline 34

**Kiro CLI** ([docs](https://kiro.dev/docs/cli/chat/session-management/))

`.github/ISSUE_TEMPLATE/config.yml`

Network accessline 2

contact_links:   - name: Strands Evals SDK Support     url: https://github.com/strands-agents/evals/discussions

Network accessline 3

- name: Strands Evals SDK Support     url: https://github.com/strands-agents/evals/discussions     about: Please ask and answer questions here

Network accessline 4

url: https://github.com/strands-agents/evals/discussions

Network accessline 5

about: Please ask and answer questions here   - name: Strands Evals SDK Documentation     url: https://strandsagents.com/docs/user-guide/evals-sdk/quickstart/

Network accessline 6

- name: Strands Evals SDK Documentation     url: https://strandsagents.com/docs/user-guide/evals-sdk/quickstart/     about: Visit our documentation for help

Network accessline 7

url: https://strandsagents.com/docs/user-guide/evals-sdk/quickstart/

`.github/scripts/group-release-notes.py`

Executable scriptline 1
```
#!/usr/bin/env python3
```
Access to secretsline 62
```
[Credential-related source omitted]
```
Access to secretsline 63
```
[Credential-related source omitted]
```

`.github/workflows/release.yml`

Instruction override signalline 23

type: string       dry_run:         description: 'Skip approval + tag + publish. Build/inspect/notes still run so you can review the artifact on the run page.'

Instruction override signalline 24

dry_run:         description: 'Skip approval + tag + publish. Build/inspect/notes still run so you can review the artifact on the run page.'         required: true

Instruction override signalline 25

description: 'Skip approval + tag + publish. Build/inspect/notes still run so you can review the artifact on the run page.'

Access to secretsline 215
```
[Credential-related source omitted]
```

Network accessline 411

environment:       name: pypi       url: https://pypi.org/p/strands-agents-evals

Network accessline 412

name: pypi       url: https://pypi.org/p/strands-agents-evals     permissions:

Network accessline 413

url: https://pypi.org/p/strands-agents-evals

`.github/workflows/test-package-build.yml`

Dangerous shell operationline 68
```
trap 'rm -rf "$WORK"' EXIT
```
Access to secretsline 91
```
[Credential-related source omitted]
```

`.gitignore`

Access to secretsline 6
```
[Credential-related source omitted]
```

`AGENTS.md`

Network accessline 9

- **Point people to the community.** Real questions and design discussion belong with people — the [Discord](https://discord.gg/strands) and [GitHub Discussions](https://github.com/strands-agents/evals/discussions).

Network accessline 34

- SDK source (external, not in this repo): [`strands-agents/sdk-python`](https://github.com/strands-agents/sdk-python) under `src/strands/`

Network accessline 513

- [CONTRIBUTING.md](./CONTRIBUTING.md) — Human contributor guidelines - [STYLE_GUIDE.md](./STYLE_GUIDE.md) — Logging and style conventions - [Strands Agents Documentation](https://strandsagents.com/)

Network accessline 514

- [STYLE_GUIDE.md](./STYLE_GUIDE.md) — Logging and style conventions - [Strands Agents Documentation](https://strandsagents.com/) - [Strands Agents Python SDK](https://github.com/strands-agents/sdk-python)

Network accessline 515

- [Strands Agents Documentation](https://strandsagents.com/)

Network accessline 516

- [Strands Agents Python SDK](https://github.com/strands-agents/sdk-python)

Network accessline 517

- [Strands Agents Samples](https://github.com/strands-agents/samples)

`CODE_OF_CONDUCT.md`

Network accessline 1

## Code of Conduct This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).

Network accessline 2

This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).

Network accessline 3

For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact

`CONTRIBUTING.md`

Network accessline 53

This project uses [hatchling](https://hatch.pypa.io/latest/build/#hatchling) as the build backend and [hatch](https://hatch.pypa.io/latest/) for development workflow management.

Network accessline 106

We use [pre-commit](https://pre-commit.com/) to automatically run quality checks before each commit. The hook will run `hatch run format`, `hatch run lint`, `hatch run test`, and `hatch run cz check` when you make a commit, ensuring code co

Network accessline 165

4. Run linting checks with `hatch fmt --linter`. 5. Ensure local tests pass with `hatch test` and `hatch run test-integ`. 6. Commit to your branch using clear commit messages following the [Conventional Commits](https://www.conventionalcomm

Network accessline 166

5. Ensure local tests pass with `hatch test` and `hatch run test-integ`. 6. Commit to your branch using clear commit messages following the [Conventional Commits](https://www.conventionalcommits.org) specification. 7. Send us a pull request

Network accessline 167

6. Commit to your branch using clear commit messages following the [Conventional Commits](https://www.conventionalcommits.org) specification.

Network accessline 172

## Code of Conduct This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).

Network accessline 173

This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).

Network accessline 174

For more information, see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact

Network accessline 178

## Security issue notifications If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do

Network accessline 179

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github

`LICENSE`

Network accessline 2

Apache License                            Version 2.0, January 2004                         http://www.apache.org/licenses/

Network accessline 4
```
http://www.apache.org/licenses/
```

`README.md`

Network accessline 1

<div align="center">   <div>     <a href="https://strandsagents.com">

Network accessline 2

<div>     <a href="https://strandsagents.com">       <picture>

Network accessline 3
```
<a href="https://strandsagents.com">
```

Network accessline 4

<picture>         <source media="(prefers-color-scheme: dark)" srcset="https://strandsagents.com/latest/assets/wordmark-github-dark.svg">         <img src="https://strandsagents.com/latest/assets/wordmark-github-light.svg" alt="Strands" wid

Network accessline 5

<source media="(prefers-color-scheme: dark)" srcset="https://strandsagents.com/latest/assets/wordmark-github-dark.svg">

Network accessline 6

<img src="https://strandsagents.com/latest/assets/wordmark-github-light.svg" alt="Strands" width="320">

Network accessline 18

<div align="center">     <a href="https://github.com/strands-agents/evals/graphs/commit-activity"><img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/strands-agents/evals"/></a>

Network accessline 19

<a href="https://github.com/strands-agents/evals/graphs/commit-activity"><img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/strands-agents/evals"/></a>     <a href="https://github.com/strands-agents/evals

Network accessline 20

<a href="https://github.com/strands-agents/evals/issues"><img alt="GitHub open issues" src="https://img.shields.io/github/issues/strands-agents/evals"/></a>     <a href="https://github.com/strands-agents/evals/pulls"><img alt="GitHub open p

Network accessline 21

<a href="https://github.com/strands-agents/evals/pulls"><img alt="GitHub open pull requests" src="https://img.shields.io/github/issues-pr/strands-agents/evals"/></a>

Network accessline 22

<a href="https://github.com/strands-agents/evals/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/strands-agents/evals"/></a>

Network accessline 23

<a href="https://pypi.org/project/strands-agents-evals/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/strands-agents-evals"/></a>

Network accessline 24

<a href="https://python.org"><img alt="Python versions" src="https://img.shields.io/pypi/pyversions/strands-agents-evals"/></a>

Network accessline 25

<a href="https://discord.gg/strands"><img alt="Strands Discord" src="https://img.shields.io/badge/Discord-Strands-5865F2?logo=discord&logoColor=white"/></a>

Network accessline 28

<p>     <a href="https://strandsagents.com/">Documentation</a>

Network accessline 29

<a href="https://strandsagents.com/">Documentation</a>

Network accessline 30

◆ <a href="https://github.com/strands-agents/samples">Samples</a>

Network accessline 31

◆ <a href="https://github.com/strands-agents/harness-sdk/tree/main/strands-cli">CLI</a>

Network accessline 32

◆ <a href="https://github.com/strands-agents/harness-sdk/tree/main/strands-mcp">MCP Server</a>

Network accessline 33

<br>     Harness (<a href="https://github.com/strands-agents/harness-sdk/tree/main/harness-py">Python</a>, <a href="https://github.com/strands-agents/harness-sdk/tree/main/harness-ts">TypeScript</a>)     ◆ SDK (<a href="https://github.com/s

Network accessline 34

Harness (<a href="https://github.com/strands-agents/harness-sdk/tree/main/harness-py">Python</a>, <a href="https://github.com/strands-agents/harness-sdk/tree/main/harness-ts">TypeScript</a>)

Network accessline 35

◆ SDK (<a href="https://github.com/strands-agents/harness-sdk/tree/main/strands-py">Python</a>, <a href="https://github.com/strands-agents/harness-sdk/tree/main/strands-ts">TypeScript</a>)

Network accessline 36

◆ <a href="https://github.com/strands-agents/box">Box</a>

Network accessline 37

◆ <a href="https://github.com/strands-agents/shell">Shell</a>

Network accessline 38

◆ <a href="https://github.com/strands-agents/evals">Evals</a>

Network accessline 39

◆ <a href="https://github.com/strands-labs">Labs</a>

Network accessline 823

- [User Guide](https://strandsagents.com/latest/documentation/docs/user-guide/evals-sdk/quickstart/)

Network accessline 824

- [Evaluator Reference](https://strandsagents.com/latest/documentation/docs/user-guide/evals-sdk/evaluators/)

Network accessline 825

- [Simulators Guide](https://strandsagents.com/latest/documentation/docs/user-guide/evals-sdk/simulators/)

Network accessline 835

## Stay in touch with the team Come meet the Strands team and other users on [**Discord**](https://discord.com/invite/strands)

Network accessline 836

Come meet the Strands team and other users on [**Discord**](https://discord.com/invite/strands)

`SKILL.md`

Network accessline 537

- Logging style: `STYLE_GUIDE.md` - Human contributor guide: `CONTRIBUTING.md` - User docs: https://strandsagents.com/latest/documentation/docs/user-guide/evals-sdk/quickstart/

Network accessline 539

- User docs: https://strandsagents.com/latest/documentation/docs/user-guide/evals-sdk/quickstart/

`src/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/main.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/_async.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/case.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/chaos/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/chaos/_context.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/chaos/case.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/chaos/effects.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/chaos/experiment.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/chaos/plugin.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/main.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/_agent_task.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/_common.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/_entrypoint.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/_io.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/commands/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/commands/diagnose.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/commands/fetch.py`

Executable scriptline 1
```
Executable file
```

Network accessline 138

required=True,         metavar="ID",         help="session id to fetch (matches attributes.session.id)",

Network accessline 139

metavar="ID",         help="session id to fetch (matches attributes.session.id)",     )

Network accessline 140

help="session id to fetch (matches attributes.session.id)",

Network accessline 190

required=True,         metavar="ID",         help="session id to fetch (Langfuse trace.list session_id filter)",

Network accessline 191

metavar="ID",         help="session id to fetch (Langfuse trace.list session_id filter)",     )

Network accessline 192

help="session id to fetch (Langfuse trace.list session_id filter)",

Network accessline 195

"--host",         default=None,         help="Langfuse API host (default: LANGFUSE_HOST env var, then https://us.cloud.langfuse.com)",

Network accessline 196

default=None,         help="Langfuse API host (default: LANGFUSE_HOST env var, then https://us.cloud.langfuse.com)",     )

Network accessline 197

help="Langfuse API host (default: LANGFUSE_HOST env var, then https://us.cloud.langfuse.com)",

Network accessline 231

required=True,         metavar="ID",         help="session id to fetch (conversation id)",

Network accessline 232

metavar="ID",         help="session id to fetch (conversation id)",     )

Network accessline 233

help="session id to fetch (conversation id)",

Network accessline 235

os_.add_argument(         "--host",         default="https://localhost:9200",

Network accessline 236

"--host",         default="https://localhost:9200",         help="OpenSearch endpoint URL (default: https://localhost:9200)",

Network accessline 237
```
default="https://localhost:9200",
```

Network accessline 238

help="OpenSearch endpoint URL (default: https://localhost:9200)",

`src/strands_evals/cli/commands/generate.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/commands/report.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/commands/run.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/commands/validate.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/cli/main.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/chunking.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/constants.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/diagnosis.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/failure_detector.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/prompt_templates/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/prompt_templates/failure_detection/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/prompt_templates/failure_detection/failure_detection_v0.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 83
```
[Credential-related source omitted]
```
Access to secretsline 85
```
[Credential-related source omitted]
```

`src/strands_evals/detectors/prompt_templates/root_cause/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/prompt_templates/root_cause/root_cause_merge_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/prompt_templates/root_cause/root_cause_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/root_cause_analyzer.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/detectors/utils.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/display/display_console.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/eval_task_handler.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluation_data_store.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/failure_communication_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/partial_completion_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/prompt_templates/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/failure_communication_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/partial_completion_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/recovery_strategy_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/chaos/recovery_strategy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/coherence_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/conciseness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/correctness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/deterministic/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/deterministic/environment_state.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/deterministic/output.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/deterministic/skill_invoked.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/deterministic/structured_output.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/deterministic/trajectory.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/faithfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/goal_success_rate_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/harmfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/helpfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/instruction_following_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/interactions_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/multimodal_correctness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/multimodal_faithfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/multimodal_instruction_following_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/multimodal_output_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/multimodal_overall_quality_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/output_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/case_prompt_template.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/coherence/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/coherence/coherence_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/conciseness/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/conciseness/conciseness_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/correctness/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/correctness/correctness_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/correctness/correctness_with_reference_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/faithfulness/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/faithfulness/faithfulness_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/goal_success_rate/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/goal_success_rate/goal_success_rate_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/goal_success_rate/goal_success_rate_with_assertions_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/harmfulness/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/harmfulness/harmfulness_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/helpfulness/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/helpfulness/helpfulness_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/instruction_following/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/instruction_following/instruction_following_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/correctness_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/faithfulness_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/instruction_following_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal/image_to_text/overall_quality_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal_case_prompt_template.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/multimodal_judge_system_prompt.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/prompt_templates.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/refusal/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/refusal/refusal_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/response_relevance/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/response_relevance/response_relevance_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/skill_instruction_following/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/skill_instruction_following/skill_instruction_following_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/skill_selection_accuracy/skill_selection_accuracy_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/stereotyping/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/stereotyping/stereotyping_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/tool_parameter_accuracy/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/tool_parameter_accuracy/tool_parameter_accuracy_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/tool_selection_accuracy/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/tool_selection_accuracy/tool_selection_accuracy_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/prompt_templates/trajectory_prompt_template.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/redteam/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/redteam/attack_success_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/redteam/prompt_templates/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/redteam/prompt_templates/attack_success/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/redteam/prompt_templates/attack_success/attack_success_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/refusal_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/response_relevance_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/skill_instruction_following_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/skill_selection_accuracy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/stereotyping_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/tool_parameter_accuracy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/tool_selection_accuracy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/evaluators/trajectory_evaluator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/experiment.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 171
```
[Credential-related source omitted]
```
Access to secretsline 425
```
[Credential-related source omitted]
```
Access to secretsline 444
```
[Credential-related source omitted]
```

`src/strands_evals/experimental/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/graph_extractor.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/_normalize.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/_patterns.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/_common.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/claude.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/codex.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/gemini.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/openhands.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/registry.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/adapters/strands.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/extractor.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/skills/models.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/swarm_extractor.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/tools_use_extractor.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/extractors/trace_extractor.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/generators/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/generators/experiment_generator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/generators/prompt_template/prompt_templates.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/generators/topic_planner.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/local_file_task_result_store.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/adk_otel_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/cloudwatch_parser.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/cloudwatch_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/constants.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/generic_gen_ai_session_mapper.py`

Executable scriptline 1
```
Executable file
```

Network accessline 245

# Current OTel GenAI convention: single event with input/output messages                     # https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md

Network accessline 246

# https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md

Network accessline 273

# Prefer error.type (current OTel GenAI convention for failed operations),         # fall back to gen_ai.tool.status / tool.status for legacy compatibility.         # https://github.com/open-telemetry/semantic-conventions-genai/blob/main/mo

Network accessline 275

# https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/spans.yaml#L522-L562

Network accessline 420

# Try gen_ai.output.messages (current OTel GenAI convention for invoke_agent)             # https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md

Network accessline 421

# https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md

`src/strands_evals/mappers/langchain_otel_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/openai_agents_gen_ai_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/openinference_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/opensearch_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/strands_in_memory_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/mappers/utils.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/providers/README.md`

Network accessline 40

public_key="pk-...",     secret_key="sk-...",     host="https://us.cloud.langfuse.com",

Network accessline 41

secret_key="sk-...",     host="https://us.cloud.langfuse.com", )

Network accessline 42
```
host="https://us.cloud.langfuse.com",
```

Network accessline 51

# Local / development (basic auth) provider = OpenSearchProvider(     host="https://localhost:9200",

Network accessline 52

provider = OpenSearchProvider(     host="https://localhost:9200",     auth=("admin", "password"),

Network accessline 53
```
host="https://localhost:9200",
```

Network accessline 63

auth = RequestsAWSV4SignerAuth(credentials, "us-east-1", "es") provider = OpenSearchProvider(     host="https://my-domain.us-east-1.es.amazonaws.com",

Network accessline 64

provider = OpenSearchProvider(     host="https://my-domain.us-east-1.es.amazonaws.com",     auth=auth,

Network accessline 65

host="https://my-domain.us-east-1.es.amazonaws.com",

`src/strands_evals/providers/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/providers/cloudwatch_provider.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 97
```
[Credential-related source omitted]
```

`src/strands_evals/providers/exceptions.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/providers/langfuse_provider.py`

Executable scriptline 1
```
Executable file
```

Network accessline 80

secret_key: Langfuse secret API key. Falls back to LANGFUSE_SECRET_KEY env var.             host: Langfuse API host URL. Falls back to LANGFUSE_HOST env var,                 defaulting to https://us.cloud.langfuse.com.

Network accessline 81

host: Langfuse API host URL. Falls back to LANGFUSE_HOST env var,                 defaulting to https://us.cloud.langfuse.com.             timeout: Request timeout in seconds.

Network accessline 82

defaulting to https://us.cloud.langfuse.com.

Network accessline 88
```
[Credential-related source omitted]
```
Access to secretsline 88
```
[Credential-related source omitted]
```
Access to secretsline 89
```
[Credential-related source omitted]
```
Network accessline 90
```
[Credential-related source omitted]
```
Access to secretsline 90
```
[Credential-related source omitted]
```

`src/strands_evals/providers/opensearch_provider.py`

Executable scriptline 1
```
Executable file
```

Network accessline 29

# Basic auth (local / development)         provider = OpenSearchProvider(             host="https://localhost:9200",

Network accessline 30

provider = OpenSearchProvider(             host="https://localhost:9200",             auth=("admin", "password"),

Network accessline 31
```
host="https://localhost:9200",
```

Network accessline 40

auth = RequestsAWSV4SignerAuth(credentials, "us-east-1", "es")         provider = OpenSearchProvider(             host="https://my-domain.us-east-1.es.amazonaws.com",

Network accessline 41

provider = OpenSearchProvider(             host="https://my-domain.us-east-1.es.amazonaws.com",             auth=auth,

Network accessline 42

host="https://my-domain.us-east-1.es.amazonaws.com",

Network accessline 47

def __init__(         self,         host: str = "https://localhost:9200",

Network accessline 48

self,         host: str = "https://localhost:9200",         index: str = "otel-v1-apm-span-*",

Network accessline 49
```
host: str = "https://localhost:9200",
```

`src/strands_evals/providers/trace_provider.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/README.md`

Network accessline 76

| Strategy | Mechanism | Attacker LLM? | Paper | |----------|-----------|---------------|-------| | `CrescendoStrategy` | Gradual multi-turn escalation, backtracking on refusals | yes | [arXiv:2404.01833](https://arxiv.org/abs/2404.01833) |

Network accessline 77

|----------|-----------|---------------|-------| | `CrescendoStrategy` | Gradual multi-turn escalation, backtracking on refusals | yes | [arXiv:2404.01833](https://arxiv.org/abs/2404.01833) | | `GoatStrategy` | Attacker LLM picks from a 7-a

Network accessline 78

| `CrescendoStrategy` | Gradual multi-turn escalation, backtracking on refusals | yes | [arXiv:2404.01833](https://arxiv.org/abs/2404.01833) |

Network accessline 79

| `GoatStrategy` | Attacker LLM picks from a 7-attack toolbox each turn | yes | [arXiv:2410.01606](https://arxiv.org/abs/2410.01606) |

Network accessline 80

| `PairStrategy` | Refines one prompt from per-turn judge feedback (PAIR, N=1) | yes | [arXiv:2310.08419](https://arxiv.org/abs/2310.08419) |

Network accessline 81

| `BadLikertJudgeStrategy` | Casts the target as a harmfulness-rating judge, elicits a top-score example | no | [Unit 42](https://unit42.paloaltonetworks.com/multi-turn-technique-jailbreaks-llms/) |

Network accessline 82

| `SequentialBreakStrategy` | Hides the harmful request among benign siblings in one narrative scaffold | no | [arXiv:2411.06426](https://arxiv.org/abs/2411.06426) |

`src/strands_evals/redteam/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/case.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/experiment.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/generators/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/generators/adversarial.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/generators/prompt_templates/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/generators/prompt_templates/adversarial_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/report.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/_common.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/bad_likert_judge/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/bad_likert_judge/bad_likert_judge_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/base.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/crescendo/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/crescendo/crescendo_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/goat/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/goat/goat_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/pair/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/pair/pair_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/prompt_strategy/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/prompt_strategy/gradual_escalation/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/prompt_strategy/gradual_escalation/gradual_escalation_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/sequentialbreak/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/sequentialbreak/sequentialbreak_v0.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/strategies/target_session.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/task.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/redteam/utils.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/actor_simulator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/profiles/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/profiles/actor_profile.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/prompt_templates/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/prompt_templates/actor_profile_extraction.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/prompt_templates/actor_system_prompt.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/prompt_templates/goal_completion.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/prompt_templates/tool_response_generation.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/tool_simulator.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/tools/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/simulation/tools/goal_completion.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/telemetry/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/telemetry/_cloudwatch_logger.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 29
```
[Credential-related source omitted]
```
Access to secretsline 47
```
[Credential-related source omitted]
```
Access to secretsline 58
```
[Credential-related source omitted]
```
Access to secretsline 63
```
[Credential-related source omitted]
```

`src/strands_evals/telemetry/config.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/telemetry/tracer.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/tools/evaluation_tools.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/detector.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/evaluation.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/evaluation_report.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/multimodal.py`

Executable scriptline 1
```
Executable file
```

Network accessline 67

try:         with urllib.request.urlopen(url, timeout=30) as response:  # noqa: S310

Network accessline 68

with urllib.request.urlopen(url, timeout=30) as response:  # noqa: S310

Possible obfuscationline 195
```
return base64.b64decode(parts[1])
```
Possible obfuscationline 205
```
return base64.b64decode(source)
```

`src/strands_evals/types/redteam/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/redteam/config.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/redteam/risk_category.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/redteam/run_results.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/simulation/init.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/simulation/actor.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/simulation/tool.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/types/trace.py`

Executable scriptline 1
```
Executable file
```

`src/strands_evals/utils.py`

Executable scriptline 1
```
Executable file
```

`tests/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_case.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_context.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_effects.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_experiment.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_failure_communication_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_partial_completion_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_plugin.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/chaos/test_recovery_strategy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/conftest.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/fixtures/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/fixtures/agents.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/fixtures/evaluators.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/fixtures/tasks.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/test_agent_task.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/test_diagnose.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/test_entrypoint.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/test_fetch.py`

Executable scriptline 1
```
Executable file
```

Network accessline 160

"sess_1",                 "--host",                 "https://lf.example.com",

Network accessline 161

"--host",                 "https://lf.example.com",                 "-o",

Network accessline 162
```
"https://lf.example.com",
```

Network accessline 168

assert exit_code == 0     lf_cls.assert_called_once_with(host="https://lf.example.com", timeout=120)

Network accessline 169

lf_cls.assert_called_once_with(host="https://lf.example.com", timeout=120)

Network accessline 204

"sess_1",                 "--host",                 "https://os.example.com",

Network accessline 205

"--host",                 "https://os.example.com",                 "--index",

Network accessline 206
```
"https://os.example.com",
```

Network accessline 219

assert exit_code == 0     os_cls.assert_called_once_with(         host="https://os.example.com",

Network accessline 220

os_cls.assert_called_once_with(         host="https://os.example.com",         index="my-spans-*",

Network accessline 221
```
host="https://os.example.com",
```

Network accessline 240

assert exit_code == 0     os_cls.assert_called_once_with(         host="https://localhost:9200",

Network accessline 241

os_cls.assert_called_once_with(         host="https://localhost:9200",         index="otel-v1-apm-span-*",

Network accessline 242
```
host="https://localhost:9200",
```

`tests/strands_evals/cli/test_generate.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/test_report.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/test_run.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cli/test_validate.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/cloudwatch_helpers.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/detectors/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/detectors/test_chunking.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/detectors/test_diagnosis.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/detectors/test_failure_detector.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/detectors/test_root_cause_analyzer.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/detectors/test_types.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/deterministic/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/deterministic/test_environment_state.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/deterministic/test_output.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/deterministic/test_skill_invoked.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/deterministic/test_structured_output.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/deterministic/test_trajectory.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/redteam/test_attack_success_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_coherence_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_conciseness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_correctness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_faithfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_goal_success_rate_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_harmfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_helpfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_instruction_following_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_interactions_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_multimodal_case_prompt_template.py`

Executable scriptline 1
```
Executable file
```

Possible obfuscationline 20

b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01"

Possible obfuscationline 21

b"\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89"

Possible obfuscationline 22

b"\x00\x00\x00\nIDATx\x9cc\x00\x01\x00\x00\x05\x00\x01"

Possible obfuscationline 23

b"\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"

`tests/strands_evals/evaluators/test_multimodal_output_evaluator.py`

Executable scriptline 1
```
Executable file
```

Possible obfuscationline 20

b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01"

Possible obfuscationline 21

b"\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89"

Possible obfuscationline 22

b"\x00\x00\x00\nIDATx\x9cc\x00\x01\x00\x00\x05\x00\x01"

Possible obfuscationline 23

b"\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"

`tests/strands_evals/evaluators/test_multimodal_specialized_evaluators.py`

Executable scriptline 1
```
Executable file
```

Possible obfuscationline 27

b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01"

Possible obfuscationline 28

b"\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89"

Possible obfuscationline 29

b"\x00\x00\x00\nIDATx\x9cc\x00\x01\x00\x00\x05\x00\x01"

Possible obfuscationline 30

b"\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"

`tests/strands_evals/evaluators/test_output_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_refusal_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_response_relevance_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_skill_instruction_following_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_skill_selection_accuracy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_stereotyping_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_tool_parameter_accuracy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_tool_selection_accuracy_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_trajectory_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/evaluators/test_trajectory_prompt_template.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/experimental/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/extractors/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/extractors/fixtures/capture_skill_fixtures.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/extractors/skill_fixtures.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/extractors/test_graph_extractor.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/extractors/test_skills.py`

Executable scriptline 1
```
Executable file
```

Dangerous shell operationline 574

"sudo cat /skills/pdf-processing/SKILL.md",

`tests/strands_evals/extractors/test_swarm_extractor.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/extractors/test_tools_use_extractor.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/extractors/test_trace_extractor.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/generators/test_experiment_generator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/generators/test_topic_planner.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/fixtures/claude_adot_spans.json`

Network accessline 253

"http.target": "/invocations",         "http.flavor": "1.1",         "http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",

Network accessline 254

"http.flavor": "1.1",         "http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",         "net.peer.ip": "127.0.0.1",

Network accessline 255

"http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",

`tests/strands_evals/mappers/fixtures/openai_agents_genai_adot_spans.json`

Network accessline 13

"aws.local.service": "CWExperiment_OpenaiTraceloopMulti.DEFAULT",         "telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/responses",

Network accessline 14

"telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/responses",         "aws.remote.service": "api.openai.com",

Network accessline 15

"http.url": "https://api.openai.com/v1/responses",

Network accessline 189

"aws.local.service": "CWExperiment_OpenaiTraceloopMulti.DEFAULT",         "telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/responses",

Network accessline 190

"telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/responses",         "aws.remote.service": "api.openai.com",

Network accessline 191

"http.url": "https://api.openai.com/v1/responses",

Network accessline 326

"aws.local.service": "CWExperiment_OpenaiTraceloopMulti.DEFAULT",         "telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/responses",

Network accessline 327

"telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/responses",         "aws.remote.service": "api.openai.com",

Network accessline 328

"http.url": "https://api.openai.com/v1/responses",

Network accessline 533

"http.target": "/invocations",         "http.flavor": "1.1",         "http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",

Network accessline 534

"http.flavor": "1.1",         "http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",         "net.peer.ip": "127.0.0.1",

Network accessline 535

"http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",

Network accessline 568

"aws.local.service": "CWExperiment_OpenaiTraceloopMulti.DEFAULT",         "telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/traces/ingest",

Network accessline 569

"telemetry.extended": "true",         "http.url": "https://api.openai.com/v1/traces/ingest",         "aws.remote.service": "api.openai.com",

Network accessline 570

"http.url": "https://api.openai.com/v1/traces/ingest",

`tests/strands_evals/mappers/fixtures/openai_agents_openinference_adot_spans.json`

Network accessline 10

"aws.local.service": "CWExperiment_OpenaiOpenInferenceMulti.DEFAULT",       "telemetry.extended": "true",       "http.url": "https://api.openai.com/v1/responses",

Network accessline 11

"telemetry.extended": "true",       "http.url": "https://api.openai.com/v1/responses",       "aws.remote.service": "api.openai.com",

Network accessline 12

"http.url": "https://api.openai.com/v1/responses",

Network accessline 170

"aws.local.service": "CWExperiment_OpenaiOpenInferenceMulti.DEFAULT",       "telemetry.extended": "true",       "http.url": "https://api.openai.com/v1/responses",

Network accessline 171

"telemetry.extended": "true",       "http.url": "https://api.openai.com/v1/responses",       "aws.remote.service": "api.openai.com",

Network accessline 172

"http.url": "https://api.openai.com/v1/responses",

Network accessline 319

"aws.local.service": "CWExperiment_OpenaiOpenInferenceMulti.DEFAULT",       "telemetry.extended": "true",       "http.url": "https://api.openai.com/v1/responses",

Network accessline 320

"telemetry.extended": "true",       "http.url": "https://api.openai.com/v1/responses",       "aws.remote.service": "api.openai.com",

Network accessline 321

"http.url": "https://api.openai.com/v1/responses",

Network accessline 524

"http.target": "/invocations",       "http.flavor": "1.1",       "http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",

Network accessline 525

"http.flavor": "1.1",       "http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",       "net.peer.ip": "127.0.0.1",

Network accessline 526

"http.url": "http://cell01.us-east-1.prod.arp.kepler-analytics.aws.dev/invocations",

`tests/strands_evals/mappers/fixtures/smolagents_live_spans.json`

Network accessline 13

"llm.input_messages.0.message.role": "system",       "llm.input_messages.0.message.contents.0.message_content.type": "text",       "llm.input_messages.0.message.contents.0.message_content.text": "You are an expert assistant who can solve an

Network accessline 14

"llm.input_messages.0.message.contents.0.message_content.type": "text",       "llm.input_messages.0.message.contents.0.message_content.text": "You are an expert assistant who can solve any task using code blobs. You will be given a task to

Network accessline 15

"llm.input_messages.0.message.contents.0.message_content.text": "You are an expert assistant who can solve any task using code blobs. You will be given a task to solve as best you can.\nTo do so, you have been given access to a list of tool

Network accessline 49

"tool.description": "Performs a duckduckgo web search based on your query (think a Google search) then returns the top search results.",       "tool.parameters": "{\"query\": {\"type\": \"string\", \"description\": \"The search query to per

Network accessline 50

"tool.parameters": "{\"query\": {\"type\": \"string\", \"description\": \"The search query to perform.\"}}",       "output.value": "## Search Results\n\n[Population of Tokelau](https://en.wikipedia.org/wiki/Population_of_Tokelau)\nTokelau (

Network accessline 51

"output.value": "## Search Results\n\n[Population of Tokelau](https://en.wikipedia.org/wiki/Population_of_Tokelau)\nTokelau ( ; lit. 'north-northeast' or 'north wind'; known previously as the Union Islands, and, until 1976, known officially

Network accessline 71

"attributes": {       "input.value": "{\"memory_step\": \"ActionStep(step_number=1, timing=Timing(start_time=1784230697.9639382, end_time=None, duration=None), model_input_messages=None, tool_calls=None, error=None, model_output_message=Non

Network accessline 72

"input.value": "{\"memory_step\": \"ActionStep(step_number=1, timing=Timing(start_time=1784230697.9639382, end_time=None, duration=None), model_input_messages=None, tool_calls=None, error=None, model_output_message=None, model_output=None,

Network accessline 73

"output.value": "Execution logs:\n## Search Results\n\n[Population of Tokelau](https://en.wikipedia.org/wiki/Population_of_Tokelau)\nTokelau ( ; lit. 'north-northeast' or 'north wind'; known previously as the Union Islands, and, until 1976,

Network accessline 96

"llm.input_messages.0.message.role": "system",       "llm.input_messages.0.message.contents.0.message_content.type": "text",       "llm.input_messages.0.message.contents.0.message_content.text": "You are an expert assistant who can solve an

Network accessline 97

"llm.input_messages.0.message.contents.0.message_content.type": "text",       "llm.input_messages.0.message.contents.0.message_content.text": "You are an expert assistant who can solve any task using code blobs. You will be given a task to

Network accessline 98

"llm.input_messages.0.message.contents.0.message_content.text": "You are an expert assistant who can solve any task using code blobs. You will be given a task to solve as best you can.\nTo do so, you have been given access to a list of tool

Network accessline 108

"llm.input_messages.4.message.role": "tool-response",       "llm.input_messages.4.message.contents.0.message_content.type": "text",       "llm.input_messages.4.message.contents.0.message_content.text": "Observation:\nExecution logs:\n## Sea

Network accessline 109

"llm.input_messages.4.message.contents.0.message_content.type": "text",       "llm.input_messages.4.message.contents.0.message_content.text": "Observation:\nExecution logs:\n## Search Results\n\n[Population of Tokelau](https://en.wikipedia.

Network accessline 110

"llm.input_messages.4.message.contents.0.message_content.text": "Observation:\nExecution logs:\n## Search Results\n\n[Population of Tokelau](https://en.wikipedia.org/wiki/Population_of_Tokelau)\nTokelau ( ; lit. 'north-northeast' or 'north

`tests/strands_evals/mappers/test_adk_otel_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_cloudwatch_parser.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_cloudwatch_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_generic_gen_ai_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_langchain_mapper_integration.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_langchain_otel_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_openai_agents_gen_ai_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_openinference_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_opensearch_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_session_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_strands_in_memory_mapper.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/mappers/test_utils.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/opensearch_helpers.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/providers/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/providers/test_cloudwatch_provider.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 102
```
[Credential-related source omitted]
```
Access to secretsline 106
```
[Credential-related source omitted]
```
Access to secretsline 113
```
[Credential-related source omitted]
```
Access to secretsline 117
```
[Credential-related source omitted]
```
Access to secretsline 126
```
[Credential-related source omitted]
```
Access to secretsline 135
```
[Credential-related source omitted]
```

`tests/strands_evals/providers/test_langfuse_provider.py`

Executable scriptline 1
```
Executable file
```

Network accessline 88

LangfuseProvider(public_key="pk-1", secret_key="sk-2", host="https://custom.langfuse.com")

Network accessline 89

cls.assert_called_once_with(public_key="pk-1", secret_key="sk-2", host="https://custom.langfuse.com")

Access to secretsline 92
```
[Credential-related source omitted]
```
Access to secretsline 94
```
[Credential-related source omitted]
```
Access to secretsline 98
```
[Credential-related source omitted]
```

Network accessline 103

LangfuseProvider()             cls.assert_called_once_with(public_key="pk-env", secret_key="sk-env", host="https://us.cloud.langfuse.com")

Network accessline 104
```
[Credential-related source omitted]
```
Access to secretsline 104
```
[Credential-related source omitted]
```

Network accessline 108

"LANGFUSE_PUBLIC_KEY": "pk-env",             "LANGFUSE_SECRET_KEY": "sk-env",             "LANGFUSE_HOST": "https://my-langfuse.example.com",

Network accessline 109

"LANGFUSE_SECRET_KEY": "sk-env",             "LANGFUSE_HOST": "https://my-langfuse.example.com",         }

Network accessline 110

"LANGFUSE_HOST": "https://my-langfuse.example.com",

Access to secretsline 113
```
[Credential-related source omitted]
```

Network accessline 118

LangfuseProvider()             cls.assert_called_once_with(                 public_key="pk-env", secret_key="sk-env", host="https://my-langfuse.example.com"

Network accessline 119

cls.assert_called_once_with(                 public_key="pk-env", secret_key="sk-env", host="https://my-langfuse.example.com"             )

Network accessline 120
```
[Credential-related source omitted]
```
Access to secretsline 120
```
[Credential-related source omitted]
```
Access to secretsline 124
```
[Credential-related source omitted]
```
Access to secretsline 125
```
[Credential-related source omitted]
```
Access to secretsline 126
```
[Credential-related source omitted]
```
Access to secretsline 128
```
[Credential-related source omitted]
```
Access to secretsline 138
```
[Credential-related source omitted]
```
Access to secretsline 141
```
[Credential-related source omitted]
```

Network accessline 146

LangfuseProvider(public_key="pk", secret_key="sk")             assert cls.call_args[1]["host"] == "https://us.cloud.langfuse.com"

Network accessline 147

assert cls.call_args[1]["host"] == "https://us.cloud.langfuse.com"

Network accessline 149

def test_explicit_host_overrides_env(self, mock_client):         env = {"LANGFUSE_HOST": "https://env-host.example.com"}

Network accessline 150

env = {"LANGFUSE_HOST": "https://env-host.example.com"}

Access to secretsline 152
```
[Credential-related source omitted]
```

Network accessline 157

LangfuseProvider(public_key="pk", secret_key="sk", host="https://explicit.example.com")

Network accessline 158

assert cls.call_args[1]["host"] == "https://explicit.example.com"

`tests/strands_evals/providers/test_opensearch_provider.py`

Executable scriptline 1
```
Executable file
```

Network accessline 126

with patch.dict("sys.modules", {"opensearch_genai_observability_sdk_py": None}):             with pytest.raises(ImportError, match="opensearch-genai-observability-sdk-py is required"):                 OpenSearchProvider(host="https://localh

Network accessline 128

OpenSearchProvider(host="https://localhost:9200")

`tests/strands_evals/providers/test_trace_provider.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_bad_likert_judge.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_case.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_crescendo.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_experiment.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_generator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_goat.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_multi_agent_session.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_pair.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_redteam_e2e.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_report.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_sequentialbreak.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_strategies.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_target_session.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/redteam/test_task.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/simulation/init.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/simulation/test_actor_simulator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/simulation/test_goal_completion.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/simulation/test_tool_simulator.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/telemetry/test_config.py`

Executable scriptline 1
```
Executable file
```

Network accessline 226

telemetry = StrandsEvalsTelemetry(tracer_provider=mock_tracer_provider)     telemetry.setup_otlp_exporter(endpoint="http://localhost:4318", headers={"key": "value"})

Network accessline 227

telemetry.setup_otlp_exporter(endpoint="http://localhost:4318", headers={"key": "value"})

`tests/strands_evals/telemetry/test_tracer.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/test_cases.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/test_eval_task.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/test_experiment.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/test_local_file_task_result_store.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/tools/test_evaluation_tools.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/types/redteam/test_risk_category.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/types/test_evaluation_report.py`

Executable scriptline 1
```
Executable file
```

`tests/strands_evals/types/test_multimodal.py`

Executable scriptline 1
```
Executable file
```

Possible obfuscationline 26

b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01"

Possible obfuscationline 27

b"\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89"

Possible obfuscationline 28

b"\x00\x00\x00\nIDATx\x9cc\x00\x01\x00\x00\x05\x00\x01"

Possible obfuscationline 29

b"\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82"

Network accessline 120

assert _detect_format("https://example.com/image.png") is None

Network accessline 160

def test_init_with_http_url(self):         url = "https://example.com/image.png"

Network accessline 161
```
url = "https://example.com/image.png"
```

Network accessline 252

def test_resolve_from_http_url(self, mock_download, sample_png_bytes):         mock_download.return_value = sample_png_bytes         image = ImageData(source="https://example.com/image.png")

Network accessline 253

mock_download.return_value = sample_png_bytes         image = ImageData(source="https://example.com/image.png")         result = resolve_image_bytes(image)

Network accessline 254

image = ImageData(source="https://example.com/image.png")

Network accessline 255

result = resolve_image_bytes(image)         assert result == sample_png_bytes         mock_download.assert_called_once_with("https://example.com/image.png")

Network accessline 257

mock_download.assert_called_once_with("https://example.com/image.png")

`tests/strands_evals/types/test_trace.py`

Executable scriptline 1
```
Executable file
```

`tests/test_integration.py`

Executable scriptline 1
```
Executable file
```

`tests_integ/conftest.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 24
```
[Credential-related source omitted]
```
Access to secretsline 26
```
[Credential-related source omitted]
```
Access to secretsline 32
```
[Credential-related source omitted]
```
Access to secretsline 40
```
[Credential-related source omitted]
```
Access to secretsline 41
```
[Credential-related source omitted]
```

`tests_integ/test_adk_eval.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 5
```
[Credential-related source omitted]
```
Access to secretsline 34
```
[Credential-related source omitted]
```
Access to secretsline 36
```
[Credential-related source omitted]
```

`tests_integ/test_claude_openinference_eval.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 38
```
[Credential-related source omitted]
```

`tests_integ/test_cloudwatch_provider.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 40
```
[Credential-related source omitted]
```
Access to secretsline 44
```
[Credential-related source omitted]
```
Access to secretsline 55
```
[Credential-related source omitted]
```

`tests_integ/test_harmfulness_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests_integ/test_instruction_following_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests_integ/test_langchain_openinference_eval.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 114
```
[Credential-related source omitted]
```

`tests_integ/test_langchain_traceloop_eval.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 117
```
[Credential-related source omitted]
```

`tests_integ/test_langfuse_provider.py`

Executable scriptline 1
```
Executable file
```
Access to secretsline 46
```
[Credential-related source omitted]
```

`tests_integ/test_multimodal_output_evaluator.py`

Executable scriptline 1
```
Executable file
```

Network accessline 22

CITY_IMAGE_URL = "https://picsum.photos/id/274/300/200.jpg"

Network accessline 23

FLOWER_IMAGE_URL = "https://picsum.photos/id/152/300/200.jpg"

Network accessline 34

with urllib.request.urlopen(CITY_IMAGE_URL, timeout=30) as resp:  # noqa: S310

Network accessline 41

with urllib.request.urlopen(FLOWER_IMAGE_URL, timeout=30) as resp:  # noqa: S310

`tests_integ/test_output_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests_integ/test_redteam.py`

Executable scriptline 1
```
Executable file
```

`tests_integ/test_refusal_evaluator.py`

Executable scriptline 1
```
Executable file
```

`tests_integ/test_stereotyping_evaluator.py`

Executable scriptline 1
```
Executable file
```

How the signals fit together

Facts from a static read of the files at the pinned commit. They describe what the code does, not why.

Some files could not be read; the facts below still hold.

Declared and executed: 2 facts

Contacts us.cloud.langfuse.com, which SKILL.md, the README and the plugin manifest do not mentionsrc/strands_evals/cli/commands/fetch.py:197High confidence
Contacts us.cloud.langfuse.com, which SKILL.md, the README and the plugin manifest do not mentionsrc/strands_evals/providers/langfuse_provider.py:90High confidence

Signal confidence: 498 facts

Network access in prose of the SKILL.md.agents/skills/session-converter/SKILL.md:24Low confidence
Network access in prose of the SKILL.md.agents/skills/session-converter/SKILL.md:29Low confidence
Network access in prose of the SKILL.md.agents/skills/session-converter/SKILL.md:34Low confidence
Executable script in code of a file that runs.github/scripts/group-release-notes.py:1High confidence
Access to secrets in code of a file that runs: a variable, placeholder or call, not a stored value.github/scripts/group-release-notes.py:62Low confidence
Access to secrets in code of a file that runs: a variable, placeholder or call, not a stored value.github/scripts/group-release-notes.py:63Low confidence
Network access in prose of documentationAGENTS.md:9Low confidence
Network access in prose of documentationAGENTS.md:34Low confidence
Network access in prose of documentationAGENTS.md:513Low confidence
Network access in prose of documentationAGENTS.md:514Low confidence
Network access in prose of documentationAGENTS.md:515Low confidence
Network access in prose of documentationAGENTS.md:516Low confidence
Network access in prose of documentationAGENTS.md:517Low confidence
Network access in prose of documentationCODE_OF_CONDUCT.md:1Low confidence
Network access in prose of documentationCODE_OF_CONDUCT.md:2Low confidence
Network access in prose of documentationCODE_OF_CONDUCT.md:3Low confidence
Network access in prose of documentationCONTRIBUTING.md:53Low confidence
Network access in prose of documentationCONTRIBUTING.md:106Low confidence
Network access in prose of documentationCONTRIBUTING.md:165Low confidence
Network access in prose of documentationCONTRIBUTING.md:166Low confidence
Network access in prose of documentationCONTRIBUTING.md:167Low confidence
Network access in prose of documentationCONTRIBUTING.md:172Low confidence
Network access in prose of documentationCONTRIBUTING.md:173Low confidence
Network access in prose of documentationCONTRIBUTING.md:174Low confidence
Network access in prose of documentationCONTRIBUTING.md:178Low confidence
Network access in prose of documentationCONTRIBUTING.md:179Low confidence
Network access in prose of documentationREADME.md:1Low confidence
Network access in prose of documentationREADME.md:2Low confidence
Network access in prose of documentationREADME.md:3Low confidence
Network access in prose of documentationREADME.md:4Low confidence
Network access in prose of documentationREADME.md:5Low confidence
Network access in prose of documentationREADME.md:6Low confidence
Network access in prose of documentationREADME.md:18Low confidence
Network access in prose of documentationREADME.md:19Low confidence
Network access in prose of documentationREADME.md:20Low confidence
Network access in prose of documentationREADME.md:21Low confidence
Network access in prose of documentationREADME.md:22Low confidence
Network access in prose of documentationREADME.md:23Low confidence
Network access in prose of documentationREADME.md:24Low confidence
Network access in prose of documentationREADME.md:25Low confidence
Network access in prose of documentationREADME.md:28Low confidence
Network access in prose of documentationREADME.md:29Low confidence
Network access in prose of documentationREADME.md:30Low confidence
Network access in prose of documentationREADME.md:31Low confidence
Network access in prose of documentationREADME.md:32Low confidence
Network access in prose of documentationREADME.md:33Low confidence
Network access in prose of documentationREADME.md:34Low confidence
Network access in prose of documentationREADME.md:35Low confidence
Network access in prose of documentationREADME.md:36Low confidence
Network access in prose of documentationREADME.md:37Low confidence

and 448 more

Only the first 500 facts are listed.

Works with

Agent compatibility, from the files in this repository
Agent	Status	Evidence	Install to
Claude Code	Compatible	Standard SKILL.md SKILL.md	`.claude/skills/<skill-name>/`
Codex	Native	Manifest AGENTS.md	`.agents/skills/<skill-name>/`
Cursor	Compatible	Standard SKILL.md SKILL.md	`.cursor/skills/<skill-name>/`
Gemini CLI	Compatible	Standard SKILL.md SKILL.md	`.gemini/skills/<skill-name>/`
GitHub Copilot	Compatible	Standard SKILL.md SKILL.md	`.github/skills/<skill-name>/`
OpenCode	Compatible	Standard SKILL.md SKILL.md	`.opencode/skills/<skill-name>/`

Native The repository ships a manifest, folder or frontmatter field for this agent.
Compatible A standard SKILL.md that this agent can read.
Unknown No evidence either way.

Derived automatically from paths, manifests and frontmatter. Not tested on each agent.

strands-agents/evals

History

SKILL.md

Share this page

Heat

Stars over time

Individual skills in this repository

How it works

strands-evals

Frontmatter

SKILL.md

Reference documents

The playbook

Scripts executed separately

Files

Skills included

Repository facts

Before you install

.agents/skills/session-converter/SKILL.md

.github/ISSUE_TEMPLATE/config.yml

.github/scripts/group-release-notes.py

.github/workflows/release.yml

.github/workflows/test-package-build.yml

.gitignore

AGENTS.md

CODE_OF_CONDUCT.md

CONTRIBUTING.md

LICENSE

README.md

SKILL.md

src/__init__.py

src/strands_evals/__init__.py

src/strands_evals/__main__.py

src/strands_evals/_async.py

src/strands_evals/case.py

src/strands_evals/chaos/__init__.py

src/strands_evals/chaos/_context.py

src/strands_evals/chaos/case.py

src/strands_evals/chaos/effects.py

src/strands_evals/chaos/experiment.py

src/strands_evals/chaos/plugin.py

src/strands_evals/cli/__init__.py

src/strands_evals/cli/__main__.py

src/strands_evals/cli/_agent_task.py

src/strands_evals/cli/_common.py

src/strands_evals/cli/_entrypoint.py

src/strands_evals/cli/_io.py

src/strands_evals/cli/commands/__init__.py

src/strands_evals/cli/commands/diagnose.py

src/strands_evals/cli/commands/fetch.py

src/strands_evals/cli/commands/generate.py

src/strands_evals/cli/commands/report.py

src/strands_evals/cli/commands/run.py

src/strands_evals/cli/commands/validate.py

src/strands_evals/cli/main.py

src/strands_evals/detectors/__init__.py

src/strands_evals/detectors/chunking.py

src/strands_evals/detectors/constants.py

src/strands_evals/detectors/diagnosis.py

src/strands_evals/detectors/failure_detector.py

src/strands_evals/detectors/prompt_templates/__init__.py

src/strands_evals/detectors/prompt_templates/failure_detection/__init__.py

src/strands_evals/detectors/prompt_templates/failure_detection/failure_detection_v0.py

src/strands_evals/detectors/prompt_templates/root_cause/__init__.py

src/strands_evals/detectors/prompt_templates/root_cause/root_cause_merge_v0.py

src/strands_evals/detectors/prompt_templates/root_cause/root_cause_v0.py

src/strands_evals/detectors/root_cause_analyzer.py

src/strands_evals/detectors/utils.py

src/strands_evals/display/display_console.py

src/strands_evals/eval_task_handler.py

src/strands_evals/evaluation_data_store.py

src/strands_evals/evaluators/__init__.py

src/strands_evals/evaluators/chaos/__init__.py

src/strands_evals/evaluators/chaos/failure_communication_evaluator.py

src/strands_evals/evaluators/chaos/partial_completion_evaluator.py

src/strands_evals/evaluators/chaos/prompt_templates/__init__.py

src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/__init__.py

src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/failure_communication_v0.py

src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/__init__.py

src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/partial_completion_v0.py

`SKILL.md`

`.agents/skills/session-converter/SKILL.md`

`.github/ISSUE_TEMPLATE/config.yml`

`.github/scripts/group-release-notes.py`

`.github/workflows/release.yml`

`.github/workflows/test-package-build.yml`

`.gitignore`

`AGENTS.md`

`CODE_OF_CONDUCT.md`

`CONTRIBUTING.md`

`LICENSE`

`README.md`

`SKILL.md`

`src/init.py`

`src/strands_evals/init.py`

`src/strands_evals/main.py`

`src/strands_evals/_async.py`

`src/strands_evals/case.py`

`src/strands_evals/chaos/init.py`

`src/strands_evals/chaos/_context.py`

`src/strands_evals/chaos/case.py`

`src/strands_evals/chaos/effects.py`

`src/strands_evals/chaos/experiment.py`

`src/strands_evals/chaos/plugin.py`

`src/strands_evals/cli/init.py`

`src/strands_evals/cli/main.py`

`src/strands_evals/cli/_agent_task.py`

`src/strands_evals/cli/_common.py`

`src/strands_evals/cli/_entrypoint.py`

`src/strands_evals/cli/_io.py`

`src/strands_evals/cli/commands/init.py`

`src/strands_evals/cli/commands/diagnose.py`

`src/strands_evals/cli/commands/fetch.py`

`src/strands_evals/cli/commands/generate.py`

`src/strands_evals/cli/commands/report.py`

`src/strands_evals/cli/commands/run.py`

`src/strands_evals/cli/commands/validate.py`

`src/strands_evals/cli/main.py`

`src/strands_evals/detectors/init.py`

`src/strands_evals/detectors/chunking.py`

`src/strands_evals/detectors/constants.py`

`src/strands_evals/detectors/diagnosis.py`

`src/strands_evals/detectors/failure_detector.py`

`src/strands_evals/detectors/prompt_templates/init.py`

`src/strands_evals/detectors/prompt_templates/failure_detection/init.py`

`src/strands_evals/detectors/prompt_templates/failure_detection/failure_detection_v0.py`

`src/strands_evals/detectors/prompt_templates/root_cause/init.py`

`src/strands_evals/detectors/prompt_templates/root_cause/root_cause_merge_v0.py`

`src/strands_evals/detectors/prompt_templates/root_cause/root_cause_v0.py`

`src/strands_evals/detectors/root_cause_analyzer.py`

`src/strands_evals/detectors/utils.py`

`src/strands_evals/display/display_console.py`

`src/strands_evals/eval_task_handler.py`

`src/strands_evals/evaluation_data_store.py`

`src/strands_evals/evaluators/init.py`

`src/strands_evals/evaluators/chaos/init.py`

`src/strands_evals/evaluators/chaos/failure_communication_evaluator.py`

`src/strands_evals/evaluators/chaos/partial_completion_evaluator.py`

`src/strands_evals/evaluators/chaos/prompt_templates/init.py`

`src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/init.py`

`src/strands_evals/evaluators/chaos/prompt_templates/failure_communication/failure_communication_v0.py`

`src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/init.py`

`src/strands_evals/evaluators/chaos/prompt_templates/partial_completion/partial_completion_v0.py`

`src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/init.py`

`src/strands_evals/evaluators/chaos/prompt_templates/recovery_strategy/recovery_strategy_v0.py`

`src/strands_evals/evaluators/chaos/recovery_strategy_evaluator.py`

`src/strands_evals/evaluators/coherence_evaluator.py`

`src/strands_evals/evaluators/conciseness_evaluator.py`

`src/strands_evals/evaluators/correctness_evaluator.py`

`src/strands_evals/evaluators/deterministic/init.py`

`src/strands_evals/evaluators/deterministic/environment_state.py`

`src/strands_evals/evaluators/deterministic/output.py`

`src/strands_evals/evaluators/deterministic/skill_invoked.py`

`src/strands_evals/evaluators/deterministic/structured_output.py`

`src/strands_evals/evaluators/deterministic/trajectory.py`

`src/strands_evals/evaluators/evaluator.py`

`src/strands_evals/evaluators/faithfulness_evaluator.py`

`src/strands_evals/evaluators/goal_success_rate_evaluator.py`

`src/strands_evals/evaluators/harmfulness_evaluator.py`

`src/strands_evals/evaluators/helpfulness_evaluator.py`