{
  "id": "page:docs-reference-repos-a5c-ai-obedience-benchmark-research",
  "_kind": "Page",
  "_file": "wiki/docs/reference-repos/a5c-ai/obedience-benchmark/research.md",
  "_cluster": "wiki",
  "attributes": {
    "nodeKind": "Page",
    "sourcePath": "docs/reference-repos/a5c-ai/obedience-benchmark/research.md",
    "sourceKind": "repo-docs",
    "title": "a5c-ai/obedience-benchmark",
    "displayName": "a5c-ai/obedience-benchmark",
    "slug": "docs/reference-repos/a5c-ai/obedience-benchmark/research",
    "articlePath": "wiki/docs/reference-repos/a5c-ai/obedience-benchmark/research.md",
    "article": "\n# a5c-ai/obedience-benchmark\n\n- **Archetype**: claude-plugin-benchmark\n- **Stars**: 1\n- **Last pushed**: 2026-03-10\n- **License**: (Not specified)\n- **Discovered**: 2026-04-13\n- **Skills found**: Multiple (process fidelity benchmarking skills)\n\n## Summary\nClaude Code plugin that measures whether AI agents follow prescribed workflows rather than just producing correct outputs. Evaluates across 7 dimensions: completeness, ordering, conditionality, parallelism, granularity, aggregation, and error handling. Features task catalog management, process definition using babysitter SDK, execution runners (local subprocess or Docker container modes), LLM-as-Judge scoring, and markdown report generation with dimension breakdowns.\n\n## Assessment\nEXTREMELY HIGH VALUE for babysitter validation. This benchmark directly addresses a critical gap in agent evaluation by focusing on process fidelity rather than just output correctness. The 7-dimension evaluation framework, babysitter SDK integration for process definition, and LLM-as-Judge scoring provide sophisticated methodologies for validating babysitter process compliance. The decoupling of \"obedience\" from \"correctness\" as orthogonal capabilities is essential for deployment trust.\n\n## Extraction Priority\n- Very High\n- Rationale: This benchmark framework is specifically designed for babysitter process validation and could become an essential component of babysitter's quality assurance. The process fidelity evaluation and 7-dimension scoring framework are directly applicable to babysitter process improvement and validation workflows.\n\n## Processes\n- **Process Fidelity Benchmarking**: Evaluate agent adherence to prescribed workflows across 7 dimensions\n- **LLM-as-Judge Process Scoring**: Analyze execution logs against process specifications for compliance assessment\n- **Multi-Dimensional Process Evaluation**: Assess completeness, ordering, conditionality, parallelism, granularity, aggregation, error handling\n- **Process Definition Validation**: Author and validate prescribed workflows using babysitter SDK patterns\n- **Execution Environment Management**: Local subprocess and Docker container modes for isolated agent testing\n\n## Plugin Ideas\n- **Obedience Benchmark Integration**: Direct integration with obedience-benchmark for continuous process validation\n\n## Patterns\n- Process fidelity vs output correctness as orthogonal evaluation dimensions\n- 7-dimension evaluation framework for workflow compliance\n- LLM-as-Judge scoring methodology\n- Babysitter SDK integration for process definition\n- Multi-mode execution environments (local, Docker)\n- Task catalog with domain/complexity filtering\n- Markdown scorecard generation with dimension breakdowns\n\n## Library Mapping\n\n| Extractable Process | Library Status | Action | Existing Path | Target Placement |\n|-------------------|----------------|--------|---------------|------------------|\n| Process Fidelity Benchmarking | NEW | Evaluate agent adherence to prescribed workflows across multiple dimensions | - | methodologies/process-fidelity-benchmarking/ |\n| LLM-as-Judge Process Scoring | NEW | Analyze execution logs against process specifications for compliance | - | specializations/shared/llm-as-judge-process-scoring.js |\n| Multi-Dimensional Process Evaluation | NEW | 7-dimension assessment framework for workflow compliance | - | specializations/shared/multi-dimensional-process-evaluation.js |\n| Process Definition Validation | UPGRADE | Enhanced process validation using babysitter SDK patterns | library/tasks/ | specializations/shared/process-definition-validation.js |\n| Execution Environment Management | NEW | Multi-mode execution environments for isolated process testing | - | specializations/shared/execution-environment-management.js |\n| Task Catalog Management | NEW | Organize and filter benchmark tasks by domain and complexity | - | specializations/shared/task-catalog-management.js |\n| Compliance Dimension Scoring | NEW | Systematic scoring across completeness, ordering, conditionality, etc. | - | specializations/shared/compliance-dimension-scoring.js |\n\n## Plugin Marketplace Mapping\n\n| Plugin Idea | Marketplace Status | Action | Existing Plugin | Target Placement |\n|-------------|-------------------|--------|-----------------|------------------|\n| Obedience Benchmark Integration | NEW | Direct integration with obedience-benchmark for continuous process validation | - | plugins/a5c/marketplace/blueprints/obedience-benchmark-integration/ |\n",
    "documents": []
  },
  "outgoingEdges": [],
  "incomingEdges": [
    {
      "from": "page:docs-reference-repos",
      "to": "page:docs-reference-repos-a5c-ai-obedience-benchmark-research",
      "kind": "contains_page"
    }
  ]
}