eval-integrityAudit an LLM evaluation or benchmark repo for integrity and credibility practices. Use when asked to "audit my benchmark," "is my eval trustworthy," "check my leaderboard for contamination," "review this benchmark's methodology," or "what would a reviewer attack in my eval." Greps the target repo for evidence across seven dimensions (pre-registration, contamination, holdout hygiene, judge validity, statistical honesty, reproducibility, leaderboard exclusions) and emits a scored report with file:line evidence, severity, and concrete fixes.
Install via ClawdBot CLI:
clawdbot install conorbronsdon/eval-integrityGrade Fair — based on market validation, documentation quality, package completeness, maintenance status, and authenticity signals.
Calls external URL not in known-safe list
https://github.com/conorbronsdon/eval-integrityAudited Sep 12, 2026 · audit v1.0
Generated Oct 7, 2026
A research lab is about to submit an LLM benchmark to a grant panel and needs to prove its headline numbers will survive adversarial scrutiny. The skill greps the repo for missing integrity scaffolding across seven dimensions and produces a scored report with file:line evidence. This lets the team fix disqualifying gaps before submission rather than after rejection.
An open-source benchmark maintainer is receiving complaints that leaderboard scores are contaminated or unreproducible. The skill audits contamination controls, holdout hygiene, and publish mechanics to determine whether the criticism is valid. Findings distinguish score-invalidating gaps from nice-to-have improvements.
A procurement team is evaluating vendor benchmark claims before signing a contract. The skill audits the vendor's public eval repo for judge validity, statistical honesty, and reproducibility so the team can quantify how much to trust the reported numbers. It produces a severity-ranked report suitable for legal and technical reviewers.
Authors of a benchmark paper expect reviewers to attack their methodology and want a preemptive list of likely criticisms. The skill identifies gaps a reviewer would target, such as missing confidence intervals or unpinned judge versions. Authors can address each gap directly in the rebuttal or camera-ready revision.
An AI platform team requires every internal benchmark to pass an integrity audit before its scores can be used in model release decisions. The skill runs as a static analysis step in the release pipeline, flagging leaderboard exclusions, contamination risks, and reproducibility failures. This prevents unreliable benchmark results from influencing deployment choices.
The core auditor is free and open-source for individual researchers running it locally. Teams pay for hosted CI integration that runs the integrity audit automatically on every pull request and tracks scores over time.
Organizations in regulated industries license the skill as part of a model governance toolchain. It provides auditable evidence, severity reports, and policy enforcement for benchmark usage before model deployment.
Experts use the skill to offer benchmark integrity certification services. Vendors pay for an independent audit and a public seal indicating their eval passed the seven-dimension review.
💬 Integration Tip
Run the skill as a read-only CI check before benchmark publication, and treat any finding that invalidates published numbers as a blocking gate rather than a warning. Ensure the subagent environment supports parallel tool calls so the seven dimension auditors can run in a single batch.
Scored Oct 7, 2026
Medical device risk management specialist implementing ISO 14971 throughout product lifecycle. Provides risk analysis, risk evaluation, risk control, and pos...
When the user wants to plan a product launch, feature announcement, or release strategy. Also use when the user mentions 'launch,' 'Product Hunt,' 'feature r...
When the user wants to build a free tool for marketing — lead generation, SEO value, or brand awareness. Use when they mention 'engineering as marketing,' 'f...
管理多类型项目看板,支持新增项目、变更状态与版本、分类管理及查看项目总览和变更日志。
Competitor Analysis — SEO/GEO Intelligence & Market Positioning. Analyze competitor SEO rankings, AI search citations, content strategy, and market posi...
Conduct structured PHQ-9 depression symptom screening and submit the completed assessment for evaluation.