TIGER-AI-Lab/ClawBench

Problem The Harbor compatibility test is skipped when Harbor is not installed, and CI does not execute a real harbor run. The Harbor runtime Dockerfile is also outside the current harness image build scan. This leaves adapter, image, runtime readiness, verifier, and cleanup regressions undetected. Required scope Pin Harbor 0.22.x and Python 3.12 in a Linux Docker job. Convert one real V2 task and validate it with Harbor's loader. Run a single Harbor task smoke covering setup, runtime readiness, verifier, artifact collection, and cleanup. Include src/clawbench/runtime/harbor/Dockerfile in the image build matrix. Upload task/runtime/verifier logs on failure and record Harbor/schema/image versions. Acceptance criteria Adapter/runtime changes require the Harbor smoke job. The job exercises successful interception, no interception, judge-inconclusive, and cleanup failure paths. Harbor artifacts are available for debugging without exposing credentials. This is HAR-001 in docs/codebase-ev

Problem .env is tracked by Git and pyproject.toml force-includes it in wheel/sdist artifacts. The file contains a non-placeholder PurelyMail API key and must be treated as exposed. Required scope Revoke and rotate the current PurelyMail credential. Remove .env from Git history and package inclusion; add a safe .env.example. Load credentials only from CI/runtime secrets. Add secret scanning and a release check that wheel/sdist artifacts contain no credentials. Update Harbor/native documentation so credentials are injected, never packaged. Acceptance criteria git ls-files .env is empty. Built wheel and sdist contain no .env or credential values. CI fails on newly introduced secrets. Historical release artifacts are audited and the rotation is documented. This is tracked as SEC-001 in docs/codebase-evaluation-2026-09.md. @Perry2004 please review ownership and the rotation plan.

Problem Generated Harbor tasks use network_mode = "public" and run stock agents with shell access. The instruction says to use only the browser, but an agent can still call a site API with curl, Python, Node, or another direct client. That can invalidate benchmark results. Required scope Define the browser-only policy and an explicit egress allowlist for sites, model endpoints, judge, PurelyMail, and browser providers. Enforce the policy at the container/tool boundary, not only in instruction.md. Detect and preserve evidence of blocked direct requests or disallowed tools. Emit failure_category = "policy_violation" without conflating it with an agent mismatch. Acceptance criteria Direct API/HTTP submission attempts are blocked or recorded as policy violations. Legitimate CDP/browser flows continue to work. Harbor and native result summaries can separate policy violations from agent failures. This is HAR-002 in docs/codebase-evaluation-2026-09.md and is related to #331. @Perry2004 pl

Problem The runtime server writes intercepted request headers to requests.jsonl without a default redaction policy. Authorization, Cookie, Set-Cookie, CSRF tokens, and provider credentials can therefore leak into shareable benchmark artifacts. Required scope Add a default header allowlist or sensitive-header redaction for persisted requests. Redact token-like fields in headers and bounded request bodies where needed. Add an explicit retention/configuration contract and fixture tests. Keep enough metadata to explain that a value was redacted. Acceptance criteria Secrets in test fixtures never appear in requests.jsonl, summaries, or uploaded artifacts. Redaction behavior is deterministic and covered by tests. Harbor and native paths use the same artifact policy. This is SEC-002 in docs/codebase-evaluation-2026-09.md. @Perry2004 please review the privacy/retention policy.