Why
The durable pattern is that an autonomous reviewer is only as useful as its false-positive rate, and no announcement can tell you that number for your code. "High-confidence" is the tool's own label for its output, not a measurement against ground truth. A reviewer that flags twenty issues per pull request is worse than no reviewer, because the team learns to skim, and the one real finding gets skimmed with the rest. Precision is the adoption blocker, not recall.
The second pattern is that free moved the cost rather than removing it. The skill is free, and you supply the model, so the invoice becomes a token bill that scales with how often you run it. "A security check before every commit" is the expensive sentence in the pitch. Before wiring it into a hook, price one run against a real repository and multiply by your commit rate, then decide whether the gate belongs on every commit, on pull requests only, or on a nightly schedule over the whole tree.
The third pattern is that sub-agent fan-out buys coverage and creates a ranking problem. Several helpers scanning different files in parallel see more than one reviewer can, but they do not share a conclusion. What comes back is a pile with duplicates, contradictions and no agreed severity, so the value is decided in the merge step. That is the same shape as any parallel-agent job, and it is where a thin wrapper and a serious tool differ.
The fourth point is what this does not overlap with. A code-level agent answers "what is wrong in these files". An authority matrix answers "who can do this action alone". A contract can be free of the first kind of finding and still let a backend owner move funds by design, which no pattern scan will call a bug. The two layers are complementary, and running one does not discharge the other.
How it works
What the announcement supports
| Claim | Source | Status |
|---|---|---|
Installs as a Claude Code skill, invoked /audit | the demo terminal shows Skill(audit) loading | shown |
| Autonomous, with sub-agent helpers | the post | stated, not demonstrated in the screenshot |
| Open source, free with your own model | the post | stated |
Scans in-scope .sol files under a directory | the demo shows a default mode over nine files | shown |
| "Findings in minutes, not weeks" | the project page | a speed claim, not an accuracy claim |
| Not a substitute for a formal audit | the project page | the tool's own boundary, and the honest part |
The benchmark to build
| Step | What you record |
|---|---|
| 1. Freeze a clean baseline of three repos | commit hash per repo |
| 2. Inject one known bug per variant | bug class, file, line, expected finding |
| 3. Run the skill on every variant and on the clean baseline | wall time, model cost, findings |
| 4. Score | detected / missed on variants; findings on clean = false positives |
| 5. Decide the gate | pre-commit, pull request, or nightly — by cost and precision |
The number that matters
precision = true findings / all findings
Recall tells you what it caught. Precision tells you whether anyone will keep reading its output after week two. Report both, per bug class, and keep the clean-baseline run in the report — a tool that is silent on clean code has earned the right to be believed when it speaks.
Where it lands in Jayverse
- Auditor: two layers, one report. The authority matrix answers who can act alone; a code agent answers what is wrong in the files. Add a line to the report naming which question was asked, so a green matrix is never read as "the contracts are safe".
- Auditor: adopt the claim-boundary habit from the source. "Not a substitute for a formal audit" is exactly the sentence this project writes about itself, and the matrix should carry an equally plain one about its own scope.
- Token, DeFi and Rails: the seeded-bug benchmark is the deliverable. Three repos with injected bugs, scored for precision and recall, is reusable against any future scanner and is worth more than the verdict on this one tool.
- Devnet and CI: price the gate before wiring it. One run against a real repo times the commit rate decides pre-commit versus nightly. A security check nobody can afford to run is not a gate.
Claim boundary
The post and project page support the install-as-skill mechanism, the /audit invocation, open-source availability, bring-your-own-model pricing, sub-agent helpers, and the tool's own statement that it does not replace a formal audit. They do not publish a detection rate, a false-positive rate, a benchmark, or a list of vulnerability classes covered. Treat "already finding vulnerabilities in smart contracts" as a report of use, not as a measured result, until the benchmark above says otherwise.
Sources: pashov (@pashov) on X, the pinned post announcing the open-source Solidity Auditor — read 2026-09-21 from jay's screenshot, which also shows the project page's summary and the demo terminal.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| an extra pair of eyes | 한 사람 더 봐 주는 것 · 리뷰를 부탁할 때. "anyone who wants an extra pair of eyes" |
| not a substitute for X | X의 대체물은 아니다 · 도구의 한계를 정직하게 그을 때 |
| a fast win | 빨리 얻는 성과 · "researchers looking for fast wins before a manual review" |
| in scope / out of scope | 검토 범위 안/밖 · 감사 대화의 첫 합의 사항 |
| a false positive | 거짓 양성(문제가 없는데 문제라고 한 것) · 스캐너 평가의 핵심 숫자 |
| precision / recall | 정밀도(맞힌 비율) / 재현율(찾아낸 비율) · 둘을 같이 보고한다 |
| ground truth | 정답 데이터 · "a measurement against ground truth" |
| to seed a bug | 버그를 일부러 심다 · 벤치마크를 만들 때 |
| a baseline | 기준선 · 비교 대상이 되는 깨끗한 상태 |
| a gate | 통과 관문(막는 검사) · pre-commit gate, CI gate |
| to wire something into X | ~을 X에 연결해 넣다 · "before wiring it into a hook" |
| fan-out | 병렬로 퍼뜨려 실행하기 · 서브에이전트 여러 개를 동시에 돌릴 때 |
| to triage | 우선순위를 분류하다 · 발견 더미를 정리할 때 |
| to skim | 대충 훑다 · "the team learns to skim" |
| to discharge an obligation | 의무를 면하다 · "running one does not discharge the other" |
| a claim boundary | 주장의 경계 · 발표가 뒷받침하는 범위를 적어 두는 습관 |