I Audited 7,522 AI Agent Skills. Here's What I Found.
Last month someone forked my governance repo to distribute malware. That was one package. It got me thinking about the rest of them. ClawHub has 7,500+ skills that people install and hand full tool access to. What’s actually in there?
I decided to scan the entire registry.
How I did it
I wrote a scanner that pulls every skill from ClawHub, downloads the scannable files — SKILL.md, source code, configs — and runs 40 regex patterns against them. The patterns come from published research: Snyk’s ToxicSkills report, Cisco’s MCP scanner, Kaspersky’s work on indirect prompt injection. The categories cover the obvious attack surface: prompt injection, credential exfiltration, reverse shells, hardcoded secrets, obfuscated payloads, data exfil endpoints.
The whole thing runs in Docker with --network=none. No skill code ever touches a live network during analysis. Download phase pulls the files, then the container goes fully airgapped.
Important caveat: this is static text analysis. Regex against file contents. It catches what’s written in code and documentation — hardcoded secrets, suspicious shell commands, injection patterns. It does not catch a skill that exfiltrates data through legitimate-looking API calls, or one that behaves differently at runtime than its source suggests. Static analysis is a floor, not a ceiling.
First pass: a lot of noise
7,522 skills scanned. 4,931 findings across 746 skills. 61% estimated false positive rate.
The top triggered pattern was api_key|token|secret with 1,923 hits. Almost all documentation. Every API integration skill has a SKILL.md that says “set your STRIPE_API_KEY environment variable” or shows a curl example with $TOKEN. That’s a readme, not exfiltration.
Second noisiest: webhook with 979 hits. The word appears in every integration skill that handles callbacks. Only a handful of actual exfiltration indicators — requestbin, pipedream, hookbin, burpcollaborator — were in there.
~/.config triggered 400 times. Nearly all benign. ~/.config/openclaw/calendar.json is not a threat.
So I built a second pass.
Deep analysis
The second-pass analyzer reads every finding, pulls surrounding context from the actual file, and applies validation rules:
- Noise filtering: generic “webhook” without exfil domains → noise.
~/.configwithout sensitive dirs → noise. - Placeholder detection: if the “hardcoded secret” contains
your_api_keyorsk-...or<placeholder>→ benign. If it referencesprocess.env→ benign. - Base64 decoding: for every
Buffer.from(...)match, decode the payload and check contents. - File location weighting: a finding in SKILL.md (the file that instructs the AI agent) is weighted 3x higher. A finding in
references/README.mdis weighted 0.5x. - Context checks: if the match is preceded by “Example:” or lives inside a documentation code fence → reduce confidence.
Result after deep analysis: 4,931 findings reduced to 1,397 actionable. 72% noise reduction. 284 likely threats. 16 confirmed threats.
What the scanner flagged as confirmed
I manually reviewed every confirmed threat. All 16 of them.
Zero are malicious.
-
5 are security tools documenting attack patterns for defensive purposes.
guardian-angelandindirect-prompt-injectionlist “ignore previous instructions” as examples of what to detect, not what to do. -
5 are benign usage.
docker-sandboxusesbash -cto run builds in isolated containers.dns-networkinguses/dev/tcp/for port connectivity checks. That’s how you check if a port is open. -
4 are false positives.
calendar-remindersmatched “print instructions” — it’s telling users how to print their calendar config.snapmaker-2matched “override safety” — it’s about overriding 3D printer safety limits, not AI safety. -
1 is a prompt injection example in a backup file.
proactive-agenthas “ignore previous instructions” inSKILL-v2.3-backup.md. 14,488 downloads. Likely a leftover from testing, not active in the current version.
The irony: good security documentation triggers security scanners. The skills that were doing the most to protect users were the ones that generated the most findings.
The “likely” category
The 284 likely threats are almost entirely curl commands with API tokens in SKILL.md files. curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" is not credential exfiltration — it’s API documentation. The scanner correctly identifies that a SKILL.md file is instructing an AI agent to use curl with credentials. But that’s what API skills do.
The few genuinely interesting ones:
- One skill references pipedream endpoints — a known data exfiltration service.
- Several skills have hardcoded strings that look like real API keys (20+ alphanumeric characters, not matching any placeholder pattern). Uncertain whether they’re live credentials or test values.
- 1 confirmed credential leak across the entire registry.
ClawHub’s own moderation misses most of this
ClawHub flags 978 skills as “suspicious.” My scanner found findings in 746 skills. The overlap? 315 skills. Jaccard agreement: 22%.
After removing noise from my results, agreement drops to about 10%. They’re flagging different things. Neither system catches the right things with high confidence.
ClawHub missed the real credential leak. They flagged calendar-reminders, which is a false positive. Neither system — theirs or mine — is reliable as a standalone detection layer.
What these numbers actually mean
The ClawHub ecosystem is cleaner than the headlines suggest. Out of 7,522 skills:
| Metric | Value |
|---|---|
| Skills with zero pattern matches | 90.1% |
| Confirmed malicious skills | 0 |
| Confirmed credential leaks | 1 |
| Findings that are documentation noise | ~72% |
But “clean today” doesn’t mean “clean tomorrow.” The registry is growing. Skills are getting more powerful — more tool access, more credential scopes, more filesystem interaction. Snyk found 12% of published skills were compromised at one point — those were cleaned up, but the attack surface is only expanding.
And static analysis has a fundamental ceiling. A skill designed to be subtle — one that exfiltrates data through legitimate-looking API calls or only activates under specific conditions — wouldn’t show up in regex patterns. This is a lower bound on safety, not a guarantee.
What this means for how you run agents
The supply chain isn’t the only attack surface, and scanning isn’t the only defense. Even if every skill in the registry were perfectly clean, you’d still have this problem: the skills your agents use have broad access to your systems, and your backend can’t tell which user initiated which tool call.
A clean skill with a shared API key is still a skill with a shared API key.
This is the gap an Agentic Control Plane fills. Not by scanning the supply chain (though that matters), but by enforcing identity, permissions, and audit at the infrastructure layer where the agent meets your backend:
- Every tool call carries the user’s identity — not a shared service account
- Deny-by-default scoping — the agent only accesses what the user is authorized to access
- Injection detection at runtime — the same 40+ patterns from this audit, applied to every tool invocation, not just at install time
- Full audit trail — who asked, what was called, what was returned, timestamped
Registry scanning tells you what’s in the package. Runtime governance tells you what the package is actually doing with your data.
The raw data
The full audit report, scanner source, and the complete ClawHub registry snapshot (9,781 skills with metadata) are available in the audit-data repository. Run it yourself, disagree with my methodology, improve the patterns. That’s the point.
What to do next
If you’re running agents in production today, start with the audit question: can your backend tell which user initiated each tool call? If every request shows the same service account, you have the identity gap regardless of how clean the supply chain is.
If you want help closing that gap — identity binding, deny-by-default scoping, runtime governance, audit trails — talk to us. We’ll walk through your agent stack and show you exactly where the exposure is and how to close it. No pitch deck, just architecture.
Read the reference architecture → · Get started free →