Security Blind Spots in AI Skill Scanners: How Test Files Bypass Detection

Recent security research has uncovered a critical vulnerability in how Anthropic Skill scanners operate. While current tools effectively scan main Skill files for malicious content, they completely overlook test files (.test.ts) that can execute dangerous code with full system access. This Q&A explains the attack vector, why it evades detection, and what large-scale security audits reveal about the broader risks in AI Skill marketplaces.

What is the new attack vector discovered in Anthropic Skills?

Security researcher Jeevan Jutla at Gecko demonstrated that malicious code can be hidden in .test.ts files bundled with an Anthropic Skill. When a developer runs npx Skills add, the installer copies the entire Skill directory—including test files—into the project repository. Testing frameworks like Jest and Vitest automatically discover these files through recursive glob patterns and execute them during npm test or even when an IDE auto-runs tests on save. The payload fires in beforeAll hooks, before any assertions run, giving it full access to environment variables, SSH keys, and other secrets present in the CI pipeline. The test output shows no unusual flags, making the attack nearly invisible.

Why didn't the Anthropic Skill scanner detect this threat?

The Anthropic Skill scanner only inspects files that are part of the agent execution surface, such as SKILL.md and direct instruction files. It never examines the .test.ts file sitting one directory over because test files are not considered part of the agent's runtime. As of publication, no publicly documented scanner inspects test files. This represents a fundamental blind spot: the scanner reads the right files for the wrong threat model. The agent is never invoked, yet the malicious code executes through the test runner with full system access. Green flags across the board for the scanner, while the actual threat goes undetected.

How does this attack compare to previous installation-based exploits?

Attack classes based on trust-on-install are not new. Malicious npm postinstall scripts and pytest plugins have exploited similar gaps for years. However, the Skill vector makes the attack significantly worse. Unlike npm packages that stay isolated in node_modules, installed Skills land in a directory designed to be committed to version control and shared across the entire team. Every teammate who clones the repository triggers the same malicious test file execution. Because the Skill directory sits outside every scanner's detection surface, the vulnerability propagates unchecked across the development lifecycle, compounding the risk of credential leakage and supply chain compromise.

What did the SkillScan academic study reveal about Skill vulnerabilities?

Published on January 15, the SkillScan study analyzed 31,132 unique Anthropic Skills from two major marketplaces. The findings were alarming: 26.1% of Skills contained at least one vulnerability spanning 14 distinct patterns across four categories. Data exfiltration appeared in 13.3% of Skills, while privilege escalation was present in 11.8%. Notably, Skills that bundled executable scripts were 2.12 times more likely to contain vulnerabilities than instruction-only Skills. This highlights a direct correlation between script inclusion and risk, yet current scanners still fail to inspect those script files, especially when they are disguised as test files.

What did Snyk's ToxicSkills audit conclude?

Three weeks after the SkillScan study, Snyk published the ToxicSkills audit—the first comprehensive security scan of the ClawHub and skills.sh marketplaces. The team examined 3,984 Skills as of February 5 and found that 13.4% of all Skills contained at least one critical vulnerability. This figure, while lower than the academic study's 26.1%, still represents a substantial threat given the thousands of Skills in active use. The discrepancy likely stems from different scanning methodologies and the fact that Snyk's audit focused on the execution surface that scanners do inspect, whereas Gecko's disclosure highlights what scanners miss.

How can developers protect against this attack vector?

Protection requires a multi-layered approach. First, never trust test files from external sources without thorough manual review. Second, configure your CI pipeline to restrict environment variable access during test runs, especially for Skills from untrusted marketplaces. Third, use recursive glob pattern exclusions in your test configuration to prevent automatic discovery of files in Skill directories. Tools like .jestignore or .mocharc settings can help. Fourth, consider running npm test in a sandboxed environment or container with minimal secrets. Finally, advocate for community-driven security audits that specifically inspect test files, not just main Skill files. Until scanners evolve, manual code review remains the last line of defense against blind spots like this one.