Optimizing AI Agent Skill Invocation for Engineering Workflows

The increasing adoption of AI agents in engineering workflows, particularly for automating tasks and providing domain-specific knowledge, introduces challenges around their reliability. Pinterest engineers observed that agents, specifically internal forks of OpenAI's Codex (Pin-agent) and Claude Code, sometimes failed to invoke custom skills, leading to inefficiencies. This prompted the development of a testing methodology to understand and optimize skill invocation performance.

Building a Reliable Skill Test Harness

To address the unreliability, a test harness was built to systematically evaluate agent performance. This harness consists of a Bash script that orchestrates automated testing by piping prompts to agents and capturing verbose output logs. The core execution flow involves running a suite of categorized prompts (positive and negative cases) multiple times to account for the non-deterministic nature of AI agents.

if echo "$prompt" | claude --print --verbose --output-format stream-json > "$log_file" 2>&1; then
  command_success=true
fi

Log parsing heuristics are then applied to the JSON-streamed debug output to detect successful skill invocations. This involves searching for specific patterns that indicate the agent has loaded and utilized the intended skill. Key metrics tracked include CORE_SUCCESS_RATE, EDGE_FALSE_POSITIVE_RATE, and OVERALL_ACCURACY to provide a quantifiable measure of performance.

Optimizations for Improved Skill Invocation

Initial testing revealed that agents had an overall accuracy of 73% (Codex) and 62% (Claude) for skill invocation, which was deemed unacceptable for critical workflows. Several optimization techniques were identified and implemented:

• Frontmatter Description: Including rich contextual information and architectural components in the skill's YAML description significantly improved performance, agnostic to the agent used.
• Aggressive Language: Using explicit, all-caps commands like "YOU MUST LOAD THIS SKILL IF" in the frontmatter was shown to boost signal importance.
• AGENTS.md File: Adding a table of skills with usage rationales to an `AGENTS.md` file also contributed to better loading rates, though teams need to balance this with context window token limits.
• Combination of Techniques: Applying multiple techniques concurrently yielded compound gains, particularly for Codex users. Surprisingly, asking agents to self-improve on these additions sometimes decreased invocation rates.