Toucan
A 1.5-million-sample synthetic tool-agentic dataset built from real-world MCP servers, used as a public comparison benchmark for tool-selection and task-to-scope matching research.
In our briefings
The claim each of the newest briefings on Toucan would have you leave with.
On the authors' own test tasks, a language model that checks an agent's requested scopes against its task did well with one tool and much worse with three. With one tool per task the LLM matcher reached 0.96 accuracy and 0.99 recall, against 0.55 recall for embedding similarity. With three tools its recall per requested tool fell to 0.57, and counting the combined effect of its decisions across each task, the false-negative rate reached 0.78. The authors call recall the most affected metric: the matcher increasingly refused tools the task needed. On the public Toucan set the same matcher's three-tool recall was 0.86, and the authors attribute the 29-point gap to their own tasks naming tools less directly. Their tests stop at three tools and single-turn prompts and rely on large proprietary models, and the wrong tool requests they test against are picked uniformly, with more deliberate requests that may mimic adversarial agents left to future work.
On its authors' tasks, an LLM check of agent scopes works at one tool and refuses most valid three-tool requestsBI-FIND-2025-0066
Toucan's server pool was built so that no tool call needs a credential. From an initial crawl of about 2,800 MCP servers on GitHub and Smithery, the authors kept only remote servers reachable over streamable HTTP and dropped any that require third-party credentials such as API keys, which left 871. Removing servers whose tools failed a test question left 495. The authors concede the cut "may overlook important servers and widely-used scenarios (e.g., Notion and GitHub)". We read this as training data in which calling a tool never depends on the agent holding a credential, so how a tuned model handles a sign-in prompt or a refused token is outside what it teaches.
Toucan's 1.5 million MCP tool-use examples come only from servers that need no API key or accountBI-FIND-2025-0153
Cited in 2 briefings
Every briefing that discusses this term, newest first.
- On its authors' tasks, an LLM check of agent scopes works at one tool and refuses most valid three-tool requests
Paper · · arXiv · Majed El Helou et al. · Oct 30, 2025
- Toucan's 1.5 million MCP tool-use examples come only from servers that need no API key or account
Paper · · arXiv · Zhangchen Xu et al. · Oct 1, 2025