Web search and data for agents: 8 companies ranked

Companies selling search, crawl and extraction APIs built for AI agents, in SalesBook agentic-sector rank order. Every company has a profile with dated public sources and a live check of its agent interfaces (MCP, llms.txt). Ranks use the published SalesBook method; tell us who is missing.

• You.com#3Web search, contents and research APIs for AI agents and LLM developers, with enterprise plans.
  MCP connector: nollms.txt: yes
• Exa#15Search API built for AI agents that returns real-time web results and page content, sold pay-as-you-go and to enterprises.
  MCP connector: yesllms.txt: yes
• Parallel Web Systems#32Web search, extraction and research APIs built for AI agents rather than human readers, sold on pay-as-you-go and enterprise terms.
  MCP connector: yesllms.txt: yes
• Linkup#35Web search API for AI applications and agents, sold to enterprises, AI labs and financial institutions, with in-cloud deployment.
  MCP connector: nollms.txt: no
• Firecrawl#52Web data API that crawls and converts websites into clean data for AI agents and LLM apps, with free and paid plans.
  MCP connector: yesllms.txt: yes
• Olostep#74Scrape, crawl, search and answer APIs that give AI agents web data, sold to AI application companies.
  MCP connector: yesllms.txt: yes
• Diffbot#89Web extraction, crawl, knowledge graph and search APIs that ground AI applications on structured web data.
  MCP connector: yesllms.txt: yes
• Keenable#101Independent web search index and API for AI agents, with live search and page fetch priced per request.
  MCP connector: yesllms.txt: yes