Web Crawler

This web crawler MCP server, implemented in TypeScript, provides a flexible and configurable tool for crawling websites and extracting content. It integrates with the Model Context Protocol SDK and uses libraries like Axios and Cheerio for efficient web scraping. The crawler respects robots.txt rules, handles concurrent requests, and offers customizable depth, delay, and timeout settings. It stands out by providing a simple JSON interface for initiating crawls and retrieving structured results, making it ideal for AI systems or applications that need to gather web content programmatically. Use cases include data collection for training models, content aggregation, or web monitoring tasks.