Docs Scraper
Summary
Documentation scraping server that enables AI assistants to extract structured content from web-based documentation through multiple crawling strategies. Built with Python and the crawl4ai library, it provides tools for single URL crawling, multi-URL batch processing, sitemap-based crawling, and menu-driven navigation extraction, with features like rate limiting, concurrent request handling, and robots.txt compliance. The implementation is particularly valuable for users who need to ingest documentation into AI systems while respecting site access policies and maintaining clean markdown output.
Available Actions(4)
Single URL Crawler
Extracts content from a single documentation page and outputs it in clean Markdown format. Requires the target documentation URL as a parameter.
Multi URL Crawler
Processes multiple URLs in parallel, generating individual Markdown files per page. Requires a file containing URLs (.txt or .json) as the first parameter, with an optional custom prefix for the output file.
Sitemap Crawler
Automatically discovers and crawls sitemap.xml, creating Markdown files for each page. The maximum recursion depth and URL patterns can be specified as optional parameters.
Menu Crawler
Extracts all menu links from documentation and outputs them in structured JSON format. Can handle nested and dynamic menus, with optional custom menu selectors.
Community Reviews
No reviews yet. Be the first to review!
Sign in to join the conversation
