Web scraping server that fetches and processes web content with robust features like robots.txt compliance, LRU caching, and markdown conversion, supporting both autonomous and manual content retrieval modes.