Doctor is a comprehensive tool designed to discover, crawl, and index websites, presenting the data through an MCP server for LLM agents. It integrates various technologies for crawling, text chunking, embedding creation, and efficient data storage, along with a user-friendly FastAPI interface for search and navigation. The system is built with Docker support and offers hierarchical site navigation and automatic title extraction for crawled pages.
web-crawling ✓
llm ✓
fastapi ✓
+ docker
indexing ✓