Programmers rejoice! DevDocs: an open source AI document tool that automatically captures, organizes, and feeds documents to AI (with MCP support)

Written by

Iris Vance

Updated on:June-26th-2025

Introduction:

Are you still going crazy over the massive amount of technical documents? Do you want AI to help you find answers directly from official documents? Today, YouywayAI will introduce you to an open source tool specially designed for developers - DevDocs ! It can automatically crawl technical documents from specified websites , organize them into clear structured data, and through the built-in MCP server , you can directly query them in AI tools such as Cursor and Claude Desktop ! Say goodbye to information overload and let AI become your programming partner who understands documents best!

text:

1. DevDocs: Your Intelligent Document Processing Expert

Simply put, DevDocs by CyberAGI is a tool that can convert online technical documents into an AI-usable knowledge base . Its core process is:

• Smart crawling: specify a starting URL for technical documentation.
• Automatic discovery and organization: DevDocs crawls relevant pages like a spider (you can control the depth), extracts core content, removes irrelevant elements, and builds the structure of the website.
• Structured output: Organize the captured content into Markdown or JSON format for easy reading or for LLM fine-tuning.
• MCP Service: Built-in MCP server

From now on, AI can directly query the latest official document content you specify!

2. Core Highlights: Why is DevDocs different?

• Depth control: Configurable 1-5 levels of grab depth.
• Automatic link discovery: Intelligently discover and categorize related content pages.
• Selective crawling: You can specify to crawl only specific parts.
• Sub-URL Detection: Automatically understand and map website structure.

• ⚡ High performance and speed:

• Parallel processing: Crawl multiple pages at the same time at a very fast speed.
• Smart caching: Avoid repeatedly crawling the same content, saving time and resources.
• Support lazy loading: can handle dynamically loaded content in modern web applications.
• Rate Limiting: Respect the target website and avoid request overload.

• ? Content processing and output:

• Clean extraction: remove irrelevant content such as navigation bars and advertisements, and only retain core information.
• Multiple formats: support export to Markdown (MD) and JSON .
• Structured output: The content is logically organized and easy to understand.
• MCP server integration: Core advantage! Turn captured documents directly into AI-queryable MCP tool sources.

• ?️ Enterprise-level features (some may be in the future or paid version): error recovery, full logs, API access, team management, etc.
• ✅ Open source and free & self-hosted: The core functions are free to use, the code is open source, and can be fully deployed on your own server.

3. Solve the pain point: Why do you need DevDocs?

• Problem: Technical documents are scattered and updated quickly, and LLM’s knowledge base is often outdated. Developers need to spend a lot of time reading, understanding, and finding information.
• DevDocs Solution:

1. Point to any technical documentation URL.
2. DevDocs automatically discovers all related pages.
3. Extract clean, useful content.
4. With the built-in MCP server, your AI tools can directly query these latest and most accurate documents!
5. (Optional) Export MD/JSON for local reference or model fine-tuning.

Goal: To enable anyone to quickly build great products using the latest technology!

4. Get started quickly (Docker recommended)

The official recommendation is to use Docker deployment, which is the simplest and most convenient:

1. Preparation: Make sure you have Docker and Git installed.

2. Clone the repository:

git  clone  https://github.com/cyberagiinc/DevDocs.git
cd  DevDocs

3. Configure environment variables:

• copy .env.template for .env: cp .env.template .env
• Important: Make sure .env In the file NEXT_PUBLIC_BACKEND_URL Set up correctly (usually http://localhost:24125) so that the front-end UI can connect to the back-end service.

4. Start the service (Mac/Linux):
```
./docker-start.sh
```
(Windows users please refer to the instructions in the official README, which may still be experimental support)
5. Visit:

• Front-end UI: http://localhost:3001 (Enter URL here to start crawling)
• Backend API: http://localhost:24125
• Crawl service (Crawl4AI): http://localhost:11235
• MCP Server: Once started, you can find the MCP URL corresponding to a specific document library in the UI (usually related to the Backend API).

5. How to integrate into AI tools? (Take Claude Desktop as an example)

1. Use DevDocs UI to capture and generate the technical documentation library you need.
2. Find the MCP server URL for the document library in the DevDocs UI (for example http://localhost:24125/mcp/{crawl_id}).
claude_desktop_config.json
).

4. Add a new MCP server configuration (requires npx and mcp-proxy or mcp-remote):

{
  "mcpServers" : {
    "my_devdocs_library" : { // Custom name
      "command" : "npx" ,
      "args" : [ "mcp-proxy" , "yourDevDocs_MCP_URL" ] // Replace with the actual URL
      // Or use mcp-remote:
      // "command": "npx",
      // "args": ["mcp-remote", "your DevDocs_MCP_URL"]
    }
    // ... other MCP servers...
}
}

(For tools like Cursor, Windsurf, Roo Code, etc., please refer to their respective methods for adding MCP servers, usually by entering the SSE URL)

5. Now, your AI assistant can query this document library directly!

6. Open Source and Resources

• GitHub repository (Star ⭐): https://github.com/cyberagiinc/DevDocs

Summarize:

DevDocs is an innovative open source project that cleverly uses intelligent crawlers and the MCP protocol to turn static technical documents into a dynamic source of knowledge that can be directly queried by AI . This is undoubtedly a great boon for developers who need to frequently consult a large amount of rapidly updated technical documents. It not only saves a lot of research time, but also ensures that AI obtains the latest and most accurate information. If you want AI to become your more powerful programming partner, DevDocs is definitely worth deploying and using!