DeepSearch: The future of AI search is more than just fast

Did you know that the average AI practitioner spends several hours a day on information retrieval?

Inefficiency in search is becoming a stumbling block to AI innovation. Imagine if your AI interns could help you research and compile high-quality research reports like experts, and independently judge the quality of the reports. Improving efficiency by 10 times is no longer a dream! Today, let's unveil Jina AI DeepSearch and see how it breaks the search bottleneck and enables AI report generation!

Note: This article was generated by me based on the notes of Dr. Xiao's live speech at JINA AI + 2 prompts, without any other workflow.

Traditional search: fast, but not deep enough

Traditional search relies on keyword matching. Although the response speed is fast (usually within 200 milliseconds[^1]), the results often lack depth and cannot meet complex information needs. Just like the hot RAG (Retrieval-Augmented Generation) in the first half of 2024, although it was popular for a while, its quality was ultimately mediocre and the improvement was limited. This is like fast food, which is convenient but lacks nutrition.

DeepSearch: A search paradigm in the AI ​​era, with both depth and insight

DeepSearch is a new search paradigm that simulates human experts in research and analysis through deep reasoning . Compared with traditional search, DeepSearch has the following advantages:

• Delay gratification in exchange for high-quality reports:
  Instead of pursuing millisecond response speeds, AI Agents are allowed to spend minutes conducting in-depth analysis and ultimately provide high-quality reports. This is like a slow-cooked soup that takes time to cook, but tastes richer.
• Structured output, ready to use:
  Users can obtain ready-to-use structured reports without having to summarize and process them themselves.
• Improve user stickiness:
  High-quality search results can increase user stickiness, making users willing to wait and continue using the product.

In simple terms, DeepSearch is like hiring an AI intern to help you research and compile reports. Although it takes some time, the final results are far beyond your expectations.

The core steps of DeepSearch

The core of DeepSearch is to empower AI , allowing AI Agents to search, read, and reason autonomously, and ultimately generate high-quality reports. The key steps of DeepSearch include: planning, searching, generating, evaluating, and iterating.

The following table summarizes the key steps, technical details, and evaluation metrics of DeepSearch:

How does Jina AI delegate power to AI?

The core of Jina AI DeepSearch is to delegate power to AI , allowing AI Agents to search, read, and reason autonomously, and ultimately generate high-quality reports. Specifically, Jina AI adopts the following strategies:

1. Iterative search:
   The AI ​​Agent continuously cycles through searching, reading, and reasoning until it finds the best answer, a process that mimics the thinking process of a human researcher, constantly digging deeper into information.
2. Sub-problem breakdown:
   Break down complex problems into multiple sub-problems and solve them one by one. For example, when conducting market research, you can break down the problem into sub-problems such as "Who are the target users?", "Who are the competitors?", "How big is the market?", etc.
3. Query extension:
   Expand the query entered by the user to explore the user's potential needs. For example, if the user searches for "second-hand price of BMW 5 Series", DeepSearch will not only search for second-hand prices, but also expand to related information such as "BMW 5 Series advantages and disadvantages" and "comparison of models of the same level". This query expansion is based on the deep understanding of user intent by the big model, not just keyword matching.
4. Long text processing:
   Use sliding window and other technologies to extract key information from long documents. Sliding window technology ensures the continuity of information and avoids information fragmentation.
5. URL sorting:
   Sort the URLs in the search results and give priority to high-quality web pages. Jina AI sorts URLs by taking into account factors such as the frequency of URL appearance, the time of last update, and relevance.

Through these strategies, Jina AI gives AI Agents greater autonomy, enabling them to conduct research and analysis like human experts and ultimately generate better reports.

The role of LLM in DeepSearch

LLM plays a key role in multiple stages of DeepSearch:

• Query extension
  : LLM is used to understand user intent and generate relevant and diverse queries, thereby expanding the search scope.
• Answer evaluation
  ： LLM is used to determine whether the generated answers meet the requirements, for example, whether they are accurate, complete, relevant, etc. LLM can classify the answers, generate evaluation reports, etc.

Case: Application of DeepSearch in market research

Suppose you need to conduct a research report on the "AI chip market". Using DeepSearch, you can:

1. Enter the keyword "AI chip market research".
2. DeepSearch will automatically expand queries, such as "AI chip market size", "AI chip competition landscape", "AI chip development trends", etc.
3. DeepSearch will crawl information from multiple channels (including news, blogs, research reports, etc.) and organize it in a structured manner.
4. In the end, you will get a detailed report covering market size, competition landscape, technology trends, investment opportunities, and more.

DeepSearch saves a lot of time and effort and provides deeper insights compared to traditional search.

Comparison of Competitive Products: Jina AI DeepSearch vs. Zhipu AI AutoGLM Shensi vs. OpenAI Deep Research

At present, some similar AI Agent products have emerged on the market, such as Zhipu AI's AutoGLM and OpenAI's Deep Research. They all aim to improve the efficiency of information acquisition by automating the research process through AI.

• Jina AI DeepSearch:
  Jina AI DeepSearch emphasizes flexibility and customizability. For example, users can adjust parameters to control search depth and breadth, or customize URL sorting rules to meet specific needs.
• Zhipu AI AutoGLM ponders:
  AutoGLM is an AI agent product launched by Zhipu AI. It integrates deep research capabilities and operational capabilities, and can perform complex thinking while executing operations. AutoGLM emphasizes the operational capabilities and flexibility of the intelligent agent, and improves the operational capabilities and flexibility of the intelligent agent by decoupling task planning and action execution[^4].
• OpenAI Deep Research:
  OpenAI Deep Research is an AI agent based on GPT-4 that is capable of multi-step autonomous research, deep information integration, and complex task processing. Deep Research emphasizes multi-step reasoning and information synthesis. OpenAI Deep Research is currently only available to ChatGPT Pro users and is priced relatively high[^5].

Limitations and Challenges

Although Jina AI DeepSearch is powerful, it also has some limitations:

• Data source bias:
  The results of DeepSearch depend on the search engine and web page content, and there may be bias in the data source. For example, if the search engine has a preference for ranking certain websites, or the content quality of certain websites is not high, then the results of DeepSearch will also be affected. To solve this problem, Jina AI is actively exploring methods for multi-source data fusion and developing bias detection and correction algorithms. In addition, if in a private environment, JINA AI's products can generate more valuable reports based on the highly refined database within the enterprise.
• Computing resource requirements:
  DeepSearch requires a lot of computing resources and may require certain hardware configurations. For example, running DeepSearch may require a server equipped with a GPU, or using cloud services such as AWS or Azure. Jina AI is working hard to optimize the algorithm and reduce computing resource requirements.

Future Directions

• Personalization
  ： Provide personalized search results based on the user's interests and needs. For example, more relevant results can be recommended to users based on the user's historical search records, browsing behavior and other information.
• Multimodality
  ：Support multimodal input, such as images, audio, video, etc. This will enable DeepSearch to handle more complex information needs, for example, users can search for related products or services by uploading a picture. However, multimodal input also brings challenges in data fusion and alignment.
• Explainability
  : Improve the interpretability of DeepSearch to make it easier for users to understand the report generation process. For example, you can show users the reasoning process of DeepSearch or explain why a certain result is considered relevant.

Jina AI DeepSearch represents a new direction for search in the AI ​​era. It abandons the traditional thinking chain and delegates power to AI, using simpler methods to achieve deeper information mining and higher-quality report generation. Although it is still in its early stages of development, the potential of DeepSearch has already emerged. With the rapid improvement of AI memory, the decline in AI computing costs, and the construction of private databases for enterprises, we are expected to see AGI realized in the consulting field at a speed beyond our expectations.