Visual RAG models are here! From text to images, how does AI "understand" the world?

What is RAG?

RAG (Retrieval-Augmented Generation) is an important breakthrough in the field of artificial intelligence in recent years. While traditional language models rely on pre-trained data to generate text, RAG augments generation by retrieving external information sources. Simply put, it finds documents or data from external databases that are relevant to a question, and then combines this information to generate a more accurate, timely, and contextualized answer.

For example, if you ask a traditional language model, "What's the weather like today?" It can only give a generic answer based on pre-trained data. But with a RAG model, it can retrieve the latest data from real-time weather sites and give you an accurate weather forecast for your area. This ability makes RAG perform more intelligently and reliably when dealing with complex problems.

Vision RAG: Letting AI "See" the World

Vision RAG is an extension of the RAG model, which incorporates visual data (such as images, charts, videos, etc.) into the processing scope. Unlike the traditional RAG model, which mainly deals with text, Vision RAG utilizes visual language models (VLMs) to index, retrieve, and process visual information. This means that it can process complex documents containing both text and visual content, such as PDF files.

The core strength of Vision RAG is its ability to generate responses that are not only textually correct but also visually rich and accurate. For example, you can upload a scientific report that contains both charts and text and ask "What does this chart say?" Vision RAG will not only understand the content of the chart, but will also give a complete explanation in conjunction with the textual information.

Features of Vision RAG

Vision RAG makes AI smarter and more efficient when dealing with multimodal data. Here are some of its main features:

1. Multimodal Retrieval and Generation

Vision RAG is able to process both textual and visual information in a document. This means that it can answer questions about images, tables, etc., not just text. For example, you can ask, "What style of building is in this picture?" It will give you an answer that combines the image with the textual information in the document.

2. Direct Visual Embedding

Unlike traditional OCR (Optical Character Recognition) or manual parsing, Vision RAG uses a visual language model to embed visual information directly. This approach preserves semantic relationships and context, making retrieval and understanding more accurate.

3. Unified cross-modal search

Vision RAG enables semantically meaningful search and retrieval in a single vector space, covering mixed-modal content. Whether you are asking about text or images in a document, it can find the answer within a unified framework.

These features enable Vision RAG to support more natural and flexible interactions. Users can ask questions in natural language, and the model will extract answers from text and visual sources to provide more comprehensive information.

How to use Vision RAG?

To integrate Vision RAG functionality into our work, we can use a model called localGPT-vision. LocalGPT-vision is a powerful, end-to-end Vision RAG system that processes visual document data (such as scanned PDFs or images) directly, without relying on OCR.

Currently, localGPT-vision supports the following visual language models:

• Qwen2-VL-7B-Instruct
• LLAMA-3.2-11B-Vision
• Pixtral-12B-2409
• Molmo-&B-O-0924
• Google Gemini
• OpenAI GPT-4o
• LLAMA-32 with Ollama

localGPT-Vision Architecture

The system architecture of localGPT-Vision consists of two main components:

1. Visual document retrieval

Colqwen and ColPali are visual coders specifically designed for understanding the image representation of documents. During the indexing process, document pages are converted into image embeddings and user questions are embedded and matched against the indexed page embeddings. This approach allows retrieval to be based not only on text, but also on visual layouts, diagrams, and other content.

2. Response Generation

The pages with the best match to the document are submitted as images to the Visual Language Model (VLM), which generates contextually relevant responses by decoding the visual and textual signals.

Note that the quality of the response depends heavily on the VLM used and the resolution of the document image.

This design eliminates the need for a complex text extraction process and understands the document directly from a visual perspective, without the need to select an embedding model or retrieval strategy as in traditional RAG systems.

Features of localGPT-Vision

• Interactive chat interface: Users can upload documents and ask questions through the chat interface.
• End-to-end visual RAG: Completely vision-based retrieval and generation without OCR.
• Document uploading and indexing: Supports uploading PDFs and images, indexing via ColPali.
• Persistent Indexing: All indexes are stored locally and loaded automatically after reboot.
• Model Selection: You can select a variety of VLMs, such as GPT-4, Gemini, and so on.
• Session management: you can create, rename, switch, and delete chat sessions.

Practical Operation of localGPT-Vision

Let's see how localGPT-Vision works with a simple example.

In the video below, you can see the model in action. On the left side of the screen is a settings panel where you can select the VLM model to be used for processing PDFs. After selecting the model, upload the PDF file, and the system will start indexing. After the indexing is complete, you just need to enter a question about the PDF, and the model will generate a correct and relevant answer based on the content.

Since this setup requires a GPU for optimal performance, I've shared a Google Colab notebook that contains the entire model implementation. All you need is a model API key (e.g., Gemini, OpenAI, or other) and an Ngrok key to deploy the application publicly.

VI. Application Scenarios for Vision RAG

The emergence of Vision RAG opens up new possibilities in many areas. The following are some typical application scenarios:

1. Medical Imaging

Vision RAG can combine medical images and medical records to help doctors make smarter and more accurate diagnoses. For example, it can analyze textual information in X-rays and medical records to provide more comprehensive diagnostic recommendations.

2. Document Search

Vision RAG is able to extract information from documents containing text and visual content to generate summaries. This is very useful for researchers and professionals who can quickly find the key information they need.

3. Customer Support

Vision RAG can solve problems with photos uploaded by the user. For example, a customer can upload a photo of a malfunctioning piece of equipment, and the model provides a solution in combination with a textual description.

4. Education

Vision RAG can help teachers and students better understand complex concepts. It can provide students with a personalized learning experience through a combination of diagrams and text.

5. E-Commerce

Vision RAG can generate more accurate product recommendations based on product images and descriptions. For example, if a user uploads a picture of a favorite garment, the model can recommend products of a similar style.

Summary

Vision RAG is an important advance in the field of artificial intelligence, which allows AI to not only "read" text, but also "understand" images and charts. With the wide application of the Vision RAG model, we can expect smarter, faster, and more accurate solutions. It has huge potential not only in education and healthcare but also in many other areas, unlocking new possibilities for innovation and insight.

AI is now beginning to understand and perceive the world in the same way that humans do. With Vision RAG, we can look forward to the future of AI. If you're interested in Vision RAG, try localGPT-vision and experience multimodal AI for yourself!