The Nature and Limitations of Context Windows

Memory mechanism of large models

Imagine that you are chatting with a friend with limited memory. When your conversation lasts for several hours, your friend can only remember the most recent and first few sentences, while a large amount of the content discussed in the middle is blurred or even forgotten.

This is how large language models work today, and the first few sentences are usually system instructions, and the most recent few sentences are the new content you send to the large model.

The so-called "context window" is essentially the amount of text that LLMs can "remember" and process, usually measured in "tokens". For English, a token is about 4 characters or 1 word; for Chinese, it is about one Chinese character. This window is like the model's short-term working memory, which determines how much previous information it can review and use at any moment.

Context Window Wars

In recent years, major model providers have been engaged in advertising wars on the "context window size", from GPT-4's 128K, Claude 3's 200K, to Google Gemini's claimed 1M or even 10M.

It’s an open secret here, too: there’s a significant gap between the nominal context length and the context that the model can effectively exploit.

Perplexity

For a long time, Perplexity was considered the main indicator for determining the upper limit of length extrapolation , but subsequent studies found that perplexity cannot truly reflect the actual performance of large language models in long-context downstream tasks. Downstream tasks may include question answering (QA), summaries (Summ.), information retrieval (Retrieval), etc. A model may perform well in perplexity = it can better predict the next word in a long text, but this does not mean that it can effectively understand the key information in the long text, answer complex questions, or generate high-quality long text summaries.

Previously, many people believed that "perplexity" was the best indicator for evaluating the ability of a language model to process long texts. Low perplexity means that the model is good at predicting the next word in a long text. But now research has found that this indicator has limitations.

In simple terms: Good prediction of the next word ≠ true understanding of long text

In real-world applications, a model may be good at continuing writing, but poor at answering questions, extracting key information, or summarizing long texts.

It's like the difference between memorizing and understanding. A person can memorize a text very well (predict the next word) but may not truly understand the content of the text (answer questions, make summaries, etc.).

Lost in the Middle

Lost in the Middle: How Language Models Use Long Contexts ^[1] mentioned:

In principle, large language models with global attention can theoretically exploit the entire conversation history or document context without forgetting. However, in practice, LLMs exhibit strong position bias, favoring certain parts of the input and ignoring others.

Empirical studies have shown that LLM sometimes pays more attention to the most recent token and sometimes to the beginning of the context, and less attention to the middle part, which is also called primacy bias and recency bias.

It mentions a U-shaped performance curve: the model answers best when the relevant information is at the beginning or end of a long input, but performance drops sharply if the desired information is buried in the middle.

Later, there was also  ∞ Bench: Extending Long Context Evaluation Beyond 100K Tokens ^[2]
 which said that it did not find that the accuracy of LLMs would decrease when the answer was located at the center of the context:

Of course, the conclusion of this paper is still that "although current LLMs claim to be good at handling such a wide range of contexts, in practical applications they show significant performance degradation"

Fiction.LiveBench

There is also  Fiction.LiveBench ^[3] , which is more famous because they have an internal evaluation mechanism and can test the latest models in a timely manner.

In simple terms, Fiction.liveBench is like an exam that tests AI's reading comprehension ability, focusing on whether AI can truly understand the content of a long story, remember all the important characters and events, and answer questions that require deep thinking. Currently, only OpenAI o3 has achieved 100 points in most context windows in their tests.

Context Management Methods

In real-world applications, when AI assistants "forget" key requirements or constraints previously mentioned by users, user experience and trust will be greatly reduced.

For example, a customer service chatbot that forgets key information explicitly stated by the user during a long conversation, or a legal document analysis tool that omits a key clause in the middle of a contract, can lead to very bad user experience problems.

In the product design methodology of context management, there are two most important directions: Full Context and RAG (retrieval enhancement).

The full-context approach is relatively straightforward: cram all the history into the model’s context window and let the model decide for itself which information is important.

This is like giving a student a full textbook instead of condensed notes. The benefit of this approach is that no potentially relevant information is lost, and in theory the best answer quality can be obtained.

However, it comes at a significant cost: long processing time, high API call cost, and complete failure when the conversation history exceeds the context window.

RAG is like providing students with targeted notes and reference materials rather than entire textbooks; according to my personal definition, RAG is mainly about finding relatively short but most relevant content in various ways, and only these contents will be stuffed into the context window of the model - retrieving appropriate content and using as little context as possible .

It has multiple implementation forms: text segmentation cuts long text into small segments and retrieves only the most relevant segments each time; summarization compresses large amounts of information into refined summaries; sliding windows retain the most recent part of the content and discard earlier parts; and there is also reordering technology that adjusts the position of the text in the context and places important content in a position that is not easily "forgotten".

In actual applications, RAG is not a single technology, but a general term for a series of methods. The effects of different RAG solutions may vary greatly. For example, the method of simply retaining the latest n messages and the method of intelligently extracting key facts and integrating them into structured memory will have significant differences in effect.

Each of these two methods has its own applicable scenarios: the full context method is suitable for situations where accuracy is extremely high and the conversation length is limited; while the RAG method is more suitable for scenarios that need to handle extremely long conversations or are sensitive to response speed and cost.

In actual product design, choosing which method to use is not an either-or decision, but a trade-off based on specific needs.

Mem0 case study

In April 2025, Mem0 released Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory ^[4], comparing the full-context approach with their proposed memory management system.

They created a system that dynamically extracts, integrates, and retrieves important information from a conversation. In simple terms, this system automatically extracts important facts and information (such as preferences, experiences, etc. you mentioned) when you chat with AI, stores this information, and retrieves relevant information when needed:

• •  First look at the background :
• • View the last ten messages ("Where were we?")
• • Review chat summaries up to the last ten messages (it does not remember messages within a certain number, only summaries)
• •  Then extract the key points : AI will think: "What information is worth remembering in this new conversation?" For example, if you say "I like Italian food, but I am allergic to dairy products", AI will extract these two points as important information
• •  Finally organize your notes (memory) :
• • Check for conflicts with existing notes (if you previously said you liked cheese and now say you’re allergic to dairy, the system will update that information)
• • Decide whether to add a new note, update an old note, or delete an incorrect message

Their results show that in terms of accuracy, the full-context method is slightly better, achieving a LLM-as-a-Judge score of about 73%, while the Mem0 system achieves about 67-68%.

Of course, this does not mean that Mem0's RAG is useless. Its biggest advantages are response time and cost. The response time is reduced by 85-91%, and token consumption is also greatly reduced.

More interestingly, the performance of the two methods varies on different types of problems. For problems that require temporal reasoning (e.g., “How is the progress of the project we discussed last month?”), the graph-enhanced version of Mem0 even outperforms the full-context method.

However, the model used in the above-mentioned Mem0 paper is only GPT 4o mini, and according to OpenAI’s own article, the newly released GPT 4.1 mini has comprehensively surpassed most previous models in long-context evaluations.

But I think this study also brings some insights: there is no one-size-fits-all approach to context management. The choice of full context or various systems based on the RAG concept should be based on specific application scenarios, user needs and resource constraints.

In some scenarios where accuracy is extremely important, it may be reasonable to sacrifice some speed and cost in exchange for the highest accuracy (full context); in large-scale deployments, when users are sensitive to response speed or when resources are limited, choosing various systems based on the RAG concept (such as memory systems like Mem0) may be a wiser decision.

Context management strategy for user decision making

The above papers provide us with a certain "factual basis", but when it comes to product decisions, more detailed strategies are needed. In fact, the choice of context management strategy should be based on multi-dimensional considerations rather than simply choosing between two options.

For products in high-risk areas for professional users, such as legal document analysis, medical diagnosis assistance, or financial advice, accuracy is usually the primary consideration. In these scenarios, even at the cost of higher computational costs and slightly longer response times, the full-context approach may still be a better choice, and it can also be combined with RAG-based and Multi-Agent-based repeated checks. After all, in these fields, a wrong suggestion or missing key information can lead to serious consequences.

On the contrary, for products targeting C-end users, applications that need to handle a large number of concurrent requests, mobile applications, or scenarios that are particularly sensitive to cost, the advantages of methods such as RAG are more prominent. Imagine a customer service chatbot that needs to serve thousands of users at the same time. Response speed and computing cost may be more important than extreme accuracy (of course, there is a threshold of availability, and you must cross that threshold of availability before you can further consider speed and computing cost).

In actual products, a hybrid strategy is often the most practical. For example, full context is used in the initial stage for quick response. As the conversation deepens, complexity increases, and context accumulates, you can consider switching to lightweight RAG. However, for some key functions that require special effects, you can enable full context mode to ensure the highest accuracy.

Take Deep Research as an example. Imagine how you work when using Deep Research to perform a complex market research task. It should not be a simple read of all the materials at once, but like a professional researcher, make a plan first, and then adopt different information processing strategies at different stages:

Research plan and core orientation : During the process, the research plan, the core needs of users, and the key facts that have been confirmed must be kept in context. This information is the "North Star" of the research and needs to be kept clearly visible throughout the process to ensure that the research does not deviate from the direction. This part adopts a strategy close to the full context.

Detailed information and supporting materials : For a large number of minor details, background materials and supporting evidence, the RAG method is used to retrieve them on demand. Rather than always keeping all the information in context, this "take it when you need it, put it away when you need it" approach can improve efficiency.

Cross-session memory management : When research lasts for tens of minutes or even longer, the processed information is intelligently compressed to extract key conclusions and discard original details, retaining the essence rather than all. These compressed memories are reactivated in subsequent sessions, allowing research to proceed coherently.

The cleverness of this hybrid approach is that it balances the global perspective with detailed exploration. The core arguments and research directions are always kept in the "consciousness" of the system, while the vast amount of details are organized into an easily retrievable form and called up when needed.

It is crucial for product designers to understand the value of this hybrid strategy. It not only improves efficiency and accuracy, but also greatly reduces computational costs. If Deep Research relies entirely on the full-context approach, it will not be able to handle research tasks beyond a certain scale; if it relies entirely on RAG, the coherence and depth of the research may be lost.

By intelligently deciding what should always be remembered and what can be queried temporarily, we are able to handle complex research tasks that extend far beyond their context window within limited computing resources while maintaining research quality and coherence.

This is why, when building complex AI products like Deep Research, a mixed-context strategy is not a choice but a necessity. The core is still the control of user scenarios and the trade-off between computing resources, costs and effects.

When assessing your needs, consider these key questions:

• • How sensitive are users to response speed?
• • How important is accuracy to the product?
• • Do the product’s usage scenarios involve very long conversations or documents?
• • What are the computing resource and cost constraints?
• • What are user expectations?

The answers to these questions will guide you in choosing the most appropriate context management strategy.

Some common techniques for dealing with context constraints

The most important thing is task decomposition design. Complex dialogue tasks should be designed as manageable chains of subtasks rather than a single large task. Each subtask can have clear input, output, and context requirements, so that the system can more effectively manage the context required for each link. For example, a travel planning assistant can decompose tasks into subtasks such as destination selection, accommodation search, and activity arrangement. Each subtask focuses on its own context requirements rather than trying to manage all information in a single dialogue. Building effective agents ^[5]  lists different workflow designs.

At the same time, during task execution, it is best to make the status as explicit as possible to make it clearer to the user. In long-term workflows, key status (such as user goals, constraints, and completed steps) can be explicitly recorded and displayed to the user for confirmation when necessary, rather than relying solely on AI's "memory". This not only enhances the reliability of the system, but also provides users with the opportunity to correct misunderstandings.

Finally, we can consider memory hierarchical management. It is generally believed that inspired by the human memory system, AI's memory can be divided into short-term memory and long-term memory. Short-term memory contains the most recent conversation content, which is directly placed in the context window; long-term memory is stored in an external database, including user preferences, key information in historical interactions, etc. The key is to establish an intelligent retrieval mechanism to recall relevant content in long-term memory to short-term memory when needed. For example, when a user mentions "the project we discussed last time", the system should be able to quickly identify and retrieve relevant historical information.

These techniques not only help solve the problem of contextual limitations at the technical level, but also significantly improve the user experience and make AI products more reliable and practical in long conversation scenarios. The key is to realize that context management is not only a technical issue, but also a product design issue that needs to be considered comprehensively from multiple dimensions.

Tail

When designing AI products, context management should be considered a core design consideration rather than a technical detail. Products should consider:

• • How long will our scenario produce a conversation or document?
• • What information do users expect AI to remember?
• • How does the accuracy vs. responsiveness tradeoff affect our product?

The choice between full-context and RAG is not black and white. The full-context approach may provide slightly higher accuracy, but at the expense of significantly increased latency and cost. For most products, some form of RAG is a practical choice, especially when the product needs to be deployed at scale or support long-term interactions. However, the specific implementation strategy needs to be customized according to the unique needs of the product, and it will also significantly increase development costs.

As technology develops rapidly, product strategies need to be flexible. New research and models are constantly being introduced, such as the GPT-4.1 series recently released by OpenAI, which has significant improvements in long context processing. Today's best practices may soon become outdated, so it is also crucial to establish a continuous monitoring mechanism for product performance and keep an eye on the latest research.

Ultimately, I believe that only by better understanding and balancing user needs and technical limitations can we build smarter, more natural, and more reliable product experiences.

As AI technology continues to advance rapidly, we can expect context management strategies to evolve as well, but no matter how the technology changes, user-centered design thinking and a deep understanding of the technology’s limitations will always be the cornerstones of building AI products.