Temperature parameters: balancing the determinism and creativity of AI output

The "dual personality" of the large model

In daily model use, have you ever wondered why the big model gives different answers to the same question? For example, when asked "How is the sunrise at the seaside?", it may say "The sun slowly sinks into the sea level, the sky presents a gradient of orange and red, and the light gradually dims." It may also say "The sky is like an overturned palette, with gold, crimson and indigo splashed on the clouds and the sparkling sea, and finally even the shy sun quietly hides in the warm embrace of the sea." Behind this phenomenon, there is a little-known secret hidden - sampling technology. Although the big model should theoretically give the same answer to the same input like a precision calculator, in fact they are designed to be "intentionally uncertain."

Large model prediction probability map

This uncertainty is not a defect, but the key to the power of the big model. The model does not always choose the "safest" and most common answer, but is given the ability to choose among multiple possibilities. This seemingly illogical design is precisely the source of the creativity of the big model and the core of its ability to simulate the natural and changeable conversations of humans. This allows the model to switch freely between rigor and creativity - sometimes like a meticulous scholar, sometimes like a talented poet.

So what exactly are these two distinct output modes—precision versus creativity—like? Before we dive into the sampling techniques that make this switch possible , let’s walk through a few examples to gain an intuitive understanding of the difference between the two styles.

Precision and creativity

When generating responses, the large language model can adjust between two main tendencies: precision and creativity . Precision refers to the focus on factual accuracy, clear logic, and direct expression. It is suitable for scenarios that require obtaining reliable information or completing specific tasks. Creativity focuses on novelty, imagination, diversity of expression, and stylization. It is suitable for scenarios that require inspiration, exploring different possibilities, or engaging in literary and artistic creation. The same question may give completely different answers depending on which characteristics we want the model to focus on.

When the large model was asked to describe the sun , the precision answer was  "The sun is a star located at the center of the solar system, a G-type main sequence star (yellow dwarf). It is mainly composed of hydrogen (about 74%) and helium (about 24%), and generates energy through nuclear fusion reactions in the core, which is radiated in the form of light and heat. Its diameter is about 1.39 million kilometers, which is 109 times the diameter of the earth." The creative answer was  "The sun is the golden furnace in the sky and the king of the day. It generously spreads its light to the earth, awakens sleeping creatures, and paints the world with warm colors. It is a messenger of hope at sunrise and leaves a splendid farewell at sunset."

When the large model was asked to describe a cat , the precision answer was:  "Cats (scientific name: Felis catus) are small carnivorous mammals belonging to the family Felidae. They typically have soft fur, retractable claws, keen hearing and night vision. Domestic cats have been domesticated by humans for thousands of years and are common pets." Creative answer:  "Cats are elegant and independent elves, sometimes curled up lazily in the sun to take a nap, and sometimes patrolling their territory like a curious explorer. They communicate with the world with mysterious eyes and soft purrs, and their gait is light, as if they are stepping on a silent melody."

Since the model can generate such different styles of answers, how can we guide it to be more accurate or creative in a specific scenario? This is due to its internal "decision-making" mechanism, which is the sampling technology we are about to introduce.

Sampling mechanism

"Sampling" is the key technology to achieve this control. It determines how the model selects from many possible next words when generating text.

There are three main sampling techniques:

Temperature sampling : Control the randomness of the results by adjusting the "steepness" of the probability distribution

Top-k sampling : limit the selection range and only extract from the k candidate words with the highest probability

Top-p sampling (core sampling): dynamically set the threshold and select only from the candidate set with cumulative probability reaching p

Unlike looking up a definite answer from a dictionary, sampling is more like a carefully designed probability game. The temperature parameter is like a slider to adjust the AI's adventurous spirit, directly affecting the model's "expressive personality".

These three technologies can be used individually or in combination to form a more sophisticated control mechanism. Next, let’s take a closer look at the most basic and intuitive parameter, temperature.

temperature

Temperature is a number that adjusts the concentration of probability distribution. The lower the temperature (e.g. 0.1), the probability will produce a more "sharp" distribution, tending to choose the option with the highest score, and the result will be more certain. The higher the temperature (e.g. 2.0), the smoother the distribution will be, and the chances of each option being selected will be more balanced, resulting in more diverse results. This flexible adjustment mechanism allows us to control the certainty or randomness of the model output according to the needs of different scenarios.

Temperature parameter effect diagram

As shown in the figure above, when the model receives the input prompt "Today's weather", the big model first generates a set of raw prediction probabilities: the probability of answering "sunny" is 0.55, the probability of "very good" is 0.25, the probability of "suitable for going out" is 0.15, and the probability of "wearing sunscreen when going out" is 0.05. This set of values ​​represents the original output distribution of the model under the standard temperature setting (usually 1.0).

The chart below shows the effect of adjusting the temperature parameter on this probability distribution. When we lower the temperature (left side of the chart), the probability distribution changes significantly: the probability of "sunny", which originally had the highest probability, increases significantly from 0.55 to 0.80, while the probabilities of other options decrease accordingly, "very good" drops to 0.10, "suitable for going out" drops to 0.07, and "wear sunscreen when going out" drops to 0.03. This low temperature effect essentially amplifies the probability differences in the original distribution, making the high-probability options more dominant, resulting in more certain and predictable output results.

On the contrary, when we increase the temperature (right side of the chart), we observe that the probability distribution tends to be uniform: the probability of the originally dominant "sunny" drops significantly from 0.55 to 0.28, while the options with lower probabilities are improved, "very good" rises to 0.26, "suitable for going out" rises to 0.24, and the most significant is that the originally lowest probability "wear sunscreen when going out" rises sharply from 0.05 to 0.22. This high temperature effect actually compresses the probability differences between different options, creates a more balanced selection environment, and increases the uncertainty and diversity of the model output.

The role of temperature

This adjustment of temperature is of great value in practical applications. When users need the model to provide accurate, consistent and reliable answers (such as factual queries, instruction execution or code generation), a lower temperature setting can ensure that the model tends to choose the most confident answer . When users expect creative and diverse content (such as creative writing, story generation or brainstorming), a higher temperature setting can encourage the model to explore a wider range of possibilities and produce richer and more varied outputs .

Temperature essentially provides a simple and powerful control mechanism for large models, allowing users to find the best balance between the two competing characteristics of determinism and creativity according to specific needs. By adjusting the temperature appropriately, the same model can be flexibly adapted to a variety of application scenarios from strictly following instructions to freely expressing creativity, greatly enhancing the practicality and adaptability of generative models.

Summarize

Temperature is an important mechanism for controlling the output of large models. It balances the certainty and creativity of the model by adjusting the "sharpness" of the probability distribution. A low temperature (such as 0.1) will produce a more concentrated distribution, which will make the model tend to choose the words with the highest probability and the output more certain; a high temperature (such as 2.0) will smooth the distribution, making each option more equal and producing more diverse results. This simple and powerful mechanism allows us to flexibly adjust the behavior of the model according to different needs, whether it is a factual query that requires an accurate answer or content generation that pursues creativity, so as to obtain the output that best suits a specific scenario.