Temperature & Top-P
0.0|0 ratingsLog in to rate
* **Temperature**: A hyperparameter that scales the logits (raw probability scores) output by the model, controlling the randomness of the generation. * **Top-P...
#ai-engineering#course#temperature
Definition
- Temperature: A hyperparameter that scales the logits (raw probability scores) output by the model, controlling the randomness of the generation.
- Top-P (Nucleus Sampling): A hyperparameter that limits generation choices to a subset of tokens whose cumulative probability exceeds the value .
Why It Exists
LLM generation is probabilistic. If we always select the token with the highest probability (greedy search), the model outputs repetitive, dry text. If we pick completely random tokens, the text becomes incoherent. Temperature and Top-P let you fine-tune this balance.
How It Works
python
1
2
3
4
5
6
7
8
9
10
11
12
13
LOGITS (Raw scores) ──► Softmax ──► Probability Distribution
│
┌──────────────────────────┴──────────────────────────┐
[Apply Temperature] [Apply Top-P]
Scales scores: Truncates choices:
- Low (0.2): Accentuates peak probabilities - High (0.95): Broad subset
- High (1.0): flattens distribution - Low (0.10): Narrow subset
1. **Logit Generation**: The model calculates raw scores (logits) for every word in the vocabulary.
2. **Temperature Scaling**: Logits are divided by the temperature ($T$).
* $T \to 0$: The probability of the highest-rated token approaches 1.0 (highly deterministic).
* $T \ge 1.0$: The probability distribution flattens, making less likely words easier to pick (highly creative).
3. **Top-P Truncation**: Tokens are sorted by probability, and the model only samples from the top tokens whose cumulative probability is $\le P$, filtering out low-probability nonsense.Real-World Example
Example
- For SQL query writing or JSON generation, set temperature to 0.0 to ensure structured, syntax-correct outputs.
- For creative writing or brainstorming, set temperature to 0.8+ to produce diverse, engaging text.
Python Example
Example
Here is a simulation of the mathematical steps behind Temperature and Top-P sampling:
python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
import math
import random
from typing import Dict, List, Tuple
def softmax_with_temp(logits: Dict[str, float], temp: float) -> Dict[str, float]:
"""Applies temperature scaling and softmax normalization to raw logits."""
# Prevent divide-by-zero for temp=0 by returning greedy choice
if temp <= 1e-5:
max_token = max(logits, key=logits.get)
return {k: (1.0 if k == max_token else 0.0) for k in logits}
scaled = {k: math.exp(v / temp) for k, v in logits.items()}
total = sum(scaled.values())
return {k: v / total for k, v in scaled.items()}
def sample_nucleus(probabilities: Dict[str, float], top_p: float) -> str:
"""Filters choices to cumulative probability top_p and samples a token."""
sorted_tokens = sorted(probabilities.items(), key=lambda x: x[1], reverse=True)
cumulative = 0.0
valid_tokens: List[Tuple[str, float]] = []
for token, prob in sorted_tokens:
valid_tokens.append((token, prob))
cumulative += prob
if cumulative >= top_p:
break
# Re-normalize subset probabilities
tokens, probs = zip(*valid_tokens)
total_prob = sum(probs)
normalized_probs = [p / total_prob for p in probs]
# Sample from the valid subset
return random.choices(tokens, weights=normalized_probs, k=1)[0]
if __name__ == "__main__":
# Raw logit scores output by LLM for the next word
mock_logits = {"code": 5.0, "python": 4.5, "banana": 0.5, "car": 0.2}
print("--- Greedy Generation (T=0) ---")
probs_greedy = softmax_with_temp(mock_logits, temp=0.0)
print("Probabilities:", probs_greedy)
print("Sampled:", sample_nucleus(probs_greedy, top_p=1.0))
print("\n--- Creative Generation (T=1.0, Top-P=0.90) ---")
probs_creative = softmax_with_temp(mock_logits, temp=1.0)
print("Probabilities:", probs_creative)
print("Sampled:", sample_nucleus(probs_creative, top_p=0.90))Interview Questions
- Beginner: What does a temperature of 0.0 accomplish?
- Intermediate: What is the difference between Top-K and Top-P sampling?
- Advanced: Why is setting a high temperature (e.g., 1.5) without limiting Top-P or Top-K likely to produce gibberish?
Interview Answers
- Beginner: A temperature of 0.0 makes the generation deterministic. The model will always select the token with the highest probability, ensuring consistent responses.
- Intermediate: Top-K sampling limits choices to a fixed number () of top tokens. Top-P (Nucleus Sampling) dynamically limits choices to a variable number of tokens whose cumulative probability exceeds the threshold .
- Advanced: A high temperature flattens the probability distribution, raising the relative likelihood of low-probability, irrelevant words. Without Top-P or Top-K truncation to filter them out, the model is highly likely to sample these low-probability tokens, leading to formatting errors and nonsensical sentences.
Common Mistakes
- Combining conflicting settings: Setting temperature to 1.5 and Top-P to 0.05. A low Top-P restricts the model to a tiny subset of tokens, overriding the high temperature and making the output deterministic anyway. Tune one parameter primarily, or adjust them in harmony.
Assignment
Write a Python script that generates 5 sample completions for a list of logits. Test with temperature values of 0.1 and 1.2 and compare the consistency of the outputs.
Discussion
Loading discussion...