Top-p (Nucleus Sampling)
ConceptDefinition
A sampling strategy where the model selects the next token from the smallest set of candidates whose cumulative probability exceeds p. Balances diversity and coherence more adaptively than fixed top-k sampling. Often used alongside temperature.
Related Terms
Temperature
A sampling hyperparameter controlling output randomness. At temperature 0, the model always picks the most probable next token (deterministic). Higher values increase diversity and creativity. Values above 1.0 introduce significant noise.
Inference
The process of running a trained model to generate predictions or outputs — as opposed to training (updating weights). Inference efficiency (speed, cost, latency) is the primary concern for production deployments.
Prompt Engineering
The practice of crafting inputs to elicit optimal model outputs. Encompasses techniques like chain-of-thought, few-shot examples, role prompting, structured output instructions, and system prompt design.