8 Generation Parameters

When large language models produce text, they do so through a probabilistic process—selecting one token at a time based on learned probability distributions derived from vast amounts of training data. Generation parameters govern how that probabilistic process unfolds. Rather than altering what the model “knows,” these parameters influence how it samples from its learned distribution: how much variability is allowed, how long a response can be, and how the model balances predictability with diversity while generating language.

From an educational measurement perspective, generation parameters serve a role analogous to setting conditions for test administration or scoring protocols. They define the boundaries within which the model operates, affecting reliability, reproducibility, and interpretability. Understanding these controls allows researchers and educators to better align model behavior with the goals of a particular task.

Note: Many of these generation parameters were more commonly exposed in earlier generations of foundational models and remain common in many open-source models. Newer generations of “reasoning models” may expose fewer direct sampling controls, or may handle some of these behaviors differently under the hood, but I have kept this section for completeness. In addition, if you’re going to download and use an open-source model locally (which has many benefits, including more control over the broader data ecosystem—you are not sending data to a third-party resource), these parameters may still be useful. There is a section below on generation parameters for reasoning models.

8.1 Sampling Controls

These parameters affect the randomness and diversity of the model output.

8.1.1 Temperature

Temperature controls how much randomness is introduced during text generation. A value near 0 makes the output more deterministic and focused; higher values (e.g., 0.8–1.0) make the output more varied and exploratory. Technically, temperature rescales the model’s token logits before sampling, which has the effect of sharpening or flattening the probability distribution over possible next tokens. For reproducible outputs or grading tasks, low temperature is usually preferred; for brainstorming or ideation, higher values often work better.

8.1.2 `top_p` (Nucleus Sampling)

top_p defines how much of the total probability mass is considered when sampling the next token. The model sorts possible next tokens by probability and keeps only the smallest set whose cumulative probability exceeds p. For example, top_p = 0.9 means the model samples only from the set of tokens that together account for 90% of the probability mass. This is another way to control diversity—lower values produce more predictable text, while higher values allow a wider range of candidate tokens.

8.1.3 `top_k`

top_k restricts the number of candidate tokens the model can choose from at each step. If k = 50, only the 50 most likely next tokens are considered. This parameter is conceptually similar to top_p, but it uses a fixed number of candidates rather than a probability threshold. Some APIs expose either top_p or top_k, while others expose both; in practice, many users adjust only one of them to control randomness.

8.2 Length and Structure Controls

These parameters constrain how much or what kind of text the model can produce.

8.2.1 Max Tokens

max_tokens sets the upper limit for how long the model’s output can be, measured in tokens. If the model reaches this limit, it stops generating even if the response is not complete. This parameter is useful for keeping outputs concise or fitting within budget constraints, since longer outputs consume more tokens (and thus cost more).

8.2.2 Stop Sequences

Stop sequences define one or more strings that tell the model when to stop generating text. When the model outputs any of these sequences, generation ends immediately. This helps control response boundaries—useful for cutting off unwanted explanations or ensuring that responses end cleanly at a specific marker, such as “END SCORE” or “###”.

8.3 Bias and Repetition Controls

These parameters discourage certain token patterns.

8.3.1 Frequency Penalty

frequency_penalty discourages the model from repeating the same words or phrases. It adjusts token probabilities based on how often tokens have already appeared in the current response. Higher values push the model toward more varied wording, while lower or zero values allow freer repetition. It is especially useful for generating longer outputs that should not sound redundant.

8.3.2 Presence Penalty

presence_penalty discourages the model from reusing tokens that have already appeared in the text. Unlike frequency_penalty, which scales with the number of repetitions, the presence penalty applies once a token has appeared at all. Increasing this value nudges the model to introduce newer concepts or vocabulary, which can make generated text feel more diverse and exploratory.

8.4 Reasoning Model Generation Parameters

8.4.1 Reasoning Effort

One of the biggest changes from earlier foundational models to newer reasoning models is that developers now have more explicit control over how much computation a model uses before producing its final answer. Inference time (also called test-time compute) refers to the computational process by which a trained language model generates a response to a given input. Recent advances allow some models to expend additional computation during inference, sometimes by generating internal reasoning tokens or by carrying out more extensive intermediate processing before producing a final output.

Earlier foundational models were generally optimized for quick responses and often exposed fewer direct controls over this kind of extra test-time computation. As widely documented in both research and practice, these models could produce hallucinations or other reasoning errors. One common prompting strategy for earlier models was to include language like “think through your response step-by-step,” which often improved performance on more complex tasks. Model developers recognized the value of giving models more room to reason, and newer systems now expose this more directly through API parameters and chatbot settings. As a result, you may have noticed that chatbot versions of generative AI models sometimes take a bit longer to respond after you submit a prompt. You may also have noticed that some providers offer options such as “Extended Thinking” or “Thinking” instead of “Auto” or “Instant.”

When using an LLM via API, you often have more precise control over how much reasoning a model performs before generating a response. Longer or more intensive reasoning can be helpful for complex tasks, but that does not mean you should always use the highest setting—even lower settings are quite powerful for many use cases. It is also important to remember that this additional reasoning uses tokens. As a result, the cost of a response may reflect not only the tokens in the visible output, but also tokens consumed during intermediate test-time computation.

The effort parameter in the call_claude function that we are using is what gives you control over how much computation Claude is willing to spend on a response. In Anthropic’s current documentation, high is the default for Claude Sonnet 4.6 if you do not explicitly set the parameter (I have used medium as the default for this course). Below is a copy of the table in the Anthropic documentation describing the different effort settings:

Level	Description	Typical use case
`max`	Absolute maximum capability with no constraints on token spending. Opus 4.6 only. Requests using `max` on other models return an error.	Tasks requiring the deepest possible reasoning and most thorough analysis
`high`	High capability. Equivalent to not setting the parameter.	Complex reasoning, difficult coding problems, agentic tasks
`medium`	Balanced approach with moderate token savings.	Agentic tasks that require a balance of speed, cost, and performance
`low`	Most efficient. Significant token savings with some capability reduction.	Simpler tasks that need the best speed and lowest costs, such as subagents

8.5 Prompt Components

8.5.1 System Prompt

The system prompt sets the model’s overall role, tone, or behavior—essentially, the “meta” instruction that defines how the model should interpret everything that follows. For example, it might specify, “You are an R assistant who explains concepts clearly and uses examples.” This prompt influences style, scope, and behavior across the entire interaction. In many chat interfaces and APIs, a system prompt may be supplied by the developer, the platform, or both; if you do not provide one yourself, the provider may still apply hidden system-level instructions or defaults behind the scenes.

8.5.2 User Prompt

The user prompt is the immediate question or task you are asking the model to perform. It represents the actual input or query, such as “Write an R function that calculates bootstrapped confidence intervals.” Together, the system and user prompts help define both who the model should be and what it should do—analogous to a function’s global defaults and its current arguments.

8.5.3 Response Schema

A response schema specifies the structure or format the model should follow when producing its output. For example, you might require responses in JSON with fields like "score" and "rationale". In more formal workflows, this may involve providing a JSON schema or another structured-output specification that the model is asked to follow. Defining a schema encourages consistency across runs, simplifies parsing in R workflows, and reduces the need for post-processing or cleanup.

# Generation Parameters {#sec-gen-params} When large language models produce text, they do so through a probabilistic process—selecting one token at a time based on learned probability distributions derived from vast amounts of training data. Generation parameters govern how that probabilistic process unfolds. Rather than altering what the model “knows,” these parameters influence how it samples from its learned distribution: how much variability is allowed, how long a response can be, and how the model balances predictability with diversity while generating language. From an educational measurement perspective, generation parameters serve a role analogous to setting conditions for test administration or scoring protocols. They define the boundaries within which the model operates, affecting reliability, reproducibility, and interpretability. Understanding these controls allows researchers and educators to better align model behavior with the goals of a particular task. **Note: Many of these generation parameters were more commonly exposed in earlier generations of foundational models and remain common in many open-source models. Newer generations of "reasoning models" may expose fewer direct sampling controls, or may handle some of these behaviors differently under the hood, but I have kept this section for completeness.** In addition, if you're going to download and use an open-source model locally (which has many benefits, including more control over the broader data ecosystem—you are not sending data to a third-party resource), these parameters may still be useful. There is a [section below on generation parameters](08-gen-ai-parameters.qmd#sec-reason-gen-params) for reasoning models. <br> ## Sampling Controls These parameters affect the randomness and diversity of the model output. ### Temperature {#sec-temperature} Temperature controls how much randomness is introduced during text generation. A value near 0 makes the output more deterministic and focused; higher values (e.g., 0.8–1.0) make the output more varied and exploratory. Technically, temperature rescales the model’s token logits before sampling, which has the effect of sharpening or flattening the probability distribution over possible next tokens. For reproducible outputs or grading tasks, low temperature is usually preferred; for brainstorming or ideation, higher values often work better. ### `top_p` (Nucleus Sampling) `top_p` defines how much of the total probability mass is considered when sampling the next token. The model sorts possible next tokens by probability and keeps only the smallest set whose cumulative probability exceeds *p*. For example, `top_p = 0.9` means the model samples only from the set of tokens that together account for 90% of the probability mass. This is another way to control diversity—lower values produce more predictable text, while higher values allow a wider range of candidate tokens. ### `top_k` `top_k` restricts the number of candidate tokens the model can choose from at each step. If `k = 50`, only the 50 most likely next tokens are considered. This parameter is conceptually similar to `top_p`, but it uses a fixed number of candidates rather than a probability threshold. Some APIs expose either `top_p` or `top_k`, while others expose both; in practice, many users adjust only one of them to control randomness. ## Length and Structure Controls These parameters constrain how much or what kind of text the model can produce. ### Max Tokens `max_tokens` sets the upper limit for how long the model’s output can be, measured in tokens. If the model reaches this limit, it stops generating even if the response is not complete. This parameter is useful for keeping outputs concise or fitting within budget constraints, since longer outputs consume more tokens (and thus cost more). ### Stop Sequences Stop sequences define one or more strings that tell the model when to stop generating text. When the model outputs any of these sequences, generation ends immediately. This helps control response boundaries—useful for cutting off unwanted explanations or ensuring that responses end cleanly at a specific marker, such as “END SCORE” or “###”. ## Bias and Repetition Controls These parameters discourage certain token patterns. ### Frequency Penalty `frequency_penalty` discourages the model from repeating the same words or phrases. It adjusts token probabilities based on how often tokens have already appeared in the current response. Higher values push the model toward more varied wording, while lower or zero values allow freer repetition. It is especially useful for generating longer outputs that should not sound redundant. ### Presence Penalty `presence_penalty` discourages the model from reusing tokens that have already appeared in the text. Unlike `frequency_penalty`, which scales with the number of repetitions, the presence penalty applies once a token has appeared at all. Increasing this value nudges the model to introduce newer concepts or vocabulary, which can make generated text feel more diverse and exploratory. ## Reasoning Model Generation Parameters {#sec-reason-gen-params} ### Reasoning Effort One of the biggest changes from earlier foundational models to newer reasoning models is that developers now have more explicit control over how much computation a model uses before producing its final answer. _Inference time_ (also called test-time compute) refers to the computational process by which a trained language model generates a response to a given input. Recent advances allow some models to expend additional computation during inference, sometimes by generating internal reasoning tokens or by carrying out more extensive intermediate processing before producing a final output. Earlier foundational models were generally optimized for quick responses and often exposed fewer direct controls over this kind of extra test-time computation. As widely documented in both research and practice, these models could produce hallucinations or other reasoning errors. One common prompting strategy for earlier models was to include language like "think through your response step-by-step," which often improved performance on more complex tasks. Model developers recognized the value of giving models more room to reason, and newer systems now expose this more directly through API parameters and chatbot settings. As a result, you may have noticed that chatbot versions of generative AI models sometimes take a bit longer to respond after you submit a prompt. You may also have noticed that some providers offer options such as "Extended Thinking" or "Thinking" instead of "Auto" or "Instant." When using an LLM via API, you often have more precise control over how much reasoning a model performs before generating a response. Longer or more intensive reasoning can be helpful for complex tasks, but that does not mean you should always use the highest setting—even lower settings are quite powerful for many use cases. It is also important to remember that this additional reasoning uses tokens. As a result, the cost of a response may reflect not only the tokens in the visible output, but also tokens consumed during intermediate test-time computation. The `effort` parameter in the `call_claude` function that we are using is what gives you control over how much computation Claude is willing to spend on a response. In Anthropic's current documentation, `high` is the default for Claude Sonnet 4.6 if you do not explicitly set the parameter (I have used `medium` as the default for this course). Below is a copy of the table in the [Anthropic documentation](https://platform.claude.com/docs/en/build-with-claude/effort) describing the different effort settings: | Level | Description | Typical use case | |-------|-------------|-----------------| | `max` | Absolute maximum capability with no constraints on token spending. Opus 4.6 only. Requests using `max` on other models return an error. | Tasks requiring the deepest possible reasoning and most thorough analysis | | `high` | High capability. Equivalent to not setting the parameter. | Complex reasoning, difficult coding problems, agentic tasks | | `medium` | Balanced approach with moderate token savings. | Agentic tasks that require a balance of speed, cost, and performance | | `low` | Most efficient. Significant token savings with some capability reduction. | Simpler tasks that need the best speed and lowest costs, such as subagents | ## Prompt Components ### System Prompt {#sec-system-prompt} The _system prompt_ sets the model’s overall role, tone, or behavior—essentially, the “meta” instruction that defines how the model should interpret everything that follows. For example, it might specify, “You are an R assistant who explains concepts clearly and uses examples.” This prompt influences style, scope, and behavior across the entire interaction. In many chat interfaces and APIs, a system prompt may be supplied by the developer, the platform, or both; if you do not provide one yourself, the provider may still apply hidden system-level instructions or defaults behind the scenes. ### User Prompt The _user prompt_ is the immediate question or task you are asking the model to perform. It represents the actual input or query, such as “Write an R function that calculates bootstrapped confidence intervals.” Together, the system and user prompts help define both who the model should be and what it should do—analogous to a function’s global defaults and its current arguments. ### Response Schema A response schema specifies the structure or format the model should follow when producing its output. For example, you might require responses in JSON with fields like `"score"` and `"rationale"`. In more formal workflows, this may involve providing a JSON schema or another structured-output specification that the model is asked to follow. Defining a schema encourages consistency across runs, simplifies parsing in R workflows, and reduces the need for post-processing or cleanup.