9 Activity: Generation Parameter Testing – Integrating Generative AI into R Workflows: From APIs to Shiny Apps

9.1 `claude_plus` function

You can download the following function by clicking the copy button (looks like a clipboard) in the code chunk displayed below. Alternatively, you can click the icon below:

📥 Download claude_plus

Code

claude_plus <- function(prompt,
                        model = "claude-sonnet-4-5-20250929",
                        system = NULL,
                        temperature = NULL,
                        top_p = NULL,
                        top_k = NULL,
                        max_tokens = 1024) {
  
  # Check if both temperature and top_p are supplied
  # if (!is.null(temperature) && !is.null(top_p)) {
  #   warning("Both temperature and top_p arguments are supplied. The Anthropic API does not support using both simultaneously. Only temperature will be used.")
  # }
  
  # Get API key from environment
  # You will need to change this to your own API key after workshop
  api_key <- Sys.getenv("ANTHROPIC_API_KEY")
  # Convert text prompt to required message format
  messages <- list(list(role = "user", content = prompt))
  
  # Build request body
  request_body <- list(
    model = model,
    messages = messages,
    max_tokens = max_tokens # Required; will be an argument in other functions
  )
  
  # Add system prompt if provided
  if (!is.null(system)) {
    request_body$system <- system
  }
  
  # Add temperature if provided (takes precedence over top_p)
  # if (!is.null(temperature)) {
  request_body$temperature <- temperature
  # } else if (!is.null(top_p)) {
  #   # Only add top_p if temperature is not provided
  request_body$top_p <- top_p
  # }
  # 
  # Add top_k if provided
  if (!is.null(top_k)) {
    request_body$top_k <- top_k
  }
  
  # Set up headers
  headers <- add_headers(
    "x-api-key" = api_key,
    "anthropic-version" = "2023-06-01",
    "content-type" = "application/json"
  )
  
  # Make the API request
  response <- POST(
    url = "https://api.anthropic.com/v1/messages",
    headers,
    body = toJSON(request_body, auto_unbox = TRUE)
  )
  
  # Check if request was successful
  if (http_status(response)$category != "Success") {
    stop(paste("API request failed:", http_status(response)$message, 
               "\nDetails:", content(response, "text", encoding = "UTF-8")))
  }
  
  # Parse response and extract text content
  result <- fromJSON(content(response, "text", encoding = "UTF-8"))
  return(as.character(result$content)[2])
}

Understanding how these generation parameters change model output is more obvious when you can see the different outputs from the model using the same parameters. I’ve made the claude_param_test function to help with this - it includes a n_reps argument (default = 5, max of 20) that replicates the call several times.

9.2 `claude_param_test` function

You can download the following function by clicking the copy button (looks like a clipboard) in the code chunk displayed below. Alternatively, you can click the icon below:

📥 Download claude_param_test

Code

claude_param_test <- function(prompt,
                              temperature = NULL,
                              top_p = NULL,
                              top_k = NULL,
                              max_tokens = 1024,
                              n_reps = 5) {
  
  # Check if n_reps is within allowed range
  if (n_reps > 20) {
    warning("n_reps exceeds maximum allowed value of 20. Setting n_reps to 20.")
    n_reps <- 20
  }
  
  if (n_reps < 5) {
    warning("n_reps is below minimum value of 5. Setting n_reps to 5.")
    n_reps <- 5
  }
  
  # Initialize results dataframe
  results <- data.frame(
    rep_n = integer(),
    temp = numeric(),
    top_p = numeric(),
    top_k = integer(),
    output = character(),
    stringsAsFactors = FALSE
  )
  
  # Loop through n_reps
  for (i in 1:n_reps) {
    # Call claude_plus
    output <- claude_plus(
      prompt = prompt,
      temperature = temperature,
      top_p = top_p,
      top_k = top_k,
      max_tokens = max_tokens
    )
    
    # Add to results
    results <- rbind(results, data.frame(
      rep_n = i,
      temp = ifelse(is.null(temperature), NA, temperature),
      top_p = ifelse(is.null(top_p), NA, top_p),
      top_k = ifelse(is.null(top_k), NA, top_k),
      output = output,
      stringsAsFactors = FALSE
    ))
    
    # Sleep between calls (except after last call)
    if (i < n_reps) {
      Sys.sleep(0.2)
    }
  }
  
  return(results)
}

9.3 Quick Reference Generation Parameter Table

Generation Parameter	Description	Range
`temperature`	Affects creativity by changing the probability distribution when choosing the next token	\(0 \leq \texttt{temperature} \leq 1\)
`top_p`	Uses nucleus sampling to limit token choices by selecting options that make up the `top_p` proportion of options	\(0 < \texttt{top\_p} \leq 1\)
`top_k`	Samples from only the top `k` most probable tokens	\(\texttt{top\_k} \geq 1\)

9.4 Task 1: High `temperature`

For each of the following tasks, you’re welcome to use my prompt or update the syntax with your own.

Code

high_pizza <- claude_param_test("In 15 words or fewer, tell me why pizza is so good.",
                                temperature = 1,
                                n_reps = 10)
save(high_pizza, file = "./data/high_pizza.Rdata")

Code

library(knitr)
load("data/high_pizza.Rdata")
kable(high_pizza)

rep_n	temp	top_p	top_k	output
1	1	NA	NA	Perfect combination of carbs, cheese, and toppings creates irresistible flavors and satisfying textures.
2	1	NA	NA	Perfect blend of cheese, sauce, and bread hits all our cravings at once.
3	1	NA	NA	Perfect combo of savory cheese, tangy sauce, crispy crust, and endless topping possibilities.
4	1	NA	NA	Perfect blend of cheese, sauce, and crispy crust satisfies multiple cravings at once.
5	1	NA	NA	Perfect combo of carbs, cheese, and toppings creates irresistible savory flavors and satisfying textures.
6	1	NA	NA	Perfect combo of melty cheese, tangy sauce, crispy crust, and endless topping possibilities.
7	1	NA	NA	Perfect blend of cheese, sauce, bread, and toppings creates irresistible savory, satisfying comfort food.
8	1	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates an irresistible savory flavor explosion.
9	1	NA	NA	Perfect combo of cheese, sauce, carbs, and endless toppings. Universally craveable comfort food.
10	1	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates an irresistible savory flavor explosion.

9.5 Task 2: Low `temperature`

Code

low_pizza <- claude_param_test("In 15 words or fewer, tell me why pizza is so good.",
                                temperature = 0,
                                n_reps = 10)
save(low_pizza, file = "./data/low_pizza.Rdata")

Code

load("data/low_pizza.Rdata")
kable(low_pizza)

rep_n	top_p	top_k	output
1	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
2	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
3	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
4	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
5	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
6	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
7	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
8	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
9	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
10	NA	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.

Note although the results from this experiment show that the output was identical across the 10 calls, this was by chance; setting the temperature to 0 does not guarantee the exact same outputs will be obtained across different calls.

9.6 Task 3: Low `temperature`, longer output

I repeated the above task, but increased max_token = 2048 and use the prompt of “Write a short essay about the importance of educational measurement.” I was shocked to see that all the essays were still identical!

Code

edmeasure_low <- claude_param_test("Write a short essay about the importance of educational measurement.",
                                   temperature = 0,
                                   n_reps = 5,
                                   max_tokens = 2048)

The Importance of Educational Measurement

Educational measurement serves as the compass guiding modern education systems, providing essential information that shapes teaching, learning, and policy decisions. Far from being merely about assigning grades, effective measurement practices are fundamental to educational quality and equity.

Informing Instruction

At its core, educational measurement helps teachers understand what students know and can do. Through formative assessments, educators identify learning gaps in real-time, allowing them to adjust instruction to meet diverse student needs. This diagnostic function transforms teaching from a one-size-fits-all approach into a responsive, personalized practice that maximizes learning opportunities for every student.

Ensuring Accountability

Measurement provides transparency and accountability within education systems. Standardized assessments offer comparable data across schools and districts, helping stakeholders evaluate whether educational resources are being used effectively and whether students are meeting established learning standards. This accountability, when implemented thoughtfully, drives continuous improvement and ensures that all students receive quality education regardless of their background.

Supporting Student Growth

Valid and reliable assessments give students clear feedback about their progress, helping them understand their strengths and areas for improvement. This information empowers learners to take ownership of their education and set meaningful goals. Moreover, measurement data helps identify students who need additional support, enabling early intervention before small difficulties become major obstacles.

Guiding Policy and Resource Allocation

Educational measurement informs critical decisions about curriculum development, resource distribution, and policy reform. Data-driven insights help administrators and policymakers identify successful programs worth expanding and ineffective practices requiring revision.

In conclusion, educational measurement is indispensable to effective education. When conducted ethically and interpreted wisely, it illuminates the path toward educational excellence and equity for all learners.

The Importance of Educational Measurement

Educational measurement serves as the compass guiding modern education systems, providing essential information that shapes teaching, learning, and policy decisions. Far from being merely about assigning grades, effective measurement practices are fundamental to educational quality and equity.

Informing Instruction

At its core, educational measurement helps teachers understand what students know and can do. Through formative assessments, educators identify learning gaps in real-time, allowing them to adjust instruction to meet diverse student needs. This diagnostic function transforms teaching from a one-size-fits-all approach into a responsive, personalized practice that maximizes learning opportunities for every student.

Ensuring Accountability

Measurement provides transparency and accountability within education systems. Standardized assessments offer comparable data across schools and districts, helping stakeholders evaluate whether educational resources are being used effectively and whether students are meeting established learning standards. This accountability, when implemented thoughtfully, drives continuous improvement and ensures that all students receive quality education regardless of their background.

Supporting Student Growth

Valid and reliable assessments give students clear feedback about their progress, helping them understand their strengths and areas for improvement. This information empowers learners to take ownership of their education and set meaningful goals. Moreover, measurement data helps identify students who need additional support, enabling early intervention before small difficulties become major obstacles.

Guiding Policy and Resource Allocation

Educational measurement informs critical decisions about curriculum development, resource distribution, and policy reform. Data-driven insights help administrators and policymakers identify successful programs worth expanding and ineffective practices requiring revision.

In conclusion, educational measurement is indispensable to effective education. When conducted ethically and interpreted wisely, it illuminates the path toward educational excellence and equity for all learners.

The Importance of Educational Measurement

Educational measurement serves as the compass guiding modern education systems, providing essential information that shapes teaching, learning, and policy decisions. Far from being merely about assigning grades, effective measurement practices are fundamental to educational quality and equity.

Informing Instruction

At its core, educational measurement helps teachers understand what students know and can do. Through formative assessments, educators identify learning gaps in real-time, allowing them to adjust instruction to meet diverse student needs. This diagnostic function transforms teaching from a one-size-fits-all approach into a responsive, personalized practice that maximizes learning opportunities for every student.

Ensuring Accountability

Measurement provides transparency and accountability within education systems. Standardized assessments offer comparable data across schools and districts, helping stakeholders evaluate whether educational resources are being used effectively and whether students are meeting established learning standards. This accountability, when implemented thoughtfully, drives continuous improvement and ensures that all students receive quality education regardless of their background.

Supporting Student Growth

Valid and reliable assessments give students clear feedback about their progress, helping them understand their strengths and areas for improvement. This information empowers learners to take ownership of their education and set meaningful goals. Moreover, measurement data helps identify students who need additional support, enabling early intervention before small difficulties become major obstacles.

Guiding Policy and Resource Allocation

Educational measurement informs critical decisions about curriculum development, resource distribution, and policy reform. Data-driven insights help administrators and policymakers identify successful programs worth expanding and ineffective practices requiring revision.

In conclusion, educational measurement is indispensable to effective education. When conducted ethically and interpreted wisely, it illuminates the path toward educational excellence and equity for all learners.

The Importance of Educational Measurement

Educational measurement serves as the compass guiding modern education systems, providing essential information that shapes teaching, learning, and policy decisions. Far from being merely about assigning grades, effective measurement practices are fundamental to educational quality and equity.

Informing Instruction

At its core, educational measurement helps teachers understand what students know and can do. Through formative assessments, educators identify learning gaps in real-time, allowing them to adjust instruction to meet diverse student needs. This diagnostic function transforms teaching from a one-size-fits-all approach into a responsive, personalized practice that maximizes learning opportunities for every student.

Ensuring Accountability

Measurement provides transparency and accountability within education systems. Standardized assessments offer comparable data across schools and districts, helping stakeholders evaluate whether educational resources are being used effectively and whether students are meeting established learning standards. This accountability, when implemented thoughtfully, drives continuous improvement and ensures that all students receive quality education regardless of their background.

Supporting Student Growth

Valid and reliable assessments give students clear feedback about their progress, helping them understand their strengths and areas for improvement. This information empowers learners to take ownership of their education and set meaningful goals. Moreover, measurement data helps identify students who need additional support, enabling early intervention before small difficulties become major obstacles.

Guiding Policy and Resource Allocation

Educational measurement informs critical decisions about curriculum development, resource distribution, and policy reform. Data-driven insights help administrators and policymakers identify successful programs worth expanding and ineffective practices requiring revision.

In conclusion, educational measurement is indispensable to effective education. When conducted ethically and interpreted wisely, it illuminates the path toward educational excellence and equity for all learners.

The Importance of Educational Measurement

Educational measurement serves as the compass guiding modern education systems, providing essential information that shapes teaching, learning, and policy decisions. Far from being merely about assigning grades, effective measurement practices are fundamental to educational quality and equity.

Informing Instruction

At its core, educational measurement helps teachers understand what students know and can do. Through formative assessments, educators identify learning gaps in real-time, allowing them to adjust instruction to meet diverse student needs. This diagnostic function transforms teaching from a one-size-fits-all approach into a responsive, personalized practice that maximizes learning opportunities for every student.

Ensuring Accountability

Measurement provides transparency and accountability within education systems. Standardized assessments offer comparable data across schools and districts, helping stakeholders evaluate whether educational resources are being used effectively and whether students are meeting established learning standards. This accountability, when implemented thoughtfully, drives continuous improvement and ensures that all students receive quality education regardless of their background.

Supporting Student Growth

Valid and reliable assessments give students clear feedback about their progress, helping them understand their strengths and areas for improvement. This information empowers learners to take ownership of their education and set meaningful goals. Moreover, measurement data helps identify students who need additional support, enabling early intervention before small difficulties become major obstacles.

Guiding Policy and Resource Allocation

Educational measurement informs critical decisions about curriculum development, resource distribution, and policy reform. Data-driven insights help administrators and policymakers identify successful programs worth expanding and ineffective practices requiring revision.

In conclusion, educational measurement is indispensable to effective education. When conducted ethically and interpreted wisely, it illuminates the path toward educational excellence and equity for all learners.

9.7 Task 4: High `top_p`

Code

highp_pizza <- claude_param_test("In 15 words or fewer, tell me why pizza is so good.",
                                 top_p = .90,
                                 n_reps = 10)
save(highp_pizza, file = "./data/highp_pizza.Rdata")

Code

load("data/highp_pizza.Rdata")
kable(highp_pizza)

rep_n	temp	top_p	top_k	output
1	NA	0.9	NA	Perfect combo of savory cheese, tangy sauce, crispy crust, and endless topping possibilities.
2	NA	0.9	NA	Perfect combo of cheese, sauce, bread, and toppings hits all our taste buds.
3	NA	0.9	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
4	NA	0.9	NA	Perfect combo of cheese, sauce, bread, and toppings—salty, savory, and endlessly customizable.
5	NA	0.9	NA	Perfect combo of crispy crust, tangy sauce, melted cheese, and endless topping possibilities.
6	NA	0.9	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
7	NA	0.9	NA	Perfect combo of cheese, sauce, carbs, and endless toppings satisfies multiple cravings at once.
8	NA	0.9	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
9	NA	0.9	NA	Perfect combo of cheese, sauce, carbs, and endless toppings. Plus it’s shareable and fun!
10	NA	0.9	NA	Perfect combo of crispy crust, tangy sauce, melted cheese, and endless topping possibilities.

9.8 Task 5: Low `top_p`

Code

lowp_pizza <- claude_param_test("In 15 words or fewer, tell me why pizza is so good.",
                                top_p = .10,
                                n_reps = 10)
save(lowp_pizza, file = "./data/lowp_pizza.Rdata")

Code

load("data/lowp_pizza.Rdata")
kable(lowp_pizza)

rep_n	temp	top_p	top_k	output
1	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
2	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
3	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
4	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
5	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
6	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
7	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
8	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
9	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.
10	NA	0.1	NA	Perfect combo of cheese, sauce, bread, and toppings creates irresistible savory flavors and textures.

9.1 claude_plus function

9.2 claude_param_test function

9.3 Quick Reference Generation Parameter Table

9.4 Task 1: High temperature

9.5 Task 2: Low temperature

9.6 Task 3: Low temperature, longer output

9.7 Task 4: High top_p

9.8 Task 5: Low top_p

9.1 `claude_plus` function

9.2 `claude_param_test` function

9.4 Task 1: High `temperature`

9.5 Task 2: Low `temperature`

9.6 Task 3: Low `temperature`, longer output

9.7 Task 4: High `top_p`

9.8 Task 5: Low `top_p`