# Activity: Batch Processing
Download the notebook for this activity:
<a href="downloads/activity-batch-processing.Rmd" download style="display: inline-block; padding: 10px 20px; background-color: #4A90E2; color: black; text-decoration: none; border-radius: 5px;">
📥 Download `activity-batch-processing.Rmd`
</a>
Before running the examples below, source the workshop's canonical helper and confirm your Anthropic API key is available in the session. This activity reuses the same `call_claude()` interface described in the earlier connection chapter, rather than redefining a separate local version. We'll also reload the essays and function from the previous activity.
```{r, eval = FALSE}
# Packages
library(dplyr)
library(purrr)
library(glue)
if (Sys.getenv("ANTHROPIC_API_KEY") == "") {
Sys.setenv(ANTHROPIC_API_KEY = rstudioapi::askForSecret("Anthropic API Key"))
}
source("downloads/call_claude_AIMECON26.R")
source("downloads/score_essay.R")
# Download the data
load(url("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/student_essays.Rdata"))
# Creating new column of prompts to keep it in the same df
student_essays$scoring_prompt <- sapply(student_essays$essay, score_essay)
# Trying the first one
student_essays$scoring_output <- NA
student_essays$scoring_output[1] <- call_claude(student_essays$scoring_prompt[1])
```
The shared helper returns the model's final text block as a single character string, which makes it easy to pass the result directly into `parse_scores()` or `purrr::map_chr()`.
In the previous activity, we developed and refined a scoring prompt and tested it on a single essay.
In the overview chapter, we saw that batch processing offers a path from that proof-of-concept to something operationally useful: consistent, traceable, and cost-efficient scoring at scale.
This activity works through that progression in two parts.
First, we'll extend the single-essay call from the previous activity into a full loop-based scoring run — the approach you'd reach for when batch processing isn't available or isn't needed.
Then we'll implement the same task using Anthropic's Batch API, compare the approaches, and use the batch infrastructure to explore a question that should feel natural to measurement professionals: how consistent is the model's scoring across repeated calls?
We'll pick up exactly where the previous activity left off: `student_essays` is loaded, `score_essay()` is defined, and we have a single scored essay in `student_essays$scoring_output[1]` to confirm the prompt is working.
---
## Part 1: Scoring Essays in a Loop
### Parsing the JSON Output
Before we can loop across all essays, we need a reliable way to extract scores from the model's JSON response.
The `score_essay()` prompt instructs the model to return a JSON object with six fields (three scores and three justifications) plus a `total_score`.
The function below handles the extraction, including stripping any markdown fencing the model might add despite being asked not to.
```{r, eval = FALSE}
library(jsonlite)
parse_scores <- function(response_text) {
# Strip markdown fencing if present (```json ... ```)
clean <- gsub("```(?:json)?\\s*|```", "", response_text, perl = TRUE)
clean <- trimws(clean)
tryCatch({
scores <- tryCatch(
fromJSON(clean),
error = function(parse_error) {
score_names <- c("content_score", "organization_score",
"language_score", "total_score")
extracted <- vapply(score_names, function(name) {
match <- regexec(
paste0("['\\\"]?", name,
"['\\\"]?\\s*:\\s*(-?[0-9]+(?:\\.[0-9]+)?)"),
clean,
perl = TRUE
)
parts <- regmatches(clean, match)[[1]]
if (length(parts) < 2L) NA_real_ else as.numeric(parts[2])
}, numeric(1))
if (anyNA(extracted)) stop(parse_error)
as.list(setNames(extracted, score_names))
}
)
values <- unlist(scores[c("content_score", "organization_score", "language_score", "total_score")])
stopifnot(length(values) == 4L, is.numeric(values), all(is.finite(values)))
data.frame(
content = values[1],
organization = values[2],
language = values[3],
total = values[4],
stringsAsFactors = FALSE
)
}, error = function(e) {
warning("Could not parse response: ", e$message)
data.frame(content = NA_real_, organization = NA_real_,
language = NA_real_, total = NA_real_)
})
}
```
You can verify it works on the essay you already scored:
```{r, eval = FALSE}
parse_scores(student_essays$scoring_output[1])
```
### Running the Full Loop
With `parse_scores()` in hand, we can score all essays using `purrr::map()`.
We'll time the run so we have a concrete reference point for the batch comparison coming up.
```{r, eval = FALSE}
library(dplyr)
library(purrr)
start_time <- Sys.time()
loop_results <- student_essays |>
mutate(
raw_response = map_chr(scoring_prompt, call_claude),
scores = map(raw_response, parse_scores)
) |>
tidyr::unnest(scores)
end_time <- Sys.time()
cat("Essays scored:", nrow(loop_results), "\n")
cat("Time elapsed: ", round(difftime(end_time, start_time, units = "secs"), 1), "seconds\n")
```
::: {.callout-note}
Because each `map_chr()` call waits for a response before moving to the next, the total runtime grows linearly with the number of essays.
For a small dataset like this one it's perfectly fine, but for hundreds (or thousands!) of records it becomes the bottleneck that motivates everything in Part 2.
:::
### Inspecting the Results
```{r, eval = FALSE}
library(DT)
loop_results |>
select(id, content, organization, language, total) |>
mutate(across(where(is.numeric), ~ round(.x, 2))) |>
arrange(total) |>
datatable(options = list(pageLength = 10),
caption = "Loop-based scores — one call per essay")
```
---
## Part 2: Batch Processing
The loop above works, but it has two limitations worth addressing as you scale up.
First, it's slow: requests are sequential, so the total time is the sum of every individual API round-trip.
Second, it costs more: Anthropic's Batch API reduces both input and output token prices by 50%, so for identical work, a batch job costs half as much as the same number of sequential calls.
The trade-off is that batch jobs aren't instantaneous — they're queued and processed asynchronously, with most completing within an hour.
For any scoring or evaluation task that doesn't require real-time results, that's an easy trade to make.
The five steps below mirror the structure described in the overview chapter: create the request objects, write them to JSONL, submit the job, poll for completion, and retrieve and parse the results.
### Step 1: Create Batch Requests
Each request in a batch is a JSON object that contains a `custom_id` (your identifier, returned with the result so you can match responses back to inputs) and a `params` block that mirrors the body of a standard `/v1/messages` call.
Batch requests go directly to the API, so we need to include the model, token limit, and effort settings used by the shared `call_claude()` helper ourselves.
```{r, eval = FALSE}
create_batch_requests <- function(essays_df) {
lapply(seq_len(nrow(essays_df)), function(i) {
list(
custom_id = paste0("essay_", essays_df$id[i]),
params = list(
model = "claude-sonnet-5",
max_tokens = 4096,
output_config = list(effort = "low"),
messages = list(
list(role = "user", content = essays_df$scoring_prompt[i])
)
)
)
})
}
batch_requests <- create_batch_requests(student_essays)
cat("Requests created:", length(batch_requests), "\n")
```
### Step 2: Write Requests to JSONL
The Batch API expects requests in JSONL format — one JSON object per line, no surrounding array.
`toJSON()` with `auto_unbox = TRUE` handles the serialization; `writeLines()` handles the one-per-line format.
```{r, eval = FALSE}
write_batch_file <- function(batch_requests, output_file = "batch_requests.jsonl") {
jsonl_lines <- sapply(batch_requests, function(req) {
toJSON(req, auto_unbox = TRUE)
})
writeLines(jsonl_lines, output_file)
cat("Written to:", output_file, "\n")
invisible(output_file)
}
batch_jsonl <- write_batch_file(batch_requests)
```
### Step 3: Submit the Batch Job
Submitting sends the JSONL contents as the `requests` field of a POST to `/v1/messages/batches`.
The response includes a batch ID that you'll use to check status and retrieve results.
```{r, eval = FALSE}
library(httr2)
submit_batch <- function(jsonl_file) {
jsonl_lines <- readLines(jsonl_file)
requests_list <- lapply(jsonl_lines, fromJSON, simplifyVector = FALSE)
result <- request("https://api.anthropic.com/v1/messages/batches") |>
req_headers(
"x-api-key" = Sys.getenv("ANTHROPIC_API_KEY"),
"anthropic-version" = "2023-06-01"
) |>
req_body_json(list(requests = requests_list)) |>
req_error(body = function(resp) resp_body_json(resp)$error$message) |>
req_perform() |>
resp_body_json()
cat("Batch submitted successfully!\n")
cat("Batch ID:", result$id, "\n")
cat("Status: ", result$processing_status, "\n")
invisible(result)
}
batch_info <- submit_batch(batch_jsonl)
```
### Step 4: Check Batch Status
Batches process asynchronously. Poll this endpoint until `processing_status` is `"ended"`.
Even a small batch can take longer than expected; check its status before retrieving results.
```{r, eval = FALSE}
check_batch_status <- function(batch_id) {
result <- request("https://api.anthropic.com/v1/messages/batches") |>
req_url_path_append(batch_id) |>
req_headers(
"x-api-key" = Sys.getenv("ANTHROPIC_API_KEY"),
"anthropic-version" = "2023-06-01"
) |>
req_error(body = function(resp) resp_body_json(resp)$error$message) |>
req_perform() |>
resp_body_json()
cat("Status: ", result$processing_status, "\n")
cat("Succeeded: ", result$request_counts$succeeded, "\n")
cat("Errored: ", result$request_counts$errored, "\n")
cat("Canceled: ", result$request_counts$canceled, "\n")
cat("Expired: ", result$request_counts$expired, "\n")
invisible(result)
}
batch_status <- check_batch_status(batch_info$id)
```
::: {.callout-tip}
For jobs you'll leave running while you move on to other work, you can wrap `check_batch_status()` in a polling loop that checks every 30 seconds and prints a message when the job finishes.
For our workshop setting, we'll just call it manually a minute or two after submitting, but I've included syntax below on how you can do this automatically with a polling loop.
:::
```{r poll-batch-status, eval = FALSE}
repeat {
batch_status <- check_batch_status(batch_info$id)
if (identical(batch_status$processing_status, "ended")) {
message("Batch processing finished.")
break
}
Sys.sleep(30)
}
```
### Step 5: Retrieve and Parse Results
Once `processing_status` is `"ended"`, fetch the results.
The response is JSONL again — one result object per line — so we split, parse, and extract the score from each.
```{r, eval = FALSE}
get_batch_results <- function(batch_id) {
results_text <- request("https://api.anthropic.com/v1/messages/batches") |>
req_url_path_append(batch_id, "results") |>
req_headers(
"x-api-key" = Sys.getenv("ANTHROPIC_API_KEY"),
"anthropic-version" = "2023-06-01"
) |>
req_error(body = function(resp) resp_body_json(resp)$error$message) |>
req_perform() |>
resp_body_string()
results_lines <- strsplit(results_text, "\n")[[1]]
results_lines <- results_lines[nzchar(results_lines)]
lapply(results_lines, fromJSON, simplifyVector = FALSE)
}
score_batch_result <- function(result) {
if (!identical(result$result$type, "succeeded")) {
warning("Request ", result$custom_id, " ended as ", result$result$type)
return(NULL)
}
if (!identical(result$result$message$stop_reason, "end_turn")) {
warning("Request ", result$custom_id, " stopped before completing")
return(NULL)
}
blocks <- result$result$message$content
text <- vapply(blocks, function(block) {
if (identical(block$type, "text")) block$text else ""
}, character(1))
scores <- parse_scores(paste(text[text != ""], collapse = ""))
if (anyNA(scores)) {
warning("No valid scores for request ", result$custom_id)
return(NULL)
}
scores
}
extract_batch_scores <- function(results_list) {
rows <- lapply(results_list, function(result) {
scores <- score_batch_result(result)
if (is.null(scores)) return(NULL)
scores$essay_id <- sub("^essay_", "", result$custom_id)
scores
})
rows <- Filter(Negate(is.null), rows)
if (length(rows) == 0L) stop("No valid scores in this batch")
do.call(rbind, rows)
}
if (!identical(batch_status$processing_status, "ended")) stop("Batch is still processing")
batch_results <- get_batch_results(batch_info$id)
batch_scores <- extract_batch_scores(batch_results)
```
### Comparing Loop and Batch Results
Both runs use the same prompt, model, and effort setting. Compare the scores rather than assuming two independent calls will agree exactly.
```{r, eval = FALSE}
library(lubridate)
# How long did the batch take?
batch_start <- ymd_hms(batch_status$created_at)
batch_end <- ymd_hms(batch_status$ended_at)
cat("Batch run time:", round(difftime(batch_end, batch_start, units = "mins"), 2), "minutes\n")
# Side-by-side score comparison
loop_scores <- loop_results |>
select(id, content, organization, language, total) |>
rename(essay_id = id)
comparison <- loop_scores |>
mutate(essay_id = as.character(essay_id)) |>
rename_with(~ paste0("loop_", .x), -essay_id) |>
left_join(
batch_scores |>
mutate(essay_id = as.character(essay_id)) |>
rename_with(~ paste0("batch_", .x), -essay_id),
by = "essay_id"
)
# save(comparison, file = "data/loop-vs-batch-comparison.RData")
load(url("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/loop-vs-batch-comparison.RData"))
datatable(comparison |> mutate(across(where(is.numeric), ~ round(.x, 2))),
options = list(pageLength = 10, scrollX = TRUE),
caption = "Loop vs. Batch scores — same prompt, same model")
```
---
## Extension: Exploring Scoring Consistency
The loop and batch runs above each scored each essay once.
That's appropriate for production scoring, but as measurement professionals you'll rightly ask: how stable are those scores?
What happens when we keep the prompt, model, and settings fixed and ask again?
We'll use Claude Sonnet 5 at `effort = "low"`, matching the workshop helper. Effort governs how much reasoning the model spends; it is not a replacement for the sampling control discussed in [Temperature](08-gen-ai-parameters.qmd#sec-temperature). Repeated calls can still produce different scores, so we'll measure what actually happens rather than predict it.
Batch processing makes a multi-rep study more economical, though it still incurs charges and may take time. The following code prepares 20 calls per essay in a single batch. Check the printed request count before submitting it.
```{r, eval = FALSE}
create_multi_rep_requests <- function(essays_df, n_reps = 20) {
requests <- list()
k <- 1L
for (i in seq_len(nrow(essays_df))) {
params <- list(
model = "claude-sonnet-5",
max_tokens = 4096,
output_config = list(effort = "low"),
messages = list(list(role = "user", content = essays_df$scoring_prompt[i]))
)
for (rep in seq_len(n_reps)) {
requests[[k]] <- list(
custom_id = paste0("essay_", essays_df$id[i], "_rep_", rep),
params = params
)
k <- k + 1L
}
}
cat("Created", length(requests), "requests for",
nrow(essays_df), "essays x", n_reps, "repetitions\n")
invisible(requests)
}
# Helper: extract scores from a multi-rep batch, preserving essay_id and rep
extract_multirep_scores <- function(results_list) {
rows <- lapply(results_list, function(result) {
scores <- score_batch_result(result)
if (is.null(scores)) return(NULL)
scores$essay_id <- sub("^essay_(.*)_rep_[0-9]+$", "\\1", result$custom_id)
scores$rep <- as.integer(sub("^.*_rep_", "", result$custom_id))
scores
})
rows <- Filter(Negate(is.null), rows)
if (length(rows) == 0L) stop("No valid scores in this batch")
do.call(rbind, rows)
}
```
```{r, eval = FALSE}
repeat_requests <- create_multi_rep_requests(student_essays, n_reps = 20)
cat("Confirm before submitting:", length(repeat_requests), "requests\n")
repeat_jsonl <- write_batch_file(repeat_requests, "batch_repeat_scores.jsonl")
```
Once you've checked the count and are ready to incur the cost, submit the batch:
```{r, eval = FALSE}
repeat_info <- submit_batch(repeat_jsonl)
```
```{r, eval = FALSE}
repeat_status <- check_batch_status(repeat_info$id)
if (!identical(repeat_status$processing_status, "ended")) stop("Batch is still processing")
repeat_results <- get_batch_results(repeat_info$id)
repeat_scores <- extract_multirep_scores(repeat_results)
```
### Summarising Consistency
```{r, eval = FALSE}
summarise_scores <- function(df, essays_df) {
valid_summary <- df |>
group_by(essay_id) |>
summarise(
n_valid = n(),
total_mean = mean(total, na.rm = TRUE),
total_sd = sd(total, na.rm = TRUE),
total_min = min(total),
total_max = max(total),
content_mean = mean(content, na.rm = TRUE),
content_sd = sd(content, na.rm = TRUE),
org_mean = mean(organization, na.rm = TRUE),
org_sd = sd(organization, na.rm = TRUE),
lang_mean = mean(language, na.rm = TRUE),
lang_sd = sd(language, na.rm = TRUE),
.groups = "drop"
) |>
mutate(across(-c(essay_id, n_valid), ~ round(.x, 2)))
essays_df |>
transmute(essay_id = as.character(id)) |>
left_join(valid_summary, by = "essay_id") |>
mutate(n_valid = coalesce(n_valid, 0L)) |>
arrange(total_mean)
}
repeat_summary <- summarise_scores(repeat_scores, student_essays)
# save(repeat_summary, file = "data/repeat-summary.RData")
```
```{r, eval = FALSE}
load(url("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/repeat-summary.RData"))
datatable(repeat_summary,
options = list(pageLength = 10, scrollX = TRUE),
caption = "Score repeatability — Sonnet 5, low effort (20 requests per essay)")
```
::: {.callout-note}
## What to look for
Compare the number of valid responses with the 20 requests per essay, then inspect the range and SD of each criterion. An SD of zero means the observed scores matched in this run, not that the model will always give the same score. With only one valid response, the SD is undefined; investigate missing or invalid responses before interpreting the results.
A few things worth reflecting on from a measurement standpoint:
- **Repeatability vs. correctness.** A model that always assigns the same score to an essay hasn't necessarily converged on the *correct* score. Comparing with human or adjudicated scores is a separate validation step.
- **Score distributions.** Which essays have the widest within-essay spread? Are the criterion scores or rank ordering stable enough for the intended use?
- **Cost implications.** This extension submits `20 * nrow(student_essays)` requests in addition to the earlier loop and batch. Anthropic's batch discount reduces the token price, not the number of requests; review the count before running a larger study.
:::