17 Batch Processing

Once a workflow has been validated for accuracy and reliability, the next natural step is batch processing—running that workflow automatically across a large set of inputs. In educational measurement, batch processing is what makes an LLM truly operational: instead of scoring or summarizing one response at a time, you can apply the same structured prompt, rubric, or analytic chain across hundreds or thousands of records.

In an R context, batch processing is straightforward because data manipulation and iteration are already native to the language. A vectorized or mapped approach (using purrr::map(), lapply(), or dplyr::rowwise()) lets you loop over responses, items, or datasets, calling the model once per record and collecting outputs into tables. When combined with proper logging, this setup produces complete audit trails of prompts, model versions, and responses, all of which are important for ensuring reproducibility and fairness in large-scale scoring or evaluation systems.

The primary advantages of batch processing are efficiency, consistency, and traceability. It eliminates manual repetition, enforces the same decision rules across all cases, and enables performance metrics (e.g., agreement rates, latency, cost per call) to be calculated directly from logs. For teams managing ongoing assessments, item review pipelines, or feedback generation, batch processing transforms LLM integration from a proof-of-concept into a dependable analytic service which can fit seamlessly within the structured, data-driven workflows familiar to educational measurement professionals.

Not only does batch processing enable you to submit many similar tasks together, it can also lower your API costs. Anthropic’s current Message Batches API charges 50% of standard input and output token prices. Other providers may use different discounts, limits, and batch interfaces, so check the documentation for the provider and model you are using.

A few extra steps are required to submit a batch job, but it is not difficult to implement. Batch jobs are asynchronous and are not necessarily processed immediately. Depending on current demand and your request volume, a job may wait in a queue, and some jobs may expire before completion. This makes batching useful for work that does not require an immediate response, but less appropriate for interactive applications. Nonetheless, it is still much faster and cheaper than doing hundreds of calls by hand, and it can be more economical than sending the same requests one at a time.

Each request in a batch is processed independently. A batch can therefore contain successful, errored, canceled, and expired requests at the same time. Results may not be returned in the order that requests were submitted, so use a unique request identifier such as Anthropic’s custom_id to match each result to its input. Before submitting a large batch, test one representative request through the ordinary Messages API and validate the result-parsing code.

I’ve included a simple batch call in our next activity to give you practice with this valuable skill!

Some additional information about batch processing can be found on Anthropic’s Message Batches API page. Each provider will likely have different functionality:

Some selected information from the Anthropic page:

This approach is well-suited to tasks that do not require immediate responses, with most batches finishing in less than 1 hour while reducing costs by 50% and increasing throughput.
A Message Batch is limited to either 100,000 Message requests or 256 MB in size, whichever is reached first.
We process each batch as fast as possible, with most batches completing within 1 hour. You will be able to access batch results when all messages have completed or after 24 hours, whichever comes first. Batches will expire if processing does not complete within 24 hours.
Batch results are available for 29 days after creation. After that, you may still view the Batch, but its results will no longer be available for download.
Rate limits apply to both Batches API HTTP requests and the number of requests within a batch waiting to be processed. See Anthropic’s Message Batches API rate limits. Processing may also slow down based on current demand and your request volume, increasing the chance that a batch will expire before completion.

# Batch Processing {#sec-batch-processing} Once a workflow has been validated for accuracy and reliability, the next natural step is batch processing—running that workflow automatically across a large set of inputs. In educational measurement, batch processing is what makes an LLM truly operational: instead of scoring or summarizing one response at a time, you can apply the same structured prompt, rubric, or analytic chain across hundreds or thousands of records. In an R context, batch processing is straightforward because data manipulation and iteration are already native to the language. A vectorized or mapped approach (using `purrr::map()`, `lapply()`, or `dplyr::rowwise()`) lets you loop over responses, items, or datasets, calling the model once per record and collecting outputs into tables. When combined with proper logging, this setup produces complete audit trails of prompts, model versions, and responses, all of which are important for ensuring reproducibility and fairness in large-scale scoring or evaluation systems. The primary advantages of batch processing are efficiency, consistency, and traceability. It eliminates manual repetition, enforces the same decision rules across all cases, and enables performance metrics (e.g., agreement rates, latency, cost per call) to be calculated directly from logs. For teams managing ongoing assessments, item review pipelines, or feedback generation, batch processing transforms LLM integration from a proof-of-concept into a dependable analytic service which can fit seamlessly within the structured, data-driven workflows familiar to educational measurement professionals. Not only does batch processing enable you to submit many similar tasks together, it can also lower your API costs. Anthropic's current Message Batches API charges 50\% of standard input and output token prices. Other providers may use different discounts, limits, and batch interfaces, so check the documentation for the provider and model you are using. A few extra steps are required to submit a batch job, but it is not difficult to implement. Batch jobs are asynchronous and are not necessarily processed immediately. Depending on current demand and your request volume, a job may wait in a queue, and some jobs may expire before completion. This makes batching useful for work that does not require an immediate response, but less appropriate for interactive applications. Nonetheless, it is still _much_ faster and cheaper than doing hundreds of calls by hand, and it can be more economical than sending the same requests one at a time. Each request in a batch is processed independently. A batch can therefore contain successful, errored, canceled, and expired requests at the same time. Results may not be returned in the order that requests were submitted, so use a unique request identifier such as Anthropic's `custom_id` to match each result to its input. Before submitting a large batch, test one representative request through the ordinary Messages API and validate the result-parsing code. **I've included a simple batch call in our next activity to give you practice with this valuable skill!** *** Some additional information about batch processing can be found on [Anthropic's Message Batches API page](https://platform.claude.com/docs/en/build-with-claude/batch-processing). Each provider will likely have different functionality: [Some selected information from the Anthropic page:]{.underline} - This approach is well-suited to tasks that do not require immediate responses, with most batches finishing in less than 1 hour while reducing costs by 50% and increasing throughput. - A Message Batch is limited to either 100,000 Message requests or 256 MB in size, whichever is reached first. - We process each batch as fast as possible, with most batches completing within 1 hour. You will be able to access batch results when all messages have completed or after 24 hours, whichever comes first. Batches will expire if processing does not complete within 24 hours. - Batch results are available for 29 days after creation. After that, you may still view the Batch, but its results will no longer be available for download. - Rate limits apply to both Batches API HTTP requests and the number of requests within a batch waiting to be processed. [See Anthropic's Message Batches API rate limits.](https://platform.claude.com/docs/en/api/rate-limits#message-batches-api) Processing may also slow down based on current demand and your request volume, increasing the chance that a batch will expire before completion.