---
title: "Activity: Prompt Chaining"
output: html_notebook
---

## Setup

Run this chunk first to load required packages, set your API key, and define the shared helper functions used throughout this activity.

```{r setup}
library(httr)
library(jsonlite)
library(dplyr)
library(glue)

if (Sys.getenv("ANTHROPIC_API_KEY") == "") {
  Sys.setenv(ANTHROPIC_API_KEY = rstudioapi::askForSecret("Anthropic API Key"))
}

call_claude <- function(prompt,
                        model = "claude-sonnet-4-6",
                        system = NULL,
                        temperature = 0.5,
                        max_tokens = 4096,
                        effort = "low") {
  api_key <- Sys.getenv("ANTHROPIC_API_KEY")
  messages <- list(list(role = "user", content = prompt))
  request_body <- list(
    model = model,
    messages = messages,
    max_tokens = max_tokens,
    temperature = temperature,
    output_config = list(effort = effort)
  )
  if (!is.null(system)) {
    request_body$system <- system
  }
  headers <- add_headers(
    "x-api-key" = api_key,
    "anthropic-version" = "2023-06-01",
    "content-type" = "application/json"
  )
  response <- POST(
    url = "https://api.anthropic.com/v1/messages",
    headers,
    body = toJSON(request_body, auto_unbox = TRUE)
  )
  if (http_status(response)$category != "Success") {
    stop(paste("API request failed:", http_status(response)$message,
               "\nDetails:", content(response, "text", encoding = "UTF-8")))
  }
  result <- fromJSON(content(response, "text", encoding = "UTF-8"))
  return(result$content$text)
}

clean_json <- function(x) {
  x <- gsub("```[a-zA-Z]*\\n?", "", x)
  x <- gsub("`", "", x)
  x <- trimws(x)
  json_match <- regmatches(x, regexpr("(\\{[\\s\\S]*\\}|\\[[\\s\\S]*\\])", x, perl = TRUE))
  if (length(json_match) > 0) {
    x <- json_match
  }
  parsed <- tryCatch(
    jsonlite::fromJSON(x),
    error = function(e) {
      stop("Could not parse JSON after cleaning. Raw content:\n", x, "\nError: ", e$message)
    }
  )
  return(x)
}
```

---

# Activity: Prompt Chaining

This activity will allow you to practice developing a workflow that involves prompts at several stages. This workflow can be useful when you need to develop a large item pool with extensive construct coverage.

The example that we'll use for this activity is developing essay prompts for a formative middle school science quiz. This will include developing the entire ecosystem for the item, from initial development to providing feedback aligned with performance-level descriptors.

This is a simple and artificial example, but will serve to help you get thinking about developing complex workflows for your own purposes. All of the materials we'll be working with were created with an LLM. In your own setting these materials would likely be created in collaboration with instructional designers, test development, subject-matter experts, etc., but the idea is the same: you can pull in important instructions or guiding materials at each step to improve the model performance. Also importantly, because these materials are not hard-coded in the prompt, you can easily revise them or swap them out for similar materials, preserving your workflow.

---

## Step 1: Select a Guiding Learning Objective

In this step, we will be pulling in a large list of candidate topics and learning objectives, much like you might have with a content blueprint.

```{r step1-setup}
# Load science learning objectives
load(url("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/science_los.Rdata"))

# Convert to text to paste into a prompt
science_lo_text <- science_los %>%
  mutate(
    entry = paste0(
      "[Sub-domain] ", sub_domain, "\n",
      "[Learning Objective] ", learning_objective
    )
  ) %>%
  pull(entry) %>%
  paste(collapse = "\n\n")

choose_topic_prompt <- function(science_lo_text) {
  glue(
    "You are developing a constructed-response item for a middle-school science assessment.

    Below is the approved list of science sub-domains and learning objectives.
    You may ONLY select from these options.

    {science_lo_text}

    TASK:
    Select ONE sub-domain and ONE learning objective pair from the list above.
    Do not invent or rephrase learning objectives; return them exactly as they are written.
    Do not include any additional text - only the selected sub-domain and learning objective."
  )
}
```

```{r step1-call}
step1_res <- call_claude(choose_topic_prompt(science_lo_text))
cat(step1_res)
```

---

## Step 2: Choose Writing Learning Objectives

```{r step2-setup}
# Load writing learning objectives
load(url("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/writing_los.Rdata"))

writing_lo_text <- writing_los %>%
  mutate(
    entry = paste0(
      "[Writing Learning Objective] ",
      writing_learning_objective
    )
  ) %>%
  pull(entry) %>%
  paste(collapse = "\n\n")

pick_writing_los <- function(science_topic, writing_lo_text){
  glue(
  "You are assisting with the development of a constructed-response item
  for a middle-school science assessment.

  ASSESSMENT PURPOSE:
  This assessment is designed for middle-school students to:
  - Demonstrate their understanding of scientific concepts, AND
  - Practice key writing skills appropriate for explaining scientific ideas in writing.

  Step 1 has already been completed.
  The selected science context is: {science_topic}

  Below is the approved list of candidate WRITING learning objectives.
  These objectives focus on writing quality and expression, not science content.
  You may ONLY select from this list.

  {writing_lo_text}

  TASK:
  Review the selected science sub-domain and learning objective above.
  Then select TWO writing learning objectives from the candidate list
  that are MOST appropriate for assessing students' written responses
  to a science prompt aligned with this content.

  SELECTION GUIDELINES:
  - Selected writing objectives should support clear explanation of scientific ideas.
  - Avoid objectives that are redundant or not well-suited to a short written response.
  - Do NOT rephrase, invent, or combine learning objectives.

  OUTPUT FORMAT:
  Return your response as a JSON object with the following structure ONLY:

  {{
    \"subdomain\": \"<selected sub-domain>\",
    \"lo\": \"<selected science learning objective>\",
    \"writing_lo1\": \"<first selected writing learning objective>\",
    \"writing_lo2\": \"<second selected writing learning objective>\"
  }}

  Do not include any explanatory text outside of the JSON object."
  )
}
```

```{r step2-call}
step2_res <- call_claude(pick_writing_los(science_topic = step1_res,
                                          writing_lo_text))
cat(step2_res)
```

---

## Step 3: Write Item Stem

Now we have guidelines in place to write the item stem.

```{r step3-setup}
write_essay_prompt <- function(guiding_los) {
  glue(
  "You are assisting with the development of a constructed-response item
  for a middle-school science assessment.

  Below is the approved context and learning objectives for this item,
  provided as a JSON object. These values must be preserved exactly.

  {guiding_los}

  ASSESSMENT CONTEXT:
  - Target students: Middle-school students (approximately grades 6-8)
  - Purpose: To assess students' understanding of scientific concepts
    while giving them practice explaining ideas clearly in writing.

  TASK:
  Using the science and writing learning objectives provided in the JSON
  above, write ONE constructed-response item stem that students will
  respond to in writing.

  ITEM STEM REQUIREMENTS:
  - Align directly with the science learning objective.
  - Encourage explanation of scientific ideas without explicitly mentioning
    writing skills, rubrics, or scoring criteria.
  - Use language appropriate for middle-school reading level or slightly below:
    clear, simple sentences and familiar vocabulary with minimal technical jargon.
  - Be answerable in one short paragraph (approximately 5-8 sentences).

  DO NOT:
  - Modify, remove, or rename any existing fields in the JSON.
  - Add commentary or explanatory text outside the JSON object.
  - Include scoring guidance, rubrics, or example responses.

  OUTPUT FORMAT:
  Return the SAME JSON object provided above with ONE additional field:

  {{
    \"subdomain\": \"...\",
    \"lo\": \"...\",
    \"writing_lo1\": \"...\",
    \"writing_lo2\": \"...\",
    \"item_stem\": \"<item stem written for students>\"
  }}

  Do not include any text outside of the JSON object."
  )
}
```

```{r step3-call}
string_guiding_los <- clean_json(step2_res)

step3_res <- call_claude(write_essay_prompt(string_guiding_los))
cat(step3_res)
```

---

## Step 4: Develop Analytic Rubric with PLDs

```{r step4-setup}
# Load PLD guidance
pld_guidance <- paste(
  readLines("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/pld_guidance.txt"),
  collapse = "\n"
)

make_pld_rubric <- function(item_components, pld_guidance) {
  glue(
"You are assisting with the development of an analytic scoring rubric for a \
middle-school science assessment.

ITEM SPECIFICATION:
The following JSON object contains the current item specification. \
All existing fields must be preserved exactly in your output.

{item_components}

ASSESSMENT CONTEXT:
- Target students: Middle-school students (grades 6-8)
- The item assesses both students' understanding of a specific science concept \
and their ability to explain ideas clearly in writing.

GUIDANCE ON PERFORMANCE LEVEL DESCRIPTORS:
{pld_guidance}

TASK:
Using the learning objectives in the JSON above, write performance level \
descriptors for each learning objective. The rubric must include:
- One criterion for the science learning objective
- One criterion for each of the two writing learning objectives

For each criterion, define three performance levels: Weak, Developing, and Competent.

CONSTRAINTS:
- Do not refer to specific scores or point values.
- Do not include feedback directed at the student.
- Do not change the wording of any learning objective.
- Do not add or remove learning objectives.
- Write descriptors in clear, practical language appropriate for educators.

OUTPUT FORMAT:
Return the same JSON object provided above with one additional top-level field \
called \"rubric\". Do not include any text outside of the JSON object.

The \"rubric\" field must use this structure:

{{
  \"rubric\": {{
    \"science_learning_objective\": {{
      \"weak\": \"<description>\",
      \"developing\": \"<description>\",
      \"competent\": \"<description>\"
    }},
    \"writing_learning_objective_1\": {{
      \"weak\": \"<description>\",
      \"developing\": \"<description>\",
      \"competent\": \"<description>\"
    }},
    \"writing_learning_objective_2\": {{
      \"weak\": \"<description>\",
      \"developing\": \"<description>\",
      \"competent\": \"<description>\"
    }}
  }}
}}"
  )
}
```

```{r step4-call}
string_step3_res <- clean_json(step3_res)

step4_res <- call_claude(make_pld_rubric(item_components = string_step3_res,
                                         pld_guidance))
cat(step4_res)
```

---

## Step 5: Scoring a Writing Sample

This is where you'd normally include a real writing sample. For the sake of demonstration, we're going to have Claude generate one for us.

```{r step5-setup}
build_synthetic_essay_prompt <- function(
    rubric_json,
    science_lo  = c("weak", "developing", "competent"),
    writing_lo1 = c("weak", "developing", "competent"),
    writing_lo2 = c("weak", "developing", "competent")
) {
  science_lo  <- match.arg(science_lo)
  writing_lo1 <- match.arg(writing_lo1)
  writing_lo2 <- match.arg(writing_lo2)

  glue(
"You are generating a realistic synthetic essay written by a middle-school student \
in response to a science constructed-response item.

ITEM RECORD:
The following JSON contains the item stem, learning objectives, and analytic rubric \
with performance-level descriptors for each criterion.

{rubric_json}

ASSIGNED PERFORMANCE LEVELS:
The essay you write must reflect the following performance levels:
- Science learning objective:     {science_lo}
- Writing learning objective 1:   {writing_lo1}
- Writing learning objective 2:   {writing_lo2}

TASK:
Write a student essay response to the item stem above. \
The essay must authentically reflect the performance-level descriptor \
for each learning objective at its assigned level.

The three performance dimensions are independent. A student may demonstrate \
competent scientific reasoning while using weak scientific terminology, or may \
show strong cause-and-effect writing while only partially addressing the science. \
Write the essay so that each dimension is clearly at its assigned level without \
artificially inflating or deflating the others.

REALISM GUIDELINES:
- Write as a genuine middle-school student would — not as a model answer and not \
as a caricature of a poor student. Voice, sentence structure, and vocabulary \
should be plausible for a grade 6-8 student.
- Weak performance should reflect genuine misunderstanding or omission, \
not obvious carelessness or random content.
- Developing performance should reflect a student with partial knowledge \
making a real attempt, not one who simply wrote less.
- Competent performance should reflect a student who meets expectations for \
this grade level, not one who writes at a high school or adult level.
- Do not include any meta-commentary, labels, or signals that reveal the \
intended performance levels.

OUTPUT FORMAT:
Return the same JSON object provided above with one additional top-level field \
called \"student_essay\". Do not include any text outside of the JSON object.

{{
  \"student_essay\": \"<essay text>\"
}}"
  )
}

build_scoring_prompt <- function(essay_json) {
  glue(
"You are scoring a middle-school student's essay response to a science \
constructed-response item.

ITEM RECORD:
The following JSON contains the item stem, learning objectives, analytic rubric \
with performance-level descriptors, and the student's written response.

{essay_json}

TASK:
Assign a performance level for each learning objective criterion based on the \
student's essay and the rubric descriptors in the JSON above.

SCORING GUIDELINES:
- Score each criterion independently. The student's performance on one dimension \
should not influence your rating of another.
- Base each rating entirely on what is present in the student's essay — \
not on what is absent, implied, or possible to infer charitably.
- Select the performance level whose descriptor best characterizes the response \
as written. If the response falls between two levels, select the lower level.
- Do not adjust ratings based on the student's apparent effort, grade level, \
or the difficulty of the item.

OUTPUT FORMAT:
Return the same JSON object provided above with three additional top-level fields. \
Do not include any text outside of the JSON object.

{{
  \"science_learning_objective_rating\": \"<weak | developing | competent>\",
  \"writing_learning_objective_1_rating\": \"<weak | developing | competent>\",
  \"writing_learning_objective_2_rating\": \"<weak | developing | competent>\"
}}"
  )
}
```

```{r step5-generate-essay}
rubric_w_plds <- clean_json(step4_res)

essay_specs <- build_synthetic_essay_prompt(rubric_json = rubric_w_plds,
                                            science_lo  = "developing",
                                            writing_lo1 = "competent",
                                            writing_lo2 = "weak")

sample_essay <- call_claude(essay_specs)
cat(sample_essay)
```

### Scoring the Essay

Now we'll score the essay. 
It will be interesting to see if Claude provides the same judgments of the performance-level descriptors in _scoring_ the essay that were used in _generating_ the essay. 
For our purposes, this isn't something we need to worry about at this time — we just want a scored essay for the final step.

```{r step5-score-essay}
string_sample_essay <- clean_json(sample_essay)

scored_essay <- call_claude(build_scoring_prompt(string_sample_essay))
cat(scored_essay)
```

---

## Step 6: Generate Student Feedback Aligned to Rubric

This step helps to ensure that feedback provided to students is appropriately grounded in the learning objectives and performance-level descriptors. 
Note that I'm again pulling in external instructions to help with this step to mirror specific use-case instructions you might have for your task.

```{r step6-setup}
# Load feedback guidance
feedback_guidance <- paste(
  readLines("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/feedback_guidance.txt"),
  collapse = "\n"
)

feedback_prompt <- function(scored_essay, feedback_guidance) {
  glue(
"You are assisting with the generation of written feedback for a middle-school \
student based on their performance on a constructed-response science item.

ITEM RECORD:
The following JSON object contains the complete item record. \
All fields should be treated as final and authoritative.

It includes:
- the item context and learning objectives
- the analytic rubric with performance-level descriptors
- the student's written response
- the performance level already assigned for each learning objective

{scored_essay}

GUIDANCE ON WRITING STUDENT FEEDBACK:
{feedback_guidance}

TASK:
Using the student's response, the rubric, and the assigned performance levels \
in the JSON above, write specific, constructive feedback for the student. \
Provide a separate feedback comment for each learning objective.

CONSTRAINTS:
- Write directly to the student in age-appropriate language.
- Base each comment on what is actually present in the student's response, \
not on a generic description of the performance level.
- Do not name or imply the performance level label (weak, developing, competent).
- Do not restate rubric descriptor language verbatim.
- Do not question or revise the assigned performance levels.
- Do not introduce new learning objectives or scoring criteria.
- Do not include grades, points, or score values.
- Each comment should identify one specific, actionable suggestion for improvement \
where relevant.

OUTPUT FORMAT:
Return the same JSON object provided above with one additional top-level field \
called \"student_feedback\". Do not include any text outside of the JSON object.

The \"student_feedback\" field must use this structure:

{{
  \"student_feedback\": {{
    \"science_learning_objective\": \"<feedback text>\",
    \"writing_learning_objective_1\": \"<feedback text>\",
    \"writing_learning_objective_2\": \"<feedback text>\"
  }}
}}"
  )
}
```

```{r step6-call}
string_scored_essay <- clean_json(scored_essay)

essay_w_feedback <- call_claude(feedback_prompt(string_scored_essay, feedback_guidance))
cat(essay_w_feedback)
```

---

Look how far we've come! From the simple starting place of choosing a learning objective from a pre-determined list, we've gone all the way to developing an essay question, a rubric with performance-level descriptors, _and_ provided feedback to the student on their performance. How cool!

---

## BONUS: Reading in PDFs or Word documents

In the process above I read in .txt files. Claude can also read in PDFs. For other formats - e.g., .docx - you'll have to use another package - such as `officer` to extract the text. For PDFs there's a size limit of 32MB and 100 pages, but that should be more than sufficient for most use cases.

I've re-written the main `call_claude` function to include an argument for reading in either a .pdf or .docx file. The .pdf is directly encoded via the `base64enc` function, whereas the content from the .docx document is scraped via the `read_docx` argument in from the [`officer` package.`](https://davidgohel.github.io/officer/)

### Main function

```{r, eval = FALSE}

library(httr)
library(jsonlite)
library(base64enc)
library(officer)

call_claude_doc <- function(prompt,
                            file_path,
                            model = "claude-sonnet-4-6",
                            system = NULL,
                            temperature = 0.5,
                            max_tokens = 4096,
                            effort = "low") {
  
  api_key <- Sys.getenv("ANTHROPIC_API_KEY")
  
  # ── Detect file type ──────────────────────────────────────────────────────
  ext <- tolower(tools::file_ext(file_path))
  
  if (!ext %in% c("pdf", "docx")) {
    stop("Unsupported file type '.", ext, "'. Only PDF and Word (.docx) files are supported.")
  }
  
  # ── Handle URL vs. local path ─────────────────────────────────────────────
  if (grepl("^https?://", file_path)) {
    tmp <- tempfile(fileext = paste0(".", ext))
    download.file(file_path, tmp, mode = "wb", quiet = TRUE)
    file_path <- tmp
    on.exit(unlink(tmp))
  }
  
  # ── Build message content based on file type ──────────────────────────────
  if (ext == "pdf") {
    
    # PDFs are natively supported — send as base64-encoded document block
    file_base64 <- base64enc::base64encode(file_path)
    
    message_content <- list(
      list(
        type = "document",
        source = list(
          type       = "base64",
          media_type = "application/pdf",
          data       = file_base64
        )
      ),
      list(
        type = "text",
        text = prompt
      )
    )
    
  } else if (ext == "docx") {
    
    # Word docs are not natively supported — extract text via officer
    # and prepend it to the prompt as plain text
    doc       <- officer::read_docx(file_path)
    doc_text  <- paste(officer::docx_summary(doc)$text, collapse = "\n")
    
    combined_prompt <- paste0(
      "The following is the content of a Word document:\n\n",
      doc_text,
      "\n\n---\n\n",
      prompt
    )
    
    message_content <- list(
      list(
        type = "text",
        text = combined_prompt
      )
    )
    
  }
  
  # ── Build and send request ─────────────────────────────────────────────────
  messages <- list(
    list(
      role    = "user",
      content = message_content
    )
  )
  
  request_body <- list(
    model         = model,
    messages      = messages,
    max_tokens    = max_tokens,
    temperature   = temperature,
    output_config = list(effort = effort)
  )
  
  if (!is.null(system)) {
    request_body$system <- system
  }
  
  headers <- add_headers(
    "x-api-key"         = api_key,
    "anthropic-version" = "2023-06-01",
    "content-type"      = "application/json"
  )
  
  response <- POST(
    url     = "https://api.anthropic.com/v1/messages",
    headers,
    body    = toJSON(request_body, auto_unbox = TRUE)
  )
  
  if (http_status(response)$category != "Success") {
    stop(paste("API request failed:", http_status(response)$message,
               "\nDetails:", content(response, "text", encoding = "UTF-8")))
  }
  
  result <- fromJSON(content(response, "text", encoding = "UTF-8"))
  return(result$content$text)
  
}

# Example usage:

# GitHub PDF
# cat(call_claude_doc(
#   prompt    = "Summarize the key guidance in this document.",
#   file_path = "https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/feedback_guidance.pdf"
# ))

# GitHub Word document
# cat(call_claude_doc(
#   prompt    = "Summarize this document.",
#   file_path = "https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/feedback_guidance.docx"
# ))


```

### PDF document

```{r, eval = FALSE}
pdf_summary <- call_claude_doc("Summarize the attached document in a single paragraph",
                               file_path = "https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/feedback_guidance.pdf")

cat(pdf_summary)
```

### Word document

```{r, eval = FALSE}
docx_summary <- call_claude_doc("Summarize the attached document in a single paragraph",
                               file_path = "https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/feedback_guidance.docx")

cat(docx_summary)
```