16 Activity: Iterative Prompt Development

Download the notebook for this activity: 📥 Download activity-iterative-prompt-development.Rmd

Now it’s time to practice the first element of an LLM workflow: iterative prompt development. This is the process of developing and tweaking a prompt until it’s ready to be implemented more widely in your workflow. One strategy we’ll use here is metaprompting: asking an LLM to review and improve a prompt before that prompt is used for the real task. In other words, the model is not scoring essays yet; it is helping us improve the instructions we will later use for scoring.

For the purposes of this activity, I made some synthetic essay responses with Claude Sonnet 4.5 based on the Dataset for Rubric-based Essay Scoring (DREsS) data, which is based on essays written by English as a foreign language learners. I used responses to the essay prompt, “Do you think that smartphones have destroyed communication among family and friends? Give specific reasons and details to support your opinion?” when creating the 10 synthetic essays.

The full dataset is available after filling out a consent form.

The following rubric was developed for their analysis:

Content: Paragraph is well-developed and relevant to the argument, supported with strong reasons and examples.
Organization: The argument is very effectively structured and developed, making it easy for the reader to follow the ideas and understand how the writer is building the argument. Paragraphs use coherence devices effectively while focusing on a single main idea.
Language: The writing displays sophisticated control of a wide range of vocabulary and collocations. The essay follows grammar and usage rules throughout the paper. Spelling and punctuation are correct throughout the paper.

The essays in that dataset are scored on a range from 1 to 5, in increments of 0.5 for each criterion.

16.1 First prompt

The first step is to write an initial prompt to score the essays. I very rarely use an initial prompt without asking an LLM to review it for completeness and organization. To demonstrate this first step, I purposefully have written a vague prompt that I wouldn’t use in practice so we can see what type of guidance a model might provide for us. You can use this prompt, or you can review it and improve it on your own before providing it to a model. I’ve also included the outer prompt that I would use when asking an LLM for feedback on a prompt. That outer prompt is the metaprompt: it tells the model how to review the scoring prompt, what kinds of weaknesses to look for, and what kind of revised prompt to return.

Code

score_rubric_prompt <- "You are an expert essay grader. Score the following student essay based on three criteria: Content, Organization, and Language. Each criterion should be scored from 1 to 5 in increments of 0.5 (e.g., 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5). The prompt for this essay was 'Do you think that smartphones have destroyed communication among family and friends? Give specific reasons and details to support your opinion?' 

RUBRIC:

**Content (1-5):**
Paragraph is well-developed and relevant to the argument, supported with strong reasons and examples.

**Organization (1-5):**
The argument is very effectively structured and developed, making it easy for the reader to follow the ideas and understand how the writer is building the argument. Paragraphs use coherence devices effectively while focusing on a single main idea.
 
**Language (1-5):**
The writing displays sophisticated control of a wide range of vocabulary and collocations. The essay follows grammar and usage rules throughout the paper. Spelling and punctuation are correct throughout the paper.

ESSAY: {essay}

Please provide scores for each criterion and calculate the total score. Format your response as follows:
Content: [score]
Organization: [score]
Language: [score]
Total: [sum of three scores]

Provide a brief justification (1-2 sentences) for each score."

prompt_improvement <- function(scoring_prompt){
  glue::glue("
     You are reviewing a prompt for an LLM-assisted essay-scoring workflow.

     Review the draft below and respond concisely:
     1. Identify the 3-5 most important ambiguities or missing instructions that could reduce consistent application of the existing rubric.
     2. Recommend a clearer structure while preserving the existing criteria and 1-5 scoring scale in 0.5-point increments.
     3. Provide a revised prompt wrapped in an R function named `score_essay(essay)` that uses `glue` and inserts the essay at the appropriate location.
     4. Require the model to return only a valid JSON object with separate scores and brief justifications for Content, Organization, and Language, plus a total score.

     Do not score an essay or invent new rubric criteria. Clearly label any decisions the author still needs to make, such as word-count rules or edge-case handling.

     Draft prompt:
     {scoring_prompt}
  ")
}

Loading call_claude to send the prompt.

Code

if (Sys.getenv("ANTHROPIC_API_KEY") == "") {
     Sys.setenv(ANTHROPIC_API_KEY = rstudioapi::askForSecret("Anthropic API Key"))
}

source("downloads/call_claude_AIMECON26.R")

Asking for feedback on the prompt.

Code

revised_prompt <- call_claude(prompt_improvement(score_rubric_prompt))
cat(revised_prompt)

Expand the LLM Response

I’ll work through each task systematically.

Task 1: Areas Where More Detail Would Improve Rubric Application

Content rubric is vague - “well-developed” and “strong reasons” need anchor descriptions for each score level (1, 2, 3, 4, 5). Without this, the LLM has no basis for distinguishing a 2 from a 3.
Organization rubric doesn’t clarify expectations for essay-level structure (intro, body, conclusion) vs. paragraph-level structure.
Language rubric conflates three distinct sub-dimensions (vocabulary, grammar, mechanics) making it hard to arrive at a single score when they conflict (e.g., rich vocabulary but poor spelling).
No anchor examples are provided for any score level on any criterion.
No guidance on how to handle very short essays or off-topic responses.
Scoring range behavior is undefined at the extremes — what does a “1” look like vs. a “2”?

Task 2: Structural Issues with the Prompt

The rubric criteria run together visually — the LLM may not cleanly separate scoring logic for each.
The output format instruction is at the bottom but JSON was not requested (addressed in Task 3).
Instructions for justification length are present but placement after the format block is easy to miss.
The essay prompt context is buried mid-paragraph rather than clearly labeled.

Tasks 3 & 4: Updated Prompt Wrapped in glue Function

library(glue)

score_essay <- function(essay) {
  
  prompt <- glue(
    "You are an expert essay grader with experience in evaluating argumentative writing
    at the secondary and post-secondary level.
    
    ## TASK
    Score the student essay provided below using the three criteria in the rubric.
    Each criterion is scored from 1 to 5 in increments of 0.5 (1, 1.5, 2, 2.5, 3,
    3.5, 4, 4.5, 5). Apply each criterion independently before calculating the total.
    
    ## ESSAY PROMPT GIVEN TO STUDENT
    'Do you think that smartphones have destroyed communication among family and friends?
    Give specific reasons and details to support your opinion.'
    
    ## SCORING RUBRIC
    
    ### CRITERION 1: Content (1-5)
    Score the degree to which the essay develops a clear argument with relevant reasons
    and specific supporting examples in response to the essay prompt.
    
    - 5: Argument is thoroughly developed with multiple specific, relevant reasons and
         concrete examples. All content directly supports the central claim.
    - 4: Argument is well-developed with mostly specific reasons and examples. Minor
         gaps in development or relevance.
    - 3: Argument is adequately developed but reasons are sometimes general or examples
         are vague. Some content may be loosely connected to the argument.
    - 2: Argument is underdeveloped. Reasons are largely general or repetitive and
         examples are missing or unclear.
    - 1: Little to no recognizable argument. Content is largely irrelevant, missing,
         or does not respond to the prompt.
    
    Scores between these anchors (e.g., 1.5, 2.5) should be used when the essay falls
    between two descriptors.
    
    ### CRITERION 2: Organization (1-5)
    Score the degree to which the essay is logically structured at both the essay level
    (introduction, body, conclusion) and the paragraph level (topic sentences, coherence
    devices, single main idea per paragraph).
    
    - 5: Essay has a clear introduction with a thesis, well-organized body paragraphs
         each focused on a single idea, and a conclusion. Transitions and coherence
         devices are used effectively throughout.
    - 4: Essay structure is clear and mostly effective. Minor issues with transitions
         or paragraph focus.
    - 3: Basic structure is present but inconsistently applied. Some paragraphs may
         lack focus or transitions may be absent or mechanical.
    - 2: Structure is difficult to follow. Paragraphs may lack topic sentences or blend
         multiple unrelated ideas. Few or no transitions.
    - 1: No discernible organizational structure. Ideas are presented randomly with no
         paragraph logic.
    
    ### CRITERION 3: Language (1-5)
    Score the overall quality of language use across three equally weighted
    sub-dimensions: (a) vocabulary range and accuracy, (b) grammar and usage
    correctness, and (c) spelling and punctuation accuracy. Average across these three
    sub-dimensions to arrive at a single Language score.
    
    - 5: (a) Sophisticated and varied vocabulary with accurate collocations; (b) grammar
         and usage are correct throughout; (c) spelling and punctuation are correct
         throughout.
    - 4: (a) Good vocabulary range with occasional imprecision; (b) mostly correct
         grammar with minor errors that do not impede meaning; (c) few spelling or
         punctuation errors.
    - 3: (a) Adequate but limited vocabulary, some word choice errors; (b) grammar
         errors are noticeable but meaning is generally clear; (c) some spelling and
         punctuation errors.
    - 2: (a) Narrow or frequently inaccurate vocabulary; (b) grammar errors are frequent
         and sometimes obscure meaning; (c) spelling and punctuation errors are frequent.
    - 1: (a) Very limited vocabulary with pervasive errors; (b) grammar errors throughout
         severely impede meaning; (c) spelling and punctuation errors throughout.
    
    ## HANDLING EDGE CASES
    - If the essay is off-topic or does not respond to the prompt, assign a Content
      score of 1 and note this in the justification.
    - If the essay is fewer than 50 words, assign a maximum score of 2 for Content
      and Organization, and score Language based on what is present.
      [NOTE: Adjust this word threshold based on your assignment expectations.]
    
    ## STUDENT ESSAY TO SCORE
    {essay}
    
    ## OUTPUT INSTRUCTIONS
    Return your response ONLY as a valid JSON object with no additional text before or
    after it. Use the following structure exactly:
    
    {{
      'content_score': <number>,
      'content_justification': '<1-2 sentence justification>',
      'organization_score': <number>,
      'organization_justification': '<1-2 sentence justification>',
      'language_score': <number>,
      'language_justification': '<1-2 sentence justification referencing vocabulary,
                                  grammar, and mechanics>',
      'total_score': <sum of three scores>
    }}
    
    Ensure that 'total_score' equals the sum of the three criterion scores. Do not
    include any text outside the JSON object."
  )
  
  return(prompt)
  
}

Example Usage

# Single essay
my_essay <- "Smartphones have definitely changed how families communicate..."
prompt_ready <- score_essay(essay = my_essay)

# Cycling through a data frame of essays
library(dplyr)
library(purrr)

essays_df <- tibble(
  student_id = c("S001", "S002", "S003"),
  essay_text = c(
    "Smartphones have changed communication...",
    "I think phones are bad because...",
    "Families used to talk more before smartphones..."
  )
)

essays_df <- essays_df |>
  mutate(
    prompt = map_chr(essay_text, ~ score_essay(essay = .x))
  )

# essays_df$prompt can then be passed to your LLM API call

Key Notes on Where You Should Add More Detail

Location	What to Add
Content rubric	Domain-specific examples of what counts as a “specific example” for this prompt
Word count threshold	Adjust the 50-word minimum to match your assignment requirements
Language sub-dimension weights	Change equal weighting if mechanics matter more/less than grammar in your context
Edge case handling	Add rules for plagiarism flags, non-English responses, or AI-generated text if relevant
API call	You will need to add your own LLM API call function to pass `prompt_ready` to the model and parse the returned JSON

16.2 What Makes This Metaprompting?

In this example, score_rubric_prompt is the prompt we eventually want to use for essay scoring. prompt_improvement() is the metaprompt because it asks the model to inspect that scoring prompt before we use it in the workflow. The model is being used as a reviewer of instructions rather than as the essay scorer.

The reusable pattern is:

Draft the task prompt.
Ask the model to critique the prompt for ambiguity, missing context, edge cases, and output-format problems.
Ask for a revised version that preserves the important constraints.
Review the model’s suggestions as a human expert.
Test the revised prompt on several examples.
Repeat until the prompt behaves consistently enough for the intended workflow.

A useful metaprompt checklist

A good metaprompt usually tells the model to:

identify the downstream task and intended use of the prompt;
preserve non-negotiable criteria, rubric language, scoring scales, and policy constraints;
look for ambiguity, missing context, edge cases, and output-format risks;
suggest concrete revisions rather than only giving general advice;
label decisions that still require a human expert; and
avoid inventing new criteria or rules unless explicitly asked.

16.3 Implement Revisions

As you can see, the LLM identified many areas for improvement - many of which you may have identified yourself as measurement experts!

For the purposes of this workshop, let’s blindly adopt the changes the LLM provided. This is normally something I wouldn’t do; instead, I would review the suggested edits, keep what was good, and continue refining the prompt based on ideas I received from the initial LLM feedback. I would usually do this a few more times, even if I’m working with subject-matter experts. I’ve often found that SMEs are impressed by some of the suggestions made by the LLM, and they also like it as a brainstorming partner.

Code

# Packages
library(dplyr)


Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

Code

library(purrr)
library(glue)

# Download the data
load(url("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/student_essays.Rdata"))

score_essay <- function(essay) {
  
  prompt <- glue(
    "You are an expert essay grader with experience in evaluating argumentative writing
    at the secondary and post-secondary level.
    
    ## TASK
    Score the student essay provided below using the three criteria in the rubric.
    Each criterion is scored from 1 to 5 in increments of 0.5 (1, 1.5, 2, 2.5, 3,
    3.5, 4, 4.5, 5). Apply each criterion independently before calculating the total.
    
    ## ESSAY PROMPT GIVEN TO STUDENT
    'Do you think that smartphones have destroyed communication among family and friends?
    Give specific reasons and details to support your opinion.'
    
    ## SCORING RUBRIC
    
    ### CRITERION 1: Content (1-5)
    Score the degree to which the essay develops a clear argument with relevant reasons
    and specific supporting examples in response to the essay prompt.
    
    - 5: Argument is thoroughly developed with multiple specific, relevant reasons and
         concrete examples. All content directly supports the central claim.
    - 4: Argument is well-developed with mostly specific reasons and examples. Minor
         gaps in development or relevance.
    - 3: Argument is adequately developed but reasons are sometimes general or examples
         are vague. Some content may be loosely connected to the argument.
    - 2: Argument is underdeveloped. Reasons are largely general or repetitive and
         examples are missing or unclear.
    - 1: Little to no recognizable argument. Content is largely irrelevant, missing,
         or does not respond to the prompt.
    
    Scores between these anchors (e.g., 1.5, 2.5) should be used when the essay falls
    between two descriptors.
    
    ### CRITERION 2: Organization (1-5)
    Score the degree to which the essay is logically structured at both the essay level
    (introduction, body, conclusion) and the paragraph level (topic sentences, coherence
    devices, single main idea per paragraph).
    
    - 5: Essay has a clear introduction with a thesis, well-organized body paragraphs
         each focused on a single idea, and a conclusion. Transitions and coherence
         devices are used effectively throughout.
    - 4: Essay structure is clear and mostly effective. Minor issues with transitions
         or paragraph focus.
    - 3: Basic structure is present but inconsistently applied. Some paragraphs may
         lack focus or transitions may be absent or mechanical.
    - 2: Structure is difficult to follow. Paragraphs may lack topic sentences or blend
         multiple unrelated ideas. Few or no transitions.
    - 1: No discernible organizational structure. Ideas are presented randomly with no
         paragraph logic.
    
    ### CRITERION 3: Language (1-5)
    Score the overall quality of language use across three equally weighted
    sub-dimensions: (a) vocabulary range and accuracy, (b) grammar and usage
    correctness, and (c) spelling and punctuation accuracy. Average across these three
    sub-dimensions to arrive at a single Language score.
    
    - 5: (a) Sophisticated and varied vocabulary with accurate collocations; (b) grammar
         and usage are correct throughout; (c) spelling and punctuation are correct
         throughout.
    - 4: (a) Good vocabulary range with occasional imprecision; (b) mostly correct
         grammar with minor errors that do not impede meaning; (c) few spelling or
         punctuation errors.
    - 3: (a) Adequate but limited vocabulary, some word choice errors; (b) grammar
         errors are noticeable but meaning is generally clear; (c) some spelling and
         punctuation errors.
    - 2: (a) Narrow or frequently inaccurate vocabulary; (b) grammar errors are frequent
         and sometimes obscure meaning; (c) spelling and punctuation errors are frequent.
    - 1: (a) Very limited vocabulary with pervasive errors; (b) grammar errors throughout
         severely impede meaning; (c) spelling and punctuation errors throughout.
    
    ## HANDLING EDGE CASES
    - If the essay is off-topic or does not respond to the prompt, assign a Content
      score of 1 and note this in the justification.
    - If the essay is fewer than 50 words, assign a maximum score of 2 for Content
      and Organization, and score Language based on what is present.
      [NOTE: Adjust this word threshold based on your assignment expectations.]
    
    ## STUDENT ESSAY TO SCORE
    {essay}
    
    ## OUTPUT INSTRUCTIONS
    Return your response ONLY as a valid JSON object with no additional text before or
    after it. Use the following structure exactly:
    
    {{
     \"content_score\": <number>,
     \"content_justification\": \"<1-2 sentence justification>\",
     \"organization_score\": <number>,
     \"organization_justification\": \"<1-2 sentence justification>\",
     \"language_score\": <number>,
     \"language_justification\": \"<1-2 sentence justification referencing vocabulary,
                                  grammar, and mechanics>\",
     \"total_score\": <sum of three scores>
    }}
    
     Ensure that \"total_score\" equals the sum of the three criterion scores. Do not
    include any text outside the JSON object."
  )
  
  return(prompt)
  
}

# Creating new column of prompts to keep it in the same df
student_essays$scoring_prompt <- sapply(student_essays$essay, score_essay)

# Trying the first one
student_essays$scoring_output <- NA
student_essays$scoring_output[1] <- call_claude(student_essays$scoring_prompt[1])

cat(student_essays$scoring_output[1])

{
"content_score": 2.5,
"content_justification": "The essay presents a clear opinion with two relevant examples (family dinner, meeting friends) but development is limited and repetitive without deeper analysis.",
"organization_score": 2.5,
"organization_justification": "There is a basic structure with an opinion and examples, but no clear introduction or conclusion, and ideas blend together without transitions.",
"language_score": 2,
"language_justification": "Vocabulary is narrow and repetitive, grammar errors are frequent (e.g., 'everyone looking at phone', 'my dad watch the news'), though meaning is still generally clear with minor punctuation issues.",
"total_score": 7
}

# Activity: Iterative Prompt Development Download the notebook for this activity: <a href="downloads/activity-iterative-prompt-development.Rmd" download style="display: inline-block; padding: 10px 20px; background-color: #4A90E2; color: black; text-decoration: none; border-radius: 5px;"> 📥 Download `activity-iterative-prompt-development.Rmd` </a> Now it's time to practice the first element of an LLM workflow: iterative prompt development. This is the process of developing and tweaking a prompt until it's ready to be implemented more widely in your workflow. One strategy we'll use here is **metaprompting**: asking an LLM to review and improve a prompt before that prompt is used for the real task. In other words, the model is not scoring essays yet; it is helping us improve the instructions we will later use for scoring. For the purposes of this activity, I made some synthetic essay responses with Claude Sonnet 4.5 based on the [`Dataset for Rubric-based Essay Scoring (DREsS)`](https://haneul-yoo.github.io/dress/) data, which is based on essays written by English as a foreign language learners. I used responses to the essay prompt, "Do you think that smartphones have destroyed communication among family and friends? Give specific reasons and details to support your opinion?" when creating the 10 synthetic essays. The full dataset is available after filling out a [consent form](https://docs.google.com/forms/d/e/1FAIpQLSdqlywEiCl5Ddei7T7ujMBpMHFDLRyW7OBo033e_Oe-amGqmQ/viewform). The following rubric was developed for their analysis: * **Content:** Paragraph is well-developed and relevant to the argument, supported with strong reasons and examples. * **Organization:** The argument is very effectively structured and developed, making it easy for the reader to follow the ideas and understand how the writer is building the argument. Paragraphs use coherence devices effectively while focusing on a single main idea. * **Language:** The writing displays sophisticated control of a wide range of vocabulary and collocations. The essay follows grammar and usage rules throughout the paper. Spelling and punctuation are correct throughout the paper. The essays in that dataset are scored on a range from 1 to 5, in increments of 0.5 for _each_ criterion. ## First prompt The first step is to write an initial prompt to score the essays. I _very_ rarely use an initial prompt without asking an LLM to review it for completeness and organization. To demonstrate this first step, I purposefully have written a vague prompt that I wouldn't use in practice so we can see what type of guidance a model might provide for us. You can use this prompt, or you can review it and improve it on your own before providing it to a model. I've also included the outer prompt that I would use when asking an LLM for feedback on a prompt. That outer prompt is the metaprompt: it tells the model how to review the scoring prompt, what kinds of weaknesses to look for, and what kind of revised prompt to return. ```{r, eval = FALSE} score_rubric_prompt <- "You are an expert essay grader. Score the following student essay based on three criteria: Content, Organization, and Language. Each criterion should be scored from 1 to 5 in increments of 0.5 (e.g., 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5). The prompt for this essay was 'Do you think that smartphones have destroyed communication among family and friends? Give specific reasons and details to support your opinion?' RUBRIC: **Content (1-5):** Paragraph is well-developed and relevant to the argument, supported with strong reasons and examples. **Organization (1-5):** The argument is very effectively structured and developed, making it easy for the reader to follow the ideas and understand how the writer is building the argument. Paragraphs use coherence devices effectively while focusing on a single main idea. **Language (1-5):** The writing displays sophisticated control of a wide range of vocabulary and collocations. The essay follows grammar and usage rules throughout the paper. Spelling and punctuation are correct throughout the paper. ESSAY: {essay} Please provide scores for each criterion and calculate the total score. Format your response as follows: Content: [score] Organization: [score] Language: [score] Total: [sum of three scores] Provide a brief justification (1-2 sentences) for each score." prompt_improvement <- function(scoring_prompt){ glue::glue(" You are reviewing a prompt for an LLM-assisted essay-scoring workflow. Review the draft below and respond concisely: 1. Identify the 3-5 most important ambiguities or missing instructions that could reduce consistent application of the existing rubric. 2. Recommend a clearer structure while preserving the existing criteria and 1-5 scoring scale in 0.5-point increments. 3. Provide a revised prompt wrapped in an R function named `score_essay(essay)` that uses `glue` and inserts the essay at the appropriate location. 4. Require the model to return only a valid JSON object with separate scores and brief justifications for Content, Organization, and Language, plus a total score. Do not score an essay or invent new rubric criteria. Clearly label any decisions the author still needs to make, such as word-count rules or edge-case handling. Draft prompt: {scoring_prompt} ") } ``` Loading `call_claude` to send the prompt. ```{r call_claude} if (Sys.getenv("ANTHROPIC_API_KEY") == "") { Sys.setenv(ANTHROPIC_API_KEY = rstudioapi::askForSecret("Anthropic API Key")) } source("downloads/call_claude_AIMECON26.R") ``` Asking for feedback on the prompt. ```{r, eval = FALSE} revised_prompt <- call_claude(prompt_improvement(score_rubric_prompt)) cat(revised_prompt) ``` ::: {.callout-note collapse="true"} ## Expand the LLM Response I'll work through each task systematically. **Task 1: Areas Where More Detail Would Improve Rubric Application** 1. **Content rubric** is vague - "well-developed" and "strong reasons" need anchor descriptions for each score level (1, 2, 3, 4, 5). Without this, the LLM has no basis for distinguishing a 2 from a 3. 2. **Organization rubric** doesn't clarify expectations for essay-level structure (intro, body, conclusion) vs. paragraph-level structure. 3. **Language rubric** conflates three distinct sub-dimensions (vocabulary, grammar, mechanics) making it hard to arrive at a single score when they conflict (e.g., rich vocabulary but poor spelling). 4. **No anchor examples** are provided for any score level on any criterion. 5. **No guidance on how to handle very short essays** or off-topic responses. 6. **Scoring range behavior** is undefined at the extremes — what does a "1" look like vs. a "2"? --- **Task 2: Structural Issues with the Prompt** 1. The rubric criteria run together visually — the LLM may not cleanly separate scoring logic for each. 2. The output format instruction is at the bottom but JSON was not requested (addressed in Task 3). 3. Instructions for justification length are present but placement after the format block is easy to miss. 4. The essay prompt context is buried mid-paragraph rather than clearly labeled. --- **Tasks 3 & 4: Updated Prompt Wrapped in `glue` Function** ``` library(glue) score_essay <- function(essay) { prompt <- glue( "You are an expert essay grader with experience in evaluating argumentative writing at the secondary and post-secondary level. ## TASK Score the student essay provided below using the three criteria in the rubric. Each criterion is scored from 1 to 5 in increments of 0.5 (1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5). Apply each criterion independently before calculating the total. ## ESSAY PROMPT GIVEN TO STUDENT 'Do you think that smartphones have destroyed communication among family and friends? Give specific reasons and details to support your opinion.' ## SCORING RUBRIC ### CRITERION 1: Content (1-5) Score the degree to which the essay develops a clear argument with relevant reasons and specific supporting examples in response to the essay prompt. - 5: Argument is thoroughly developed with multiple specific, relevant reasons and concrete examples. All content directly supports the central claim. - 4: Argument is well-developed with mostly specific reasons and examples. Minor gaps in development or relevance. - 3: Argument is adequately developed but reasons are sometimes general or examples are vague. Some content may be loosely connected to the argument. - 2: Argument is underdeveloped. Reasons are largely general or repetitive and examples are missing or unclear. - 1: Little to no recognizable argument. Content is largely irrelevant, missing, or does not respond to the prompt. Scores between these anchors (e.g., 1.5, 2.5) should be used when the essay falls between two descriptors. ### CRITERION 2: Organization (1-5) Score the degree to which the essay is logically structured at both the essay level (introduction, body, conclusion) and the paragraph level (topic sentences, coherence devices, single main idea per paragraph). - 5: Essay has a clear introduction with a thesis, well-organized body paragraphs each focused on a single idea, and a conclusion. Transitions and coherence devices are used effectively throughout. - 4: Essay structure is clear and mostly effective. Minor issues with transitions or paragraph focus. - 3: Basic structure is present but inconsistently applied. Some paragraphs may lack focus or transitions may be absent or mechanical. - 2: Structure is difficult to follow. Paragraphs may lack topic sentences or blend multiple unrelated ideas. Few or no transitions. - 1: No discernible organizational structure. Ideas are presented randomly with no paragraph logic. ### CRITERION 3: Language (1-5) Score the overall quality of language use across three equally weighted sub-dimensions: (a) vocabulary range and accuracy, (b) grammar and usage correctness, and (c) spelling and punctuation accuracy. Average across these three sub-dimensions to arrive at a single Language score. - 5: (a) Sophisticated and varied vocabulary with accurate collocations; (b) grammar and usage are correct throughout; (c) spelling and punctuation are correct throughout. - 4: (a) Good vocabulary range with occasional imprecision; (b) mostly correct grammar with minor errors that do not impede meaning; (c) few spelling or punctuation errors. - 3: (a) Adequate but limited vocabulary, some word choice errors; (b) grammar errors are noticeable but meaning is generally clear; (c) some spelling and punctuation errors. - 2: (a) Narrow or frequently inaccurate vocabulary; (b) grammar errors are frequent and sometimes obscure meaning; (c) spelling and punctuation errors are frequent. - 1: (a) Very limited vocabulary with pervasive errors; (b) grammar errors throughout severely impede meaning; (c) spelling and punctuation errors throughout. ## HANDLING EDGE CASES - If the essay is off-topic or does not respond to the prompt, assign a Content score of 1 and note this in the justification. - If the essay is fewer than 50 words, assign a maximum score of 2 for Content and Organization, and score Language based on what is present. [NOTE: Adjust this word threshold based on your assignment expectations.] ## STUDENT ESSAY TO SCORE {essay} ## OUTPUT INSTRUCTIONS Return your response ONLY as a valid JSON object with no additional text before or after it. Use the following structure exactly: {{ 'content_score': <number>, 'content_justification': '<1-2 sentence justification>', 'organization_score': <number>, 'organization_justification': '<1-2 sentence justification>', 'language_score': <number>, 'language_justification': '<1-2 sentence justification referencing vocabulary, grammar, and mechanics>', 'total_score': <sum of three scores> }} Ensure that 'total_score' equals the sum of the three criterion scores. Do not include any text outside the JSON object." ) return(prompt) } ``` Example Usage ``` # Single essay my_essay <- "Smartphones have definitely changed how families communicate..." prompt_ready <- score_essay(essay = my_essay) # Cycling through a data frame of essays library(dplyr) library(purrr) essays_df <- tibble( student_id = c("S001", "S002", "S003"), essay_text = c( "Smartphones have changed communication...", "I think phones are bad because...", "Families used to talk more before smartphones..." ) ) essays_df <- essays_df |> mutate( prompt = map_chr(essay_text, ~ score_essay(essay = .x)) ) # essays_df$prompt can then be passed to your LLM API call ``` --- **Key Notes on Where You Should Add More Detail** | Location | What to Add | |---|---| | Content rubric | Domain-specific examples of what counts as a "specific example" for this prompt | | Word count threshold | Adjust the 50-word minimum to match your assignment requirements | | Language sub-dimension weights | Change equal weighting if mechanics matter more/less than grammar in your context | | Edge case handling | Add rules for plagiarism flags, non-English responses, or AI-generated text if relevant | | API call | You will need to add your own LLM API call function to pass `prompt_ready` to the model and parse the returned JSON | ::: ## What Makes This Metaprompting? In this example, `score_rubric_prompt` is the prompt we eventually want to use for essay scoring. `prompt_improvement()` is the metaprompt because it asks the model to inspect that scoring prompt before we use it in the workflow. The model is being used as a reviewer of instructions rather than as the essay scorer. The reusable pattern is: 1. Draft the task prompt. 2. Ask the model to critique the prompt for ambiguity, missing context, edge cases, and output-format problems. 3. Ask for a revised version that preserves the important constraints. 4. Review the model's suggestions as a human expert. 5. Test the revised prompt on several examples. 6. Repeat until the prompt behaves consistently enough for the intended workflow. ::: {.callout-tip} ## A useful metaprompt checklist A good metaprompt usually tells the model to: - identify the downstream task and intended use of the prompt; - preserve non-negotiable criteria, rubric language, scoring scales, and policy constraints; - look for ambiguity, missing context, edge cases, and output-format risks; - suggest concrete revisions rather than only giving general advice; - label decisions that still require a human expert; and - avoid inventing new criteria or rules unless explicitly asked. ::: ## Implement Revisions As you can see, the LLM identified many areas for improvement - many of which you may have identified yourself as measurement experts! For the purposes of this workshop, let's blindly adopt the changes the LLM provided. This is normally something I wouldn't do; instead, I would review the suggested edits, keep what was good, and continue refining the prompt based on ideas I received from the initial LLM feedback. I would usually do this a few more times, even if I'm working with subject-matter experts. I've often found that SMEs are impressed by some of the suggestions made by the LLM, and they also like it as a brainstorming partner. ```{r} # Packages library(dplyr) library(purrr) library(glue) # Download the data load(url("https://raw.githubusercontent.com/runyoncr/AIMECON_R_WORKSHOP/main/data/student_essays.Rdata")) score_essay <- function(essay) { prompt <- glue( "You are an expert essay grader with experience in evaluating argumentative writing at the secondary and post-secondary level. ## TASK Score the student essay provided below using the three criteria in the rubric. Each criterion is scored from 1 to 5 in increments of 0.5 (1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5). Apply each criterion independently before calculating the total. ## ESSAY PROMPT GIVEN TO STUDENT 'Do you think that smartphones have destroyed communication among family and friends? Give specific reasons and details to support your opinion.' ## SCORING RUBRIC ### CRITERION 1: Content (1-5) Score the degree to which the essay develops a clear argument with relevant reasons and specific supporting examples in response to the essay prompt. - 5: Argument is thoroughly developed with multiple specific, relevant reasons and concrete examples. All content directly supports the central claim. - 4: Argument is well-developed with mostly specific reasons and examples. Minor gaps in development or relevance. - 3: Argument is adequately developed but reasons are sometimes general or examples are vague. Some content may be loosely connected to the argument. - 2: Argument is underdeveloped. Reasons are largely general or repetitive and examples are missing or unclear. - 1: Little to no recognizable argument. Content is largely irrelevant, missing, or does not respond to the prompt. Scores between these anchors (e.g., 1.5, 2.5) should be used when the essay falls between two descriptors. ### CRITERION 2: Organization (1-5) Score the degree to which the essay is logically structured at both the essay level (introduction, body, conclusion) and the paragraph level (topic sentences, coherence devices, single main idea per paragraph). - 5: Essay has a clear introduction with a thesis, well-organized body paragraphs each focused on a single idea, and a conclusion. Transitions and coherence devices are used effectively throughout. - 4: Essay structure is clear and mostly effective. Minor issues with transitions or paragraph focus. - 3: Basic structure is present but inconsistently applied. Some paragraphs may lack focus or transitions may be absent or mechanical. - 2: Structure is difficult to follow. Paragraphs may lack topic sentences or blend multiple unrelated ideas. Few or no transitions. - 1: No discernible organizational structure. Ideas are presented randomly with no paragraph logic. ### CRITERION 3: Language (1-5) Score the overall quality of language use across three equally weighted sub-dimensions: (a) vocabulary range and accuracy, (b) grammar and usage correctness, and (c) spelling and punctuation accuracy. Average across these three sub-dimensions to arrive at a single Language score. - 5: (a) Sophisticated and varied vocabulary with accurate collocations; (b) grammar and usage are correct throughout; (c) spelling and punctuation are correct throughout. - 4: (a) Good vocabulary range with occasional imprecision; (b) mostly correct grammar with minor errors that do not impede meaning; (c) few spelling or punctuation errors. - 3: (a) Adequate but limited vocabulary, some word choice errors; (b) grammar errors are noticeable but meaning is generally clear; (c) some spelling and punctuation errors. - 2: (a) Narrow or frequently inaccurate vocabulary; (b) grammar errors are frequent and sometimes obscure meaning; (c) spelling and punctuation errors are frequent. - 1: (a) Very limited vocabulary with pervasive errors; (b) grammar errors throughout severely impede meaning; (c) spelling and punctuation errors throughout. ## HANDLING EDGE CASES - If the essay is off-topic or does not respond to the prompt, assign a Content score of 1 and note this in the justification. - If the essay is fewer than 50 words, assign a maximum score of 2 for Content and Organization, and score Language based on what is present. [NOTE: Adjust this word threshold based on your assignment expectations.] ## STUDENT ESSAY TO SCORE {essay} ## OUTPUT INSTRUCTIONS Return your response ONLY as a valid JSON object with no additional text before or after it. Use the following structure exactly: {{ \"content_score\": <number>, \"content_justification\": \"<1-2 sentence justification>\", \"organization_score\": <number>, \"organization_justification\": \"<1-2 sentence justification>\", \"language_score\": <number>, \"language_justification\": \"<1-2 sentence justification referencing vocabulary, grammar, and mechanics>\", \"total_score\": <sum of three scores> }} Ensure that \"total_score\" equals the sum of the three criterion scores. Do not include any text outside the JSON object." ) return(prompt) } # Creating new column of prompts to keep it in the same df student_essays$scoring_prompt <- sapply(student_essays$essay, score_essay) # Trying the first one student_essays$scoring_output <- NA student_essays$scoring_output[1] <- call_claude(student_essays$scoring_prompt[1]) cat(student_essays$scoring_output[1]) ```