How to Measure an AI Writing Pilot for a PR Team

The demo drafted a release in under a minute. Two months into the pilot, the editors say they are busier than before, the vendor says drafting time fell, and both can be right, because the pilot only timed the draft. Nobody counted the checking, the rewritten quote or the search for the source page.

Measure the effort to reach an approvable draft, not the time to a first one. Time the whole task against your current process, count every unsupported claim the tool produced even when an editor caught it, and agree before the results arrive what outcome would justify continuing. Editing is where the time goes: Muck Rack's State of AI in PR 2026↗ found 98% of PR pros always or often edit AI output, and 45% still have to edit most of the text¹.

A first draft arrives quickly. Then the editor checks its claims, replaces a borrowed-sounding executive quote, finds the right source page and sends the revision for approval. Where does your pilot's stopwatch stop? That choice determines whether you are measuring text production or the effort required to produce something the team can authorize.

For a CMO or communications lead, begin with the decision the pilot should inform. An example is whether to allow assisted first drafts of routine product announcements using approved source packs. Define the permitted task, the people responsible for review and the conditions that would justify continuing. Broader use should require evidence from the broader workflow. If the team is still choosing what to test, a comparison of AI tools for PR professionals sets out the main options before you commit to one.

The formats your team actually writes, matched against the current process, including the hard cases where tools tend to fail.

Select representative briefs across the formats you intend to evaluate. A press release, an authored article and a journalist Q&A ask different things of a writer. Report them separately. Include straightforward assignments and deliberately constructed difficult cases, such as conflicting source periods, missing evidence or an unapproved quote; a challenge set built from the quote assignments your team finds hard is a good place to start. Label those cases as exercises rather than genuine client work.

Give the baseline and assisted workflow equivalent task instructions, factual material, approved person and organization voice examples, and publication constraints. Use only material authorized for the tools and people involved. Where possible, compare matched assignments and vary the order to reduce the advantage of doing a familiar brief twice. Record remaining differences in writer experience or task complexity. HM Treasury's guidance on evaluating AI interventions↗ makes the same point for government work: when comparing an AI intervention with business as usual, document precisely what business as usual consists of before the AI is introduced².

Freeze the brief and source versions for each comparison. Record the tool and model version if available, settings, relevant instructions, retries and reviewer interventions. Keep the original output as well as the repaired draft. If a result depends on extensive prompt coaching or manually supplied facts, that work belongs in the pilot record.

Four things: factual support, voice and brief fit, total human effort including repair, and readiness for approval. Drafting speed alone is the least informative number.

One ledger row per assignment, with attempts, repairs, active minutes, elapsed time and the review state it reached.

Use one record per assignment: format, difficulty, source set, workflow, attempt count, active effort by stage, elapsed time, assertion count, unsupported assertions, restricted disclosures, missing requirements, voice revisions and final review state. Keep rejected and unfinished assignments in the record. Define the denominator for every rate, and avoid combining unlike categories into a single quality score.

For a purely illustrative calculation, suppose a manual workflow uses 30 active minutes to draft and 20 to review. An assisted workflow uses 10 to prepare inputs, five to draft and 40 to review and repair. The totals are 50 and 55 minutes. These invented values are arithmetic examples, not benchmarks or observed performance. They show why measuring drafting alone could produce a different conclusion.

Review disagreements are useful evidence too. Ask a second reviewer to inspect disputed claims with the source context, then record the resolution and adjust unclear rubric wording. Keep human decisions separate from automated scores. NIST's profile also recommends reviewing and verifying sources and citations in generated outputs, and cautions against extrapolating a system's performance from narrow, non-systematic and anecdotal assessments³.

Because much of a draft's elapsed time can sit in waiting for owners rather than in writing, and only a log shows which.

Totals show how much effort a task took. A small record of interruptions and handoffs shows where it went. The purpose is diagnostic: locate the step that creates avoidable searching, repeated checking or waiting, without asking participants to narrate every minute of their day. This is close to what HM Treasury's Magenta Book↗ calls process evaluation, which examines the activities involved in implementing an intervention and asks questions such as what worked well and less well, and why⁴.

Create an entry when someone cannot proceed, repeats a completed action or transfers a question to another owner. Record the assignment, stage, time, person, trigger and next action. Use plain reasons such as source inaccessible, current version unclear, fact owner unknown or claim wording changed, and leave room for an explanation the categories do not capture. Attach the smallest useful evidence, such as a version reference, rather than copying restricted source content into a widely shared log.

Track active effort separately from elapsed waiting. If two people each spend ten minutes checking in parallel, that is twenty person-minutes within ten elapsed minutes. If a draft waits for two owners at once, the overlapping waits are not added into a longer delay. Keep the event times so another reviewer can reconstruct the sequence.

Run it as the assisted workflow against your current process, on the same briefs and documents, and score it with the same ledger as any other tool.

QuoteIt drafts from the documents you upload and places each claim beside the passage it came from, flagging the claims no passage supports. That makes the unsupported-claims column quicker to fill and easier to dispute, because the reviewer sees the evidence the tool relied on. Approvals are recorded against the version they covered, which gives you the handoff state without a separate log. Count anything it misses as a failure, exactly as you would for a general assistant; the Claude alternative for PR professionals and the ChatGPT alternative describe what the general assistants leave to the editor.

Your colleagues have already said what they need before they trust these tools. In the same Muck Rack survey, 89% wanted clear human approval before anything is published or sent, and 60% wanted transparent records of what the tool did and why¹. Put both on the scorecard, and keep a decision log the next editor can follow.

Against criteria agreed before the results come in, set with the accountable communications owner.

You might require complete human review, no unresolved factual or disclosure failures at handoff, and an acceptable total effort for the chosen task. Set the criteria with the accountable communications owner. Keep severe failures visible regardless of an attractive average, and investigate their causes before expanding use.

Keep useful scrutiny visible too. HM Treasury's guidance on evaluating AI interventions asks evaluators to take into account the breadth and unpredictability of intended and unintended outcomes, and how they vary for different groups². If the pilot shows that the old process rarely checked whether a source supported a claim, count the new checking in the total and explain its purpose; removing it to recover a drafting-time advantage would weaken the workflow. Count work that moves to another role as well, such as a coordinator preparing a cleaner input pack. Logging itself costs time and can change how people work, so keep the fields few, remove those that never inform a decision and label reconstructed estimates as estimates. The Magenta Book notes that process evaluations typically combine quantitative and qualitative methods, covering perceptions of how well something operated as well as factual details⁴.

Summarize the number and mix of completed, failed and unfinished assignments, the observed variation and the limits of the comparison. A small exercise can reveal a useful workflow or a recurring failure without proving how the tool will perform across every persona and format. If a model, source process or review method changes materially, evaluate the affected work again.

Report a close result honestly. HM Treasury's AI evaluation guidance includes a hypothetical example in which briefing notes from the AI-assisted group scored an average of 80% on accuracy and quality measures against 81% for the control group, with the evaluators noting that such preliminary findings may not hold at scale². Report a tie on quality as a tie, alongside whatever the ledger shows about effort.

No. A drafting pilot measures production effort and quality; whether audiences understand, trust or act needs its own evaluation.

AMEC's Integrated Evaluation Framework↗ separates outputs, the content and activities your audiences were exposed to, from out-takes, outcomes and organizational impact⁵. Drafting effort belongs to the production workflow. Faster drafts alone cannot establish that audiences understand a message, trust an organization or change their behavior. If those are your objectives, plan the relevant audience research and account for other influences on the result.

End the pilot with a bounded decision: continue for the tested task, repair one handoff or failure at a time and retest it on comparable work, or stop. State who made the decision and which evidence supports it. That gives the communications team a practical next step while preserving the questions the pilot has not yet answered.