Google Cloud published four evaluation lessons from building its Google Cloud Developer plugin. One central lesson is to measure the agent’s actions rather than rely on its final reply; the post also covers live and plan-only testing and durable baselines. A separate AWS-authored account of a production agent explains why ongoing checks matter: its monitoring showed that the system had responded, not whether the answer was right.
Choose the outcomes that protect rebooking
Evaluation means repeating checks against the behavior you need, rather than relying on one successful demo. Start with a narrow workflow, such as a first-visit follow-up or a reminder to a client who has not rebooked. Define what a correct run means before you decide whether an AI agent is ready to handle it.
For example, check whether the agent:
- Selects the intended client using the appropriate booking record
- Respects contact preferences and opt-outs
- Uses the right booking link for the service or location
- Records whether the message was sent, skipped or led to a booking
These are operational checks, not just a review of message tone. A friendly-sounding follow-up can still miss the client, ignore a preference or send someone to the wrong booking page. Keep a staff member responsible for reviewing exceptions and deciding what should happen when the agent is unsure.
For med spas, IV and wellness clinics, ask a qualified privacy adviser whether HIPAA applies before connecting client records. Do not send health or treatment details to ad platforms or general marketing tools. Keep the information used in this workflow limited to what is needed for the follow-up.
Check what the agent actually did
Google Cloud’s guidance is to score execution steps—the actions an agent takes—rather than judge only its final reply. For a rebooking workflow, that means reviewing the record it selected, the contact preference it checked, the link it chose and whether it saved the outcome. An execution log, meaning a record of the agent’s actions, can help staff check those steps.
Google’s example shows why this distinction matters. In its initial evaluation suite, an automated grader marked three prompts as failures; after the team checked raw tool-call logs and transcripts, it found the plugin had passed all three. Google Cloud describes the difference between the grader’s result and the team’s review.
For your workflow, keep examples of both successful and unsuccessful cases. Include straightforward bookings as well as cases where the client match is unclear, an opt-out is present or a booking link is unavailable. When a check fails, review the underlying steps with a staff member. That can help distinguish a bad data match from a broken link or a rule the agent did not follow.
Keep checking after launch
Google Cloud recommends comparing live runs with plan-only runs. A live run tests whether the agent can carry out its actions; a plan-only run shows what it proposes without executing them. For a booking workflow, use these checks before launch and when making meaningful changes to the instructions, booking connection or follow-up rules.
A baseline is a saved set of examples that represents what good handling should look like. Google Cloud calls these “golden sets” and recommends keeping them as models and tools change. For your team, the examples can be ordinary follow-ups alongside the cases that must be handled carefully, such as opt-outs and uncertain client matches.
After launch, repeat the checks and review exceptions on a schedule your team can maintain. Pay attention to practical signs of drift: more cases with an incorrect client, an unusable link or a missing outcome record. AWS’s production-evaluation framework explicitly asks whether quality is drifting in live use. Its article also cautions that monitoring a response does not establish that it is correct.
What to do
- Pick one follow-up workflow to evaluate.
- Write down the correct client, opt-out, link and outcome rules.
- Save examples of routine cases and exceptions.
- Assign a staff member to review failures and keep checks going after launch.
A rebooking agent should be judged by the work it completes and the records it leaves behind, not simply by whether it produces a convincing message. Regular checks can help you spot problems before they become a pattern of missed follow-ups.
- ai evaluation
- client retention
- rebooking
- appointment businesses
- marketing operations
Alan Jordan, founder and principal solutions architect. Data engineering, data science and business analytics for appointment-based businesses.
