Check that an AI assistant has carried out the customer’s request

The customer wants a parcel delivered somewhere else. They do not know which applications the sales team and warehouse use. When they read “Your delivery address has been changed,” they expect the current order to arrive at the new location.

Updating their customer record may only change the address for future orders. If the warehouse has already printed the label, an employee or another system also needs to replace it. The assistant’s trial must follow that step to establish whether the promise to the customer was kept.

Define what “address changed” means

Before the demonstration, ask the team to describe a successful change. For the current order, the new address needs to reach the application used to prepare the shipment. If an employee must replace the label, the request remains pending until they confirm it.

The assistant should explain that status plainly: “Your request has been sent to the warehouse. We still need to confirm whether the parcel can be relabelled.” The customer can then see that a decision is outstanding.

Anthropic’s work on agent evaluations distinguishes what an assistant reports from the state it leaves in other systems. Applied to this example, that means comparing the message sent, the order address and the label used when the parcel leaves.

Check the conditions when the change is made

The software needs to verify that this customer may change the order and that a change is still possible. Circumstances can shift during the conversation. A parcel being prepared at the start may have left the warehouse before the customer supplies the new address.

AI can interpret an ordinary-language request and ask for missing details. The connected applications must enforce permissions and the rules for changing orders. Anthropic’s guidance on workflows and agents explains how predefined steps can work alongside a model’s interpretation of a request.

Test an interrupted connection as well. The change may have succeeded even though the confirmation never reached the assistant. It needs to check the current order before repeating the action or announcing failure. While the outcome is unknown, the customer’s message should reflect that uncertainty.

Give an employee enough information to take over

If the warehouse needs to intervene, the employee needs the order, the requested address and a record of actions already attempted. Forwarding an entire conversation without a useful summary leaves the employee to reconstruct those details.

Where no change has yet been attempted, the note could read:

The customer wants to change the address for order 1042, which is already being prepared. The requested address is in the order record. No change has been applied. The warehouse needs to check whether the label can be replaced before dispatch. The customer is waiting for confirmation.

The number is fictional. In the real application, the note must link to the correct record. If an action was attempted and its result is unknown, say so. Otherwise, the employee may repeat it without checking.

Also name the team that receives the request and the person who replies to the customer. Even a clear summary can go unanswered in a queue that nobody monitors.

Ask to see several cases during the demonstration

Examine an order that can still be changed, one that has already left and one that requires warehouse staff. Add a connection failure after the change request is sent. In each case, compare the assistant’s message with what the applications recorded.

For requests passed to employees, measure the wait after transfer and the information customers have to repeat. After launch, review errors with the team. A recent request should let you find the chosen address, the warehouse confirmation and the reply sent to the customer.