Free guide · Oct 16, 2026
AutoSynthData: turn agent failures into a training-task plan
How to log what went wrong, write a check that catches it, and plan useful training tasks.
The short version
AutoSynthData is ServiceNow's method for turning an agent's failures into its next training tasks. Each task is a system specification, a user prompt and a verifier, and a task only counts if the verifier can prove the result. In their EnterpriseOps-Gym Hybrid test it generated 2,000 training samples in about 18 hours, and fine-tuning on them raised the target model's mean Pass@1 by 7.2 points.
It's a published method, not a download: I didn't find a code release linked from the post. The EnterpriseOps-Gym dataset it was tested on is public under Apache 2.0. This guide is the hands-on version: log what went wrong, write a check that catches it, and plan useful training tasks. I reviewed ServiceNow's post. I have not reproduced their training.
What's in the full guide
- 01What this guide gives you
- 02Start small
- 03Step 1: log what went wrong
- 04Step 2: write a check that catches it
- 05Step 3: plan useful training tasks
- 06Failure log
- 07What the results do and don't show
- 08Primary sources