Why Every Worked Example Here Runs on Synthetic Data
Payout Math reports that $17,597 of $521,804 in approved incentive payouts should never have been approved. No company lost that money. Galewick Climate Systems, the HVAC manufacturer whose program the tool audits, is a fictional company, and its claims and rebate statements are a synthetic dataset I generated to show a kind of audit that is real.
Every worked example on this site is built that way, on purpose. The real version of this work can't be shown, so the examples run on synthetic data made under rules that let a stranger check it: the generation is locked to a seed, the errors are planted at rates written down beforehand, the verdict is drafted before the data exists, and the files are published so any number can be recomputed.
NDAs keep the real proof out of view
I have spent nineteen years in incentive-program operations, and none of the files from that work can appear here. A program sponsor hands over its claim file and its payout ledger under an agreement that keeps both private. The analyst who reconciles a quarter of rebate statements against distributor purchase lines sees every distributor's volume and tier. That is exactly what a sponsor would not want published, and the agreement is right to forbid it.
That leaves two usual ways to show past work. One is the summary line, "reduced payout errors for a manufacturer," which nobody can check. The other is anonymization, which I covered in a separate post. Anonymizing keeps the structure of a client's data while replacing each identifying value, and it suits a case study well. It still starts from a client's file and the client's consent, and it stops short of handing a reader the whole dataset to download.
Side by side, the three differ most in what a reader can check.
| Summary line | Anonymized client data | Synthetic data | |
|---|---|---|---|
| Where the data comes from | A client's file, unseen | A client's file, with identifying values replaced | A generator, with no client involved |
| What you can see | One sentence | The tables and charts chosen for the case study | Every row, the rules, and the code |
| What you can check | Nothing | That the figures shown agree with each other | Any number, by recomputing it |
| What it shows | A claim | A real result, with the names removed | The method, working on errors that are known |
Synthetic data removes that ceiling. Nothing in the file belongs to anyone, so all of it can be published: the raw rows, the rules, the code that made them, and the list of errors hidden inside.
Three invented companies carry every example
Cinderhaven Provisions is a fictional specialty food company, and its figures are a synthetic dataset. It invoices about $25 million a year across 50 SKUs, 6 retailers, 3 distributors, and 640 stores. Its warehouse holds 36 months of orders, deductions, payments, and scan data, including 46,760 retailer shipments. It carries the CPG retail and deduction examples.
Quellwood Provisions is an invented direct-to-consumer food brand with one quarter of web analytics: 1,400,182 events and 5,556 orders from July through September 2026, spread across 12 paid campaigns. It carries Campaign Math.
Galewick Climate Systems is an invented HVAC manufacturer with 42 contractors and 20 distributors in its incentive program. In one program year it received 1,235 install-bonus claims, paid 1,084 of them, and issued 80 quarterly rebate statements. It carries Payout Math.
None of the three is a client with the name changed. For Galewick, that can be shown instead of asserted. Every value in its program rulebook, from the tier thresholds to the per-unit bonus amounts, is a seeded draw from a range with a stated basis. No number in it came from my memory of a real program.
Four rules keep invented data from flattering the method
Invented data has an obvious weakness. Its author can keep adjusting it until the tool looks good. Each rule below closes one way of doing that. No dataset here meets every rule in the same way, and a table further down says which holds where.
The seed is locked, and in the newer tools it is derived. A data generator makes a long series of random draws. Started from a fixed seed, the same code produces the same file on every run, so nobody edits rows by hand afterward. Cinderhaven uses one fixed seed for its main stream and separate fixed seeds for later additions, which means a new table never disturbs an old one. Quellwood and Galewick go a step further. Their seeds are not numbers I picked. The seed for Galewick's rulebook is the SHA-256 hash of the plain string galewick-climate-systems/program-year-2025/parameters, reduced to a 32-bit number. That takes away the option of trying seeds until one produces a striking result.
Inputs come from ranges with a stated basis. Before any rows exist, the generator draws its settings. Quellwood's parameters file lists ten of them, each with a low value, a high value, a step, and a reason. The average order value, for example, was drawn at $90 from a range of $85 to $115. Each reason carries a label that says whether it rests on a published figure, a published direction, or a structural assumption with no published number behind it.
Defects are planted at published or stated rates. Clean data would show nothing, since these tools exist to find problems. So the problems go in on purpose. Quellwood's quarter carries nine classes of tracking defect: campaign tags are missing on 6% of paid-social and email sessions, purchase events fire twice on 3% of purchases, and the payment gateway shows up as the referrer on 4% of purchases. Galewick's program year carries 18 classes of payout error. Each is planted at its rate times the number of claims or statements, rounded to a whole case. In Cinderhaven's product master, the generator corrupts the GTIN check digit at a stated rate of 20% of SKUs. Where a rate is published, the page cites it. Where it is my assumption, the page says so.
A separate script then compares the finished data with published ranges and reports pass or fail. It has no ability to change anything. At Galewick, 8.8% of paid install-bonus claims carry a defect, inside the 6-15% ProfitOptics reports from its own rebate audits. The overpaid share of approved dollars came out at 3.4%, well below the 10-15% in a 2013 Zyme press release quoting Forrester. That check failed. The planted rates stayed where they were, and the page reports the gap.
The verdict is written before the data. Once you have seen the numbers, it is tempting to write the story that suits them. For Campaign Math, the verdict sentences and the rule for choosing among them were committed on October 22, 2026, before any generator code existed. The rule is mechanical: three tests on the computed results pick the sentence. One outcome in that file said that if the four attribution rules mostly agreed, the tool would not ship.
The data came out differently. All four rules name the same top campaign, and 9 of the 12 paid campaigns move two or more ranks depending on the rule. Writing first also exposed my own mistakes. One drafted clause said the top campaign closes most one-visit sales. In the data it takes 11.6% of one-visit revenue, so the clause was removed. Each correction of that kind is dated and logged in the same file.
Payout Math guards against the same temptation another way. Every dollar figure, count, and percentage on its page is filled in by the build from the validator's results, and none is typed by hand.
The files come with an answer key
Quellwood's three monthly event exports and its daily spend table are downloadable from the Campaign Math page. So are the seed log, the true sessions, and the true orders, which together show what happened before the defects were applied. The page also prints the first 16 characters of a SHA-256 hash of the joined event rows, and the repository holds the full one. A hash is a fingerprint of a file: if yours matches mine, you are holding the same rows I used.
The repository lists a hash for each of six files, and the deploy step enforces them. It regenerates the whole dataset, then refuses to publish unless every hash matches and the attribution engine reproduces its stored results exactly. From there any number can be recomputed. Campaign Math says $5,064 of Quellwood's $72,621 in paid spend bought clicks that analytics cannot tie to a campaign. With the files and a few lines of SQL or Python, you can find out whether that is true.
Payout Math keeps its answer key away from the tool. The validator reads only what a program's own systems would hold, plus the published rulebook. A test hides the list of planted errors and confirms the output does not change by a byte. Only then are the two compared: the validator recovers all 129 planted errors and flags nothing else. Each flag names the rule it breaks, so a reader can check any one of them by hand against the published rules.
Here is one of the 129. Claim A000205 is an install bonus for a GW-HP16 unit. The claim file shows the unit installed on January 20, 2025 and the claim received on April 24. Rule A4.1 says a claim must be received within 60 calendar days of the install date. Count the days: 11 left in January, 28 in February, 31 in March, and 24 in April make 94. Rule A8 allows a reduced payment only for a claim filed on time, so it does not apply. The claim earned nothing and was paid $70, and that $70 is one line of the $17,597.
Cinderhaven lives in a database, not in a handful of files, so it is guarded by its headline figures. A reference file locks them, and a check fails the build if a dollar figure drifts by more than 2% or a rate by more than half a point. The generator code is public.
Not every rule holds for every company
Cinderhaven was built first, before these rules were written down. Laid side by side, the three datasets look like this.
| Rule | Cinderhaven | Quellwood | Galewick |
|---|---|---|---|
| Seed locked | Yes, a fixed seed I chose | Yes, derived from a string | Yes, derived from a string |
| Inputs drawn from stated ranges | No. Set by hand, audited afterward | Yes, ten settings | Yes, every rulebook value |
| Defects planted at written rates | Yes, for product data | Yes, nine classes | Yes, 18 classes |
| Defect rates left alone after results | No. Updated for realism in June 2026 | Yes | Yes |
| Verdict written before the data | No | Yes | No. Figures are filled in by the build |
| Files and answer key published | Code public, headline figures locked | Files, answer key, and six hashes | Flagged decisions and rulebook on the site, answer key in the repository |
Cinderhaven's fourth row needs explaining. In June 2026 I updated the dataset to make it more realistic. An earlier version aimed to fill 92% of the units retailers ordered, and at that level the short-ship example showed a cost of $2.2 million a year. That was not believable, because a retailer would stop buying from a supplier that filled so poorly. I raised unit fill to 99.2%, brought the rate of receiving discrepancies into the range a specialty food brand would see, and regenerated the data.
The numbers got smaller. The short-ship example now shows about $296,000 a year, and the count of chargebacks fell by about half. The tools that read Cinderhaven were updated to the new figures. To catch any page left behind, each replaced figure goes on a retired list, and a build fails if a page still shows one. The newer datasets handle realism earlier: their checks were written down before any data was generated.
Quellwood and Galewick were changed after generation too, in a narrower way. Each failed its first independent realism review, and each generator was fixed. Galewick's program was cut to roughly a tenth as many claims, because the larger one did not read like a real program of its shape. The defect rates stayed fixed through every revision, and so did Quellwood's rule for picking the verdict.
It proves the method, not a client result
Synthetic data proves that a method works on a file whose errors are known. It proves nothing about what the method has found for a client.
The first half is worth having. With real data, nobody knows how many errors a file holds, so nobody can say how many an audit missed. With planted defects, the number found can be set against the number planted, and a missed class is visible to anyone who downloads the files. Real client data never comes with that check.
The second half needs saying just as clearly. The $17,597 at Galewick is not money recovered. It is what the method finds in a file I built. Your program could hold less, or more, or a kind of error Galewick doesn't have at all. A planted rate that sits inside a published range tells you the example is plausible. It doesn't tell you your own rate.
Two limits go further than that. A synthetic file contains only the defects its author thought to plant. Galewick's 18 classes are ones I have met in real programs, and real files still go wrong in ways no generator anticipates. Also, the validator and the generator share the same rule arithmetic, so finding 129 of 129 cannot catch a mistake both of them make. Payout Math covers that with values worked out by hand from the rulebook. On a client's data, the equivalent is reading the program rules and the raw files before running anything.
Read each dollar figure as what the file contains
Start with the label. Each example names its invented company near the top. A dollar figure beside Cinderhaven, Quellwood, or Galewick came out of that company's dataset.
Then read the figure as a finding about that file. Cinderhaven fills 99% of ordered units by its own records, while 84% of shipments to its national mass retailer arrive on time and complete. The pair shows how two definitions of a good shipment can disagree inside one company, and the Fulfillment and OTIF diagnostic walks through it. It is not a forecast for your brand.
Check one rate against its source, and recompute one number. Payout Math is a good place to do that without code, since each flagged decision can be reproduced by hand from the rulebook. Campaign Math suits a reader who would rather pull the files.
The question an example cannot answer is what the same method would find in your data. Both tools are built to run on a real export for that reason. The useful thing to send me is the document you already trust least: the claim file with approvals nobody re-derived, or the campaign report whose ranking changes with the attribution rule. I'll write back with where I would look first. No call required.
Where to look next
- Campaign Math: one quarter of Quellwood's campaign data, scored under four attribution rules.
- Payout Math: Galewick's install bonuses and rebate statements, re-derived from the published program rules.
- Fulfillment and OTIF diagnostic: Cinderhaven's unit fill set against its on-time, complete shipments.
- Data Anonymization Turns NDA-Bound Work Into Case Studies You Can Show: the other way to show work that sits under an NDA.
Next step
Get the number for your own data. Seven business days.
Send the export you trust least; get back your three most expensive data problems, in dollars, with a fix plan. Or start with the written intake: a few minutes by form, no call.
Private, expiring upload, never email. Mutual NDA before anything moves. Files destroyed within 30 days of delivery, with a certificate. Methods published, tools open source.