Stefan Press Consulting

Testing

ATS · the Automated Test Suite

Every change earns its way out.

Five gates. Two machines. Real devices. One verdict.

  1. 01

    Write

    An agent writes the change, and the test that proves it.

  2. 02

    Prove

    Every file parsed, the house rules enforced, and every check shown failing on a bad sample first.

  3. 03

    Run it for real

    A real server, a throwaway database, real browser engines, simulators and emulators.

  4. 04

    Cross-test

    The other machine runs it all again on the exact revision. Mac and Windows must agree.

  5. 05

    Ship & prove

    The owner's go, one deploy at a time, then every live file's fingerprint checked.

The lab

iPhone simulator
Android emulator
Safari engine
Chrome engine

Scripted journeys tap and type through the real screens, at phone and desktop sizes, with every outside site blocked.

ats — run log
  1. parse every fileok
  2. house rulesok
  3. checks fail on bad samplesok
  4. the site, end to endok
  5. browsers · phone + desktopok
  6. cross-test on the other machinepass
  7. deploy · staged · switchedok
  8. live fingerprints matchALL GOOD
  • LoggedEvery run keeps its screenshots and logs - and someone reads them.
  • RecordedEvery build's results are written down next to its revision.
  • FingerprintedEvery deploy is proved file by file against what was tested.

· testing

Every change earns its way out.

  1. WriteThe change arrives with its own tests.
  2. ProveEach check is shown a bad sample - it must fail.
  3. Run it for realReal servers, real browsers, phone and desktop.
  4. Cross-testThe other machine runs it all again.
  5. Ship & proveEvery live file matches what was tested.

ALL GOOD

Every Build Candidate runs through the Automated Test Suite (ATS) before it can ship, on two machines, and is checked once more on the server after it goes out. Nothing reaches a release because it looked right: each step below has to pass, and each one is built so that it can fail.

The path every change takes

  1. Every file is read. Each source file is parsed for errors before anything else runs.
  2. House rules, checked by machine. Smoke checks refuse code that would break the site's security rules - no inline scripts or styles under the strict content policy, private folders closed to the web, include-only files that refuse to run on their own.
  3. Checks that prove they can fail. Static checks read the code for the rules the owner has set; each one is run against a deliberately bad sample to show it catches it.
  4. The real site, end to end. The pages run on a real web server against a throwaway database, emptied on every run: sign-in, invitations, guest access, the release board, the theatre and the visit records are exercised request by request, with a marked test account that is never a real person.
  5. Real browsers. Forms, animations and layouts are checked in real browser engines - Safari's on the Mac, Chrome's on Windows - at phone and desktop widths, with every outside site blocked so a test never reaches the live apps.
  6. The cross-test. The other machine runs the whole suite again on the exact same revision, in its own copy. A Mac change ships only after Windows passes it, and the other way round.
  7. The owner's go, one at a time. Nothing deploys without the owner's go, and Ground Control hands the shared server to one deploy at a time.
  8. A deploy that refuses. The deploy stops unless the code is exactly the tested revision and the suite passes; it only uploads, stages each file and then switches it in.
  9. Proved after it lands. A post-deploy check compares the fingerprint of every live file with the revision and makes a handful of ordinary requests; it has to end ALL GOOD.

What the apps’ test beds taught

The testing grew up with two of the apps. Pill Inspector's Automated Test Bed runs one library of scripted user journeys against the real app on Android and iPhone; AutoAppraisal turned that bed into an Automated Test System of its own that other apps can plug into. The same ideas now run through every release:

  • Tests drive the real app. Scripted journeys tap and type through the actual screens, in emulators, simulators and real browsers — not just the code underneath.
  • One set of journeys for every platform. Android and iPhone read the same scenario files, so a fix is tested the same way on both.
  • Honest results. A failure, a crash and a stalled device are reported as different things, and a check that could not run never counts as a pass.
  • The tests are tested. Deliberate faults are planted to prove each suite catches them; a green run is never trusted on its own.
  • Evidence you can look at. Every run keeps its screenshots and logs, and someone looks at them.
  • Two machines, one verdict. A Mac and a Windows PC each test every change before it ships. They pull their work from the server, and a new machine starts switched off until it is approved.
  • Test data is fenced off. Test accounts are marked, hidden from everyone else and removable, and the test machines have their own secured way in, separate from people's.
  • A gate before every deploy. Untested code is refused, and a check after each release proves exactly what was deployed.

: Overview Controlling Building Operating