Automated Smoke Testing

The problem

After every deploy, someone should confirm the build isn't fundamentally broken before the team sinks time into deeper testing or, worse, before it goes further down the pipeline. In practice that check is ad hoc — someone loads the app, clicks around, and gives a thumbs up from memory — or it's a scripted smoke suite that's quietly gone stale.

Why the usual approach falls short

• Manual smoke checks are inconsistent. Different person, different clicks, different confidence each time.
• A scripted smoke suite decays. It covers a fixed list that stops matching the product, and it still breaks on UI changes.
• "Seems fine" isn't a signal you can gate on.

How BlackBat AI fits

Point BlackBat AI at a fresh deploy and it does the smoke check for you: loads the app, signs in, exercises the main flows, and reports whether the core works or what's broken — with screenshots. A shallow pass is fast enough to fit right after a deploy, and the result is a clear go/no-go instead of a vibe.

For the handful of things that absolutely must work for a build to be worth testing further — "the app loads and a user can log in" — pin those down as plain-English checks so they're verified explicitly every run. The glossary entry on smoke vs sanity testing covers where this sits relative to other quick checks.

What your first run looks like

1. Deploy to staging.
2. Run BlackBat AI — a quick pass plus your must-work checks.
3. Green: proceed. Red: the findings tell you exactly what's broken.

What a good smoke check covers

Keep it small. A useful smoke check answers one question: is this build worth testing further? In practice that means:

• The app loads and the main pages respond without errors.
• A user can sign in with a test account.
• The one or two core journeys work, such as creating the main object in your product or reaching the checkout.
• Nothing obviously broke on the pages a recent change touched.

Anything slower or more detailed belongs in a fuller regression run, not the smoke gate.

Wiring it into a deploy

To make it a real gate rather than something someone remembers to run:

1. Create an API key under Settings → CI/CD, and store it in your CI provider's secrets.
2. After the deploy step, call POST /v1/ci/runs for the environment you just deployed to. Use a diff crawl for speed once the environment has had a full crawl, and set the idempotency key to the commit SHA so a retried job doesn't start a duplicate.
3. Poll GET /v1/ci/runs/:id. PASSED lets the pipeline continue; FAILED, TIMEOUT or CANCELLED blocks it.

The CI/CD walkthrough has the exact requests. CI/CD triggering is available on paid plans; on a free plan you can run a smoke check on demand or on a schedule instead.

Keeping it fast

• Use diff runs for routine post-deploy checks. They re-crawl only the pages that changed since your last run, so they cost less time and fewer credits. See how diff runs work.
• Do a full run periodically, and before major releases, so the diff baseline doesn't drift.
• Keep the plain-English checks short. A handful of must-work flows, not your whole regression suite.

Where it fits

Smoke testing is the first gate after a build. Unit and integration tests run earlier; the autonomous smoke pass confirms the whole thing hangs together before deeper testing or a release. See release testing for the checkpoint before production and continuous testing for keeping the loop running.