breakhound StarGitHub stars

Guide

Automated testing in GitHub Actions without writing tests

A pull request check that finds crashes, time zone bugs and broken Unicode handling in your JavaScript library takes about ten lines of YAML. Here is the workflow, what it reports, and how to keep runs reproducible.

By Alexandre KohlerUpdated 3 min read

Why add a bug hunt to CI

Unit tests only cover the cases someone thought of. A type driven check covers the rest: npx breakhound reads your TypeScript declarations, generates inputs for every parameter (including time zones, DST edges, emoji, lone surrogates, -0, NaN and empty strings), and checks invariants on the results. Running it in CI means a regression is caught on the pull request that introduces it, not in a bug report six months later. For how the checks work, see finding bugs automatically in JavaScript.

The workflow people write first

The first attempt usually has two problems. npm install runs lifecycle scripts of every dependency, and || true swallows the exit code, so the job is green no matter what was found. A check that can never fail is decoration.

.github/workflows/breakhound.yml
name: breakhound
on: [push, pull_request]
jobs:
  hunt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm install
      - run: npx breakhound . || true

A workflow that fails on real bugs

Install with npm ci --ignore-scripts for a reproducible dependency tree without running install scripts, then gate on severity with --fail-on. The process exits with code 1 when a finding has that severity or higher, which fails the job.

.github/workflows/breakhound.yml
name: breakhound
on:
  push:
  pull_request:
jobs:
  hunt:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci --ignore-scripts
      - run: npx breakhound . --fail-on high

Start with --fail-on high (lone surrogates, crashes, time zone dependence) and tighten to medium once the backlog is clean. If your package entry point is a build output, add your build step before the breakhound step so the declarations and code exist.

breakhound executes the code it analyzes. That is fine for your own repository. For pull requests from forks, keep permissions minimal as above, expose no secrets to the job, and avoid pull_request_target.

Annotations and the job summary

When it detects GITHUB_ACTIONS=true, breakhound emits workflow annotations (::error and friends) so findings show up on the run page, and it appends a markdown report to the file named by GITHUB_STEP_SUMMARY. You get a readable summary on the job page with no extra steps and no configuration. The log header and summary line look like this:

Terminal output
breakhound v0.1.0  buggy-lib@1.0.0 (local)
23 functions analyzed, 1 skipped | 4444 calls | 6 time zones | seed 42 | 6.8s

HIGH (5)
  [lone-surrogate-output] truncate returns a string with a lone surrogate (confidence high)
    truncate('\u{1F600}', 1)  -> '\ud83d'

Summary: 5 high, 5 medium, 0 low, 2 info

Keeping the evidence: JSON, JSONL and artifacts

A red check is more useful when the reproduction scripts and Vitest tests are one click away. Use --json --output report.json to write the full report to a file, --log to write a JSON Lines run log (the BREAKHOUND_LOG environment variable works too), and upload both with actions/upload-artifact. By default breakhound also saves .breakhound/last-run.jsonl and .breakhound/report.json unless you pass --no-save.

Steps added after install
      - run: npx breakhound . --fail-on high --log run.jsonl
      - run: npx breakhound . --json --output report.json
        if: always()
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: breakhound-report
          path: |
            report.json
            run.jsonl

The if: always() guards matter: without them the later steps are skipped exactly when the gate step fails, which is when you want the files.

Seeds and reproducibility

Input generation is seeded. The default seed is 42, so two runs on the same commit explore the same inputs. That is what you want on pull requests: a failure should not vanish on rerun. To widen coverage over time, run a scheduled workflow that passes a different seed, such as --seed ${{ github.run_number }}, and use the seed printed in the log header to rerun locally:

Terminal
npx breakhound . --seed 1337 --max-calls 50000 --fail-on high

You can also narrow a noisy run with --only, --exclude or --rules, and bound it with --max-calls and --timeout. Related reading: time zone bugs in JavaScript, which breakhound exercises by running under several TZ values, and negative zero bugs. See the home page for the full flag list.

FAQ

How do I make the GitHub Actions job fail on findings?

Pass --fail-on high (or medium, low, info). The command exits with code 1 when a finding of that severity or higher exists.

Where does the job summary come from?

In GitHub Actions breakhound appends a markdown report to GITHUB_STEP_SUMMARY automatically and emits annotations. You do not enable it.

Why use npm ci --ignore-scripts?

npm ci installs exactly what the lockfile says, and --ignore-scripts skips dependency lifecycle scripts. breakhound installs npm packages with --ignore-scripts as well.

Will the same commit give the same findings on every run?

That is the point of seeding. The default seed is 42; pass --seed to choose another one and record it so a failure can be reproduced locally.

How is this different from running fast-check or Jazzer.js in CI?

fast-check needs you to write properties and arbitraries, and Jazzer.js needs fuzz targets. breakhound is type driven and needs neither, so it works as a zero setup first line of defense. See fuzz testing TypeScript for the trade-offs.