How to find bugs automatically in a JavaScript library

Why library bugs hide from your test suite

A hand written test encodes what the author already believes. You test truncate("hello world", 5) because that is the example in your head. You do not test a string that ends in an emoji, a negative number between -1 and 0, or a date that is one second after a daylight saving jump, because you do not think of them. Those are exactly the inputs real users send.

To find bugs automatically you need two things: a way to produce hostile but valid inputs for every function, and a set of rules that say what a correct result looks like without knowing the specific answer. The rules are called invariants. "A string function never returns half of a surrogate pair" is an invariant. "The same input gives the same output in every time zone" is another.

A bug that passes every ordinary test

Here is a truncate helper that looks fine. It has a test for short strings, a test for long strings and a test for the empty string, and all of them pass.

export function truncate(text: string, max: number): string {
  if (text.length <= max) return text;
  return text.slice(0, max);
}

The problem is that String.prototype.slice counts UTF-16 code units, not characters. An emoji such as '\u{1F600}' occupies two code units (a surrogate pair). Cut after the first one and you return a lone high surrogate, which is not valid text. It renders as a replacement character and makes encodeURIComponent throw a URIError. This class of defect is covered in depth in unicode surrogate pair bugs.

The fix is to never end the slice on a high surrogate:

export function truncate(text: string, max: number): string {
  if (text.length <= max) return text;
  let end = max;
  const last = text.charCodeAt(end - 1);
  if (last >= 0xd800 && last <= 0xdbff) end -= 1;
  return text.slice(0, end);
}

Run breakhound on the library

You do not need to write a harness. Give breakhound an npm package name or a path to a local folder:

npx breakhound ./my-lib

It reads the TypeScript declarations (.d.ts) to learn the type of each parameter, then generates inputs for every exported function. Strings get emoji, combining marks and lone surrogates. Dates get daylight saving transitions and dates before 1970. Numbers get -0, NaN, Infinity and values between -1 and 0. These edge cases come from the edgegoat package. Parameters that are callbacks receive generated callbacks.

The output on a small fixture called buggy-lib looks like this:

breakhound v0.1.0  buggy-lib@1.0.0 (local)
23 functions analyzed, 1 skipped | 4444 calls | 6 time zones | seed 42 | 6.8s

HIGH (5)
  [lone-surrogate-output] truncate returns a string with a lone surrogate (confidence high)
    truncate('\u{1F600}', 1)  -> '\ud83d'

  [unexpected-crash] initials throws TypeError: Cannot read properties of undefined (reading 'toUpperCase') (confidence high)
    initials('')  -> throws TypeError: Cannot read properties of undefined (reading 'toUpperCase')

MEDIUM (5)
  [nan-leak] average returns NaN for finite input (confidence high)
    average([0])  -> NaN

Summary: 5 high, 5 medium, 0 low, 2 info

Every finding has a severity (high, medium, low or info), a confidence level and a rule id.

Keep the test it writes for you

A finding is only useful if you can act on it. For each one breakhound prints a minimal reproduction script you can paste into a terminal, and a Vitest test that fails until the bug is fixed. For the truncate finding above, the test is:

import { expect, test } from 'vitest';
import { truncate } from 'buggy-lib';

const LONE_SURROGATE = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/;

test("truncate returns well formed UTF-16", () => {
  const result = truncate('\u{1F600}', 1);
  expect(result).not.toMatch(LONE_SURROGATE);
});

Run it, watch it fail, apply the fix from the previous section, watch it pass, then commit the test.

The invariants it checks

breakhound ships a fixed set of oracles. Each maps to a rule id you can select with --rules:

Each call runs in an isolated child process with a timeout, and the time zone rules run under several TZ values (six by default, including UTC, America/Sao_Paulo, Pacific/Kiritimati and America/New_York). Runs are seeded, so seed 42 gives the same inputs every time.

Narrow the run and gate your pipeline

On a large package you usually want to focus. Use --only and --exclude with wildcards, pick rules with --rules, and raise the budget with --max-calls when a first pass is clean. To fail a build when something serious shows up, use --fail-on:

npx breakhound ./my-lib --only "format*" --rules nan-leak,negative-zero-sign
npx breakhound . --fail-on high --json --output findings.json

In GitHub Actions breakhound emits annotations and writes a markdown job summary on its own. The full setup is in automated testing in GitHub Actions. If you would rather write properties by hand for the same kind of checks, see property based testing without writing tests.

The approach has paid off on real projects. Fixes merged upstream include relative-time-element (a year check used UTC instead of the display time zone), budoux and vscode-js-debug (surrogate pairs split) and pdf.js (negative numbers above -1 lost their sign).

Related guides