Ledge the goat, edgegoat logoedgegoat

edgegoat Guides

Unicode test strings for JavaScript

JavaScript strings are sequences of UTF-16 code units, not characters. Anything that slices, counts, reverses or compares text by code unit will eventually split an emoji or treat two identical looking strings as different.

The bug: cutting a surrogate pair in half

Characters outside the Basic Multilingual Plane, including most emoji, take two code units (a surrogate pair). slice, substring and str[i] do not know that. Cut between the two halves and you get a lone surrogate: it renders as a replacement character, encodeURIComponent throws URIError, and some databases reject the text.

Real bugs of this kind were fixed in google/budoux#1411 (do not split surrogate pairs) and microsoft/vscode-js-debug#2423 (avoid splitting surrogate pairs in trimEnd).

A naive implementation that fails

example.js
function truncate(text, max) {
  return text.length <= max ? text : text.slice(0, max) + '...';
}

The edgegoat test that catches it

example.test.js
import { expect, test } from 'vitest';
import { check, strings } from 'edgegoat';

test('truncate never produces malformed UTF-16', () => {
  const inputs = strings({ exclude: ['malformed'] });
  const failures = check((s) => truncate(s, 10), inputs, (out) => out.isWellFormed());
  expect(failures.map((f) => f.label)).toEqual([]);
});
Fails: surrogate pair cut by slice(0, 10): the emoji starts at index 9. Other lengths also break the ZWJ family, flag and skin tone cases. (isWellFormed needs Node.js 20; older versions can use a regex.)

The fix walks grapheme clusters with Intl.Segmenter:

example.js
function truncate(text, max) {
  if (text.length <= max) return text;
  let out = '';
  for (const { segment } of new Intl.Segmenter(undefined, { granularity: 'grapheme' }).segment(text)) {
    if (out.length + segment.length > max) break;
    out += segment;
  }
  return out + '...';
}

What strings() contains

  • Whitespace and invisible characters: no-break space, ideographic space, zero width space, byte order mark, soft hyphen.
  • Emoji and graphemes: ZWJ family, regional indicator flags, skin tone modifiers, variation selectors, keycaps.
  • Surrogates: a pair at a cut boundary, lone high and low surrogates.
  • Normalization: NFC vs NFD accents, Hangul syllables vs jamo, stacked combining marks.
  • Bidirectional text: Hebrew and Arabic mixed with Latin, the right to left override used by Trojan Source, bidi isolates.
  • Casing: Turkish dotted and dotless i, German sharp s, Greek final sigma.
  • Newlines and control characters: CRLF, CR, U+2028, U+2029, NUL, vertical tab.
  • Injection shaped text: HTML, SQL quotes, regex metacharacters, template placeholders, __proto__.
  • Digits and scripts: Arabic-Indic and full width digits, Thai without spaces, wide CJK characters.

Every value is written with escapes in the source, so the catalog shows emoji as code points such as \u{1F600}, never as hidden glyphs. Filter with tags, for example strings({ tags: ['emoji', 'bidi'] }).

FAQ

Strings that break assumptions about length, equality and display: emoji outside the BMP, ZWJ sequences, NFC vs NFD pairs, right to left text, invisible spaces, lone surrogates and locale sensitive casing. strings() returns 49 of them with the reason each one breaks code.

UTF-16 encodes code points above U+FFFF as two 16 bit code units, a high surrogate followed by a low surrogate. JavaScript string indexes count code units, so one emoji has length 2 and can be cut in half.

s.length counts UTF-16 code units, [...s].length counts code points, and [...new Intl.Segmenter().segment(s)].length counts grapheme clusters, which is what users perceive as characters.

The big list of naughty strings is a large raw list, great for fuzzing. edgegoat is smaller and curated: every string has a label, a one line reason and tags, plus typed generators for dates, numbers and URLs. They work well together.

More guides