Harper Integration Testing Framework Development and Impact Analysis

About me

Experience

• Harper: Head of Open Source Engineering, 2024 to present
• Vercel: Senior Software Engineer, 2022 to 2024
• Microsoft: Software Engineer II, 2019 to 2022

Education

• Wentworth Institute of Technology, Boston, MA: B.S. Computer Science, 2016 to 2019
• Presidential Award for Academic Excellence

Open source

• Previous Fastify maintainer
• Node.js & Undici maintainer
• OpenJS Foundation
• Ecma TC39 and TC55

ethanarrowood.comgithub.com/Ethan-Arrowoodlinkedin.com/in/ethan-arrowood

Outline

1. What am I testing? Harper, the platform under test
2. The problem Harper's legacy integration suite
3. The goal Every test file gets its own Harper
4. How much parallelism? Measuring instead of guessing
5. Networking contention Everything wants port 9925
6. Framework agnostic One lifecycle, any test runner
7. Surviving CI Sharding across runners
8. Did it work? Harper core, harper-pro, and plugins
9. Looking back

What am I testing?

github.com/HarperFast/harper (Apache 2.0)

github.com/HarperFast/harper-pro (ELv2)

Harper is an application platform

Harper can run all sorts of applications

Testing is hard

Before: one long, ordered test suite

// integrationTests/apiTests/tests/testSuite.mjs
describe('Main Test Suite', async () => {
  await import('./0_envCleanup.mjs');
  await import('./1_environmentSetup.mjs');
  await import('./2_dataLoad.mjs');
  await import('./3_sqlTests.mjs');
  // … 23 more, in exactly this order
  await import('./26_crlVerification.mjs');
});

Shared state meant

• No running one file on its own
• No sharding across machines
• No parallelism
• No isolating a failure to one test

One test fails? Must rerun the entire suite to debug.

Data and method

Most of each run was waiting

Data and method

Data and method

Data and method

Fixing one bug, the old way

1. Make a change
2. Push it, then wait about 8 minutes for CI
3. A test in a file you never touched fails
4. Your change? State from an earlier file? Just flaky?
5. Rerun the whole suite and hope

Every failure was a wild goose chase.

Define "integration"

Unit

A single unit of code; everything else mocked/stubbed.

too little

Integration

The complete application. No hooking into internals.

just right

End-to-end

The full production stack. Like end users experience.

too much

Application is a black box

No mocking, no stubbing, no reaching inside. Assert on behavior, never on implementation.

Manage the lifecycle

• Start it
• Wait for it to be ready
• Restart it
• Stop it
• Clean up

Interact with the process

• Send signals to it
• Write to stdin
• Stream and read stdout and stderr
• Read and process log files
• Monitor crashes

What does one test need to be independent?

• Isolated Harper process, data directory, and network address
• Hooks for Harper application lifecycle
• Log retention for debugging
• Process cleanup after a successful or failed run

How do we…

Early decision: use node:test test runner API

• One less dependency to manage
• Simple concurrency model to architect around
• Dogfood the API
• Plan for a runner-agnostic API that works with other runners

node:test "concurrency" is overloaded

node --test \
  --test-concurrency=5 \
  "*.test.js"

suite('x', { concurrency: true }, () => {
  test('a', async () => { … });
  test('b', async () => { … });
});

Node's default concurrency is built for cheap tests

node --test --test-concurrency

os.availableParallelism() - 1 // 11 on my laptop

The experiment

• 20 generated test files, each starting an application process
• Concurrency 1 → 12 (sequential to max parallelism)
• 25 samples per level, on a normal developer laptop

Work per file randomized within a mode:

• 0-100 MB file I/O
• 0-3 processes
• 0-4 worker threads

The gain plateaus, then reverses

Total median suite duration (shorter is faster)

Past the plateau, variance more than doubles

20 test files · 25 samples per level · duration in ms · std dev: how much run times spread

Std dev jumps from 294 at the optimum to 654 and 690 just past it
The run-to-run spread more than doubles.

The real developer experience is in the tail

20 test files · 25 samples per level · duration in ms · p95: 95% of runs finish faster

At 10, the median is 9.5% faster than at 4, but the p95 is 14.9% slower (5,339 vs. 4,648).
2.5× the concurrency for a slower tail.

Calculated default

When each test file starts its own application process, a good default is
half the available parallelism, plus one

concurrency = floor(availableParallelism / 2) + 1
            = floor(12 / 2) + 1
            = 7   // the measured optimum

This might not be perfect, but it's a better starting point than max parallelism, and remains configurable

Dynamic ports would mean core changes

The easy fix: let the OS hand out a free port.

server.listen(0, () => {
  const { port } = server.address();   // the OS picked a free one
});

But two problems:

• The app has to report back which ports it received
• Harper currently requires ports to be configured

How can we solve this without modifying core?

Not a real fix: get-port

// what get-port does, roughly:
async function getPort() {
  const server = net.createServer();
  await listen(server, 0);          // OS assigns a free port
  const { port } = server.address();
  await close(server);              // ← the port is released here
  return port;
}

const port = await getPort();
const harper = await createHarper(port);   // ← nothing reserved it in between

Between the release and the reuse, the OS can hand that port to any other process, especially with many test processes racing at once.

Introducing loopback addresses

All of 127.0.0.0/8 (127.0.0.1 through 127.255.255.255) loops back to this machine: no network card, nothing leaves the computer

// every instance keeps its default ports
test/application 1 → http://127.0.0.2:9925
test/application 2 → http://127.0.0.3:9925
test/application 3 → http://127.0.0.4:9925

• Every instance can use port 9925 without conflict
• Isolation is by address

One caveat: macOS needs setup

macOS: register each address

# once per boot, or install the provided launchd daemon
npx harper-integration-test-setup-loopback

# what it runs for each address in the pool
sudo ifconfig lo0 alias 127.0.0.2 netmask 255.255.255.255 up

Three options for port conflicts

Every alias claims the same network

No explicit netmask? macOS assigns /8.

# address     claimed network
127.0.0.2  →  127.0.0.0/8
127.0.0.3  →  127.0.0.0/8
127.0.0.4  →  127.0.0.0/8

• Each address conflicts with every other.
• mDNSResponder (macOS network naming) keeps defending them.
  O(n²) work. A storm of announcements.

/32 removes the overlap

• 255.255.255.255: each alias claims only itself.
• Same pool capacity. Same test isolation.

Found and fixed by Kris Zyp · integration-testing #21 · July 2026

How do we manage which process gets which address?

Loopback Address Pool: JSON file with an array of PIDs, shared by every test process.

[42, 43, null, 45, null, null]

index 0 → 127.0.0.2 (PID 42) · index 1 → 127.0.0.3 (PID 43) · …

• Index is the address; value is the owning process ID
• First-available allocation; released once its ports are free

Coordinating separate processes

Cross-process mutex to avoid conflicts

// 'wx' fails atomically if the file already exists
const handle = await open(LOCK_PATH, 'wx');
await handle.writeFile(token);   // we hold the lock
await handle.close();

File lock was simplest. Plus a stale-lock timeout for crashed processes.

A slow writer can lose the lock

await writeFile(pendingPath, JSON.stringify(pool));
await assertLockHeld(token);          // still ours?
await rename(pendingPath, POOL_PATH);  // atomic publish

If a write outlasts the stale timeout, another process takes the lock. The token check narrows the window where the old writer could overwrite the new claim.

Kris Zyp · integration-testing #32 and #33 · October 2026

Crashed tests would leak addresses forever

// signal 0 checks liveness without signalling
try {
  process.kill(pid, 0);       // alive: leave it allocated
} catch {
  loopbackPool[index] = null;  // dead: reclaim it
}

A Ctrl+C mid-run doesn't break the pool

Crashed runners left Harper running

• Each Harper starts in its own process group
• A shared monitor reaps groups whose runner died
• macOS and Linux only; Windows relies on runner cleanup

Kris Zyp · integration-testing #30 · September 2026

Allocated is not the same as usable

// bind to port 0 just to prove the address exists
server.listen(0, loopbackAddress, () => {
  server.close(() => resolve(loopbackAddress));
});

1. Verify the whole pool at startup (ensure setup script was run)

2. Verify again per allocation to avoid collisions

A dying node can still hold the address

Harper's HTTP workers bind with SO_REUSEPORT, which lets several sockets share one port. A new node could share 9926 with an old node's lingering workers and split its connections.

// an exclusive bind fails with EADDRINUSE, even beside an SO_REUSEPORT socket
for (const port of [9925, 9926]) {
  if (await isPortInUseOnAddress(port, address)) skip(address);
}

Kris Zyp · integration-testing #17 and #20 · June 2026

Some listeners cover every address

A process on 0.0.0.0:9926, like a local Harper, overlaps the whole pool. On macOS and Windows it quietly takes a node's connections.

// an exclusive bind succeeds beside it there, so connect instead
const port = await findAcceptingPort(address, [9925, 9926]);
if (port) throw new ForeignListenerError(address, port);

Dawson Toth · integration-testing #40 · October 2026

The API is shaping up

// ctx is a general options object

await startHarper(ctx);
// 1. create a temporary install directory
// 2. allocate a loopback address from the pool
// 3. spawn Harper, wait until it is actually ready
// 4. populate ctx.harper

The API imports nothing from a test runner

startHarper(ctx, options?)
setupHarperWithFixture(ctx, fixturePath, options?)
killHarper(ctx)
teardownHarper(ctx)
createHarperContext(name?)   // for non-node:test frameworks

ctx is any object with an optional name and a harper property to write to.

Deliberately loose, so a node:test context and a plain object both satisfy it.

Keep it flexible

# the convenience runner sets a smarter concurrency default
npx harper-integration-test-run "integrationTests/**/*.test.ts"

# plain node:test runs the same files
node --test "integrationTests/**/*.test.ts"

https://github.com/HarperFast/integration-testing/blob/main/src/run.ts

The convenience runner (configures a smarter concurrency default) holds no state.

Every test file still runs under plain node --test, and parallelization still works.

Two runners, same lifecycle

import { suite, test, before, after } from 'node:test';

suite('install', (ctx: ContextWithHarper) => {
  before(async () => {
    await startHarper(ctx);
  });
  after(async () => {
    await teardownHarper(ctx);
  });
  test('serves', async () => {
    const res = await fetch(ctx.harper.httpURL);
    strictEqual(res.status, 200);
  });
});

Real example: HarperFast/harper integrationTests/apiTests/headers.test.mjs

Two runners, same lifecycle

import { test as base, expect } from '@playwright/test';

const test = base.extend<{}, { harper: HarperContext }>({
  harper: [makeHarperFixture(name), { scope: 'worker' }],
});

test('home page renders', async ({ page, harper }) => {
  await page.goto(harper.httpURL);
  await expect(page.locator('h1')).toHaveText('Next.js v16');
});

Real example: fixture.ts and next-16.pw.ts in HarperFast/nextjs

One default runner can't run all the tests

Parallelize across runners: a matrix that shards the suite, splitting the test files into groups that run on separate machines.

# .github/workflows/integration-tests.yml
strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: node --test --test-shard=${{ matrix.shard }}/4

--test-shard=1/4 always runs the same group of tests, on any machine
https://github.com/HarperFast/harper/blob/main/.github/workflows/integration-tests.yml

Who it was for

Data and method

The legacy suite stayed flat. The new one grew.

Legacy suiteFramework tests

Harper engineers added 1,681 tests after the port finished.

Data and method

More PRs come with integration tests

Merged harper pull requests (PRs) by month, and how many touched integration tests

Touched integration testsOther merged PRs

Bar height: PRs merged that month. Above each bar: PRs that touched integration tests, of all merged. On the bar: that share.

Data and method

More code changes ship with a test

PRs that change non-test code, and how many also change an integration test

Also changed an integration testCode only

Bar height: merged PRs that changed non-test code. Above each bar: how many also changed an integration test. On the bar: that share.

Data and method

Today, one CI run executes

Data and method

95 minutes of tests, finished in 7

About 13.7× less. The sum was measured while files ran concurrently, so it estimates a serial run rather than measuring one.

Data and method

On every platform we test in CI

Data and method

And returns a result in about 12 minutes

Data and method

And CI runs them all the time

Runs of harper's integration test workflow, by month

Through MayFrom June (the port finished Jun 9)

Data and method

CI is not always green

Main pushes with a failed job, by month. Failure rate, not flake rate.

Data and method

CI isn't reliable yet

• 24% of main runs failed (86 of 353)
• Without our Bun and uWS experiments: 13% (47 of 353)
• 12 failed on every Linux Node version at once: likely real regressions

Bun is an alternate JavaScript runtime and uWS an alternate HTTP server. We run both in CI but treat them as experiments. On Linux Node versions alone, 10% of main runs failed, against 9.3% for the legacy suite with under a third of the tests.

Data and method

Cluster tests used to need a cloud VM

1. Provision a VM for every run
2. Start the nodes in containers; test over SSH
3. Copy the logs back; destroy the VM

Data and method

Now a cluster fits in one test file

await Promise.all([
  startHarper(ctxA, config(hostnameA)),
  startHarper(ctxB, config(hostnameB)),
]);
// two nodes, two loopback addresses, one runner

harper-pro integration tests

Source: harper-pro integrationTests, Oct 31

Every Next.js version gets its own Harper

Source: HarperFast/nextjs integrationTests

Harper tests itself against the Next.js plugin

Workflow · Data and method

Plugin breakage shows up in Harper's PR

Data and method

Adopted across HarperFast

Data and method

Discovering core testing values for high quality tests

• IndependentNo dependence on order or shared state
• HermeticSelf-contained, no external side effects
• DeterministicSame input, same output

What I'd do differently

• Track CI data from day one
• I started this project on feel: the pain was obvious, but never measured
• A small team can get away with that. A larger one can't.
• By the time I measured, GitHub had purged most of the old suite's CI history

Measure the before, not just the after.

What it changed

• ~1,000 order-dependent tests became 3,237 tests in isolated files
• A full CI result in about 12 minutes
• 346 harper-pro test cases, cluster tests included, on one machine
• 40% of code PRs ship an integration test
• 34 HarperFast repos import the package

Next: making CI reliable