Performant, Parallelizable, Framework Agnostic Node.js Integration Testing

Testing is hard

What am I testing?

github.com/HarperFast/harper (Apache 2.0)

github.com/HarperFast/harper-pro (ELv2)

Define "integration"

Unit

A single unit of code; everything else mocked/stubbed.

too little

Integration

The complete application. No hooking into internals.

just right

End-to-end

The full production stack. Like end users experience.

too much

Application is a black box

No mocking, no stubbing, no reaching inside. Assert on behavior, never on implementation.

Application = Process

Integration testing an application means managing the lifecycle of the application process

• Start it
• Wait for it to be ready
• Restart it
• Send signals to it
• Write to stdin
• Stream and read stdout and stderr
• Read and process log files
• Monitor crashes
• Stop it
• Clean up

Harper is an application platform

Harper can run all sorts of applications

• Web apps: Vite, React, Astro
• Backend APIs: REST, MQTT, GraphQL, MCP
• Data stores: Blob, LMDB, RocksDB
• Multi-instance replication
• GPU access for LLMs

How do we…

Early decision: use node:test test runner API

• One less dependency to manage
• Simple concurrency model to architect around
• Dogfood the new API
• Still planning to build agnostic API to work with other runners

node:test "concurrency" is overloaded

node --test \
  --test-concurrency=5 \
  "*.test.js"

suite('x', { concurrency: true }, () => {
  test('a', async () => { … });
  test('b', async () => { … });
});

Node's default concurrency is built for cheap tests

node --test --test-concurrency

os.availableParallelism() // 12 on my laptop

The experiment

• 20 generated test files, each starting an application process
• Work per file randomized within a mode:
  • 0-100 MB file I/O
  • 0-3 processes
  • 0-4 worker threads
• Concurrency 1 → 12 (sequential to max parallelism)
• 25 samples per level, on a normal developer laptop

https://github.com/ethan-arrowood/node-test-runner-parallelization-analysis

The gain plateaus, then reverses

Total median suite duration (shorter is faster)

Calculated default

When each test spawns its own application process, the ideal concurrency is
half the available parallelism, plus one

concurrency = floor(availableParallelism / 2) + 1
            = floor(12 / 2) + 1
            = 7   // the measured optimum

This might not be perfect, but its a better starting point than max parallelism, and remains configurable

Dynamic ports would mean core changes

The easy fix: let the OS hand out a free port.

server.listen(0, () => {
  const { port } = server.address();   // the OS picked a free one
});

But two problems:

• The app has to report back which ports it received
• Harper currently requires ports to be configured

How can we solve this without modifying core?

Not a real fix: get-port

// what get-port does, roughly:
async function getPort() {
  const server = net.createServer();
  await listen(server, 0);          // OS assigns a free port
  const { port } = server.address();
  await close(server);              // ← the port is released here
  return port;
}

const port = await getPort();
const harper = await createHarper(port);   // ← nothing reserved it in between

Between the release and the reuse, the OS can hand that port to any other process — especially with many test processes racing at once.

Introducing loopback addresses

All of 127.0.0.0/8 (127.0.0.1 through 127.255.255.255) loops back to localhost

// every instance keeps its default ports
test/application 1 → http://127.0.0.2:9925
test/application 2 → http://127.0.0.3:9925
test/application 3 → http://127.0.0.4:9925

• Every instance can use port 9925 without conflict
• Isolation is by address

One caveat: only Linux enables by default

# macOS: register each address as a loopback alias (once per machine)
sudo ifconfig lo0 alias 127.0.0.2 up
sudo ifconfig lo0 alias 127.0.0.3 up

How do we manage which process gets which address?

Loopback Address Pool: JSON file with an array of PIDs, shared by every test process.

[42, 43, null, 45, null, null]

index 0 → 127.0.0.2 (PID 42) · index 1 → 127.0.0.3 (PID 43) · …

• Index is the address; value is the owning process ID
• First-available allocation; released on teardown

Cross-process mutex to avoid conflicts

// 'wx' fails if the file already exists — atomically
const handle = await open(LOCK_PATH, 'wx');
await handle.close();   // we hold the lock

One of many options (daemon, orchestrator, atomics).

File lock was simplest. Plus a stale-lock timeout for crashed processes.

Crashed tests would leak addresses forever

// signal 0 checks liveness without signalling
try {
  process.kill(pid, 0);       // alive — leave it allocated
} catch {
  loopbackPool[index] = null;  // dead — reclaim it
}

A Ctrl+C mid-run doesn't break the pool

Allocated is not the same as usable

// bind to port 0 just to prove the address exists
server.listen(0, loopbackAddress, () => {
  server.close(() => resolve(loopbackAddress));
});

1. Verify the whole pool at startup (ensure setup script was run)

2. Verify again per allocation to avoid collisions

Still not immune to race condition with another app, but loopback allocation collides far less often than dynamic ports.

The API is shaping up

// ctx is a general options object

await startHarper(ctx);
// 1. allocate a loopback address from the pool
// 2. create a temporary install directory
// 3. spawn Harper, wait until it is actually ready
// 4. populate ctx.harper

ctx.harper.httpURL            // 'http://127.0.0.3:9926'
ctx.harper.operationsAPIURL   // 'http://127.0.0.3:9925'
ctx.harper.rootpath           // '/var/folders/js/abc123_def456/T/harper-test-789'

await teardownHarper(ctx);    // kill, release the address, remove the directory

The API imports nothing from a test runner

startHarper(ctx, options?)
setupHarperWithFixture(ctx, fixturePath, options?)
killHarper(ctx)
teardownHarper(ctx)
createHarperContext(name?)   // for non-node:test frameworks

ctx is any object with an optional name and a harper property to write to.

Deliberately loose, so a node:test context and a plain object both satisfy it.

Keep it flexible

// src/run.ts, line 1:
// Important! This script should not be required
// to execute integration tests.

https://github.com/HarperFast/integration-testing/blob/main/src/run.ts

The convenience runner (configures a smarter concurrency default) holds no state.

Every test file still runs under plain node --test, and parallelization still works.

Two runners, same lifecycle

suite('install', (ctx: ContextWithHarper) => {
  before(async () => {
    await startHarper(ctx);
  });
  after(async () => {
    await teardownHarper(ctx);
  });

  test('serves', async () => {
    const res = await fetch(ctx.harper.httpURL);
    strictEqual(res.status, 200);
  });
});

const test = base.extend<
  {}, { harper: HarperContext }
>({
  harper: [makeHarperFixture(name),
           { scope: 'worker' }],
});

test('home page renders',
  async ({ page, harper }) => {
    await page.goto(harper.httpURL);
    await expect(page.locator('h1'))
      .toHaveText('Next.js v16');
  });

Playwright integration

export function makeHarperFixture(fixtureName: string) {
  return async ({}, use) => {
    const ctx = createHarperContext(fixtureName);
    const started = await setupHarperWithFixture(ctx, fixturePath, {
      harperBinPath: getHarperBinPath(),
      startupTimeoutMs: 120_000,   // a Next.js build is slow
    });

    await use(started.harper);

    await teardownHarper(started);
  };
}

https://github.com/HarperFast/nextjs/blob/main/integrationTests/fixtures.ts

Discovering core testing values for high quality tests

• Independent: no dependence on order or shared state
• Hermetic: self-contained, no external side effects
• Deterministic: same input, same output

One default runner can't run all the tests

Parallelize across runners: a matrix that shards the suite.

# .github/workflows/integration-tests.yml
strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: node --test --test-shard=${{ matrix.shard }}/4

--test-shard=1/3 always runs the same group of tests, on any machine
https://github.com/HarperFast/harper/blob/main/.github/workflows/integration-tests.yml