Test your oracle

Two layers — unit tests against a stub runtime, and integration tests that boot the real Nest app against real Matrix and a real LLM.

Recipe — wire vitest in two modes

The example oracle ships a single config that runs unit tests by default and switches to integration mode with --mode int. Copy it.

File: vitest.config.ts

import { defineConfig, mergeConfig } from 'vitest/config';
import nestConfig from '@ixo/vitest-config/nest';

export default defineConfig(({ mode }) => {
  if (mode === 'int') {
    const merged = mergeConfig(nestConfig, {});
    merged.test = {
      ...merged.test,
      include: ['test/**/*.int.test.ts'],
      exclude: ['node_modules', 'dist'],
      testTimeout: 120_000,
      hookTimeout: 120_000,
      setupFiles: ['./test/integration/setup.ts'],
      fileParallelism: false,
    };
    return merged;
  }
  return mergeConfig(nestConfig, {});
});

Then in package.json:

{
  "scripts": {
    "test": "vitest run",
    "test:integration": "vitest run --mode int"
  }
}

The two layers

Layer	What it boots	When to use it
Unit	Nothing — createTestRuntime fakes a RuntimeContext	Tool input parsing, middleware hooks, sub-agent prompts in isolation
Integration (Tier A)	A test runtime — invoke tools directly, no LLM, no HTTP	Verifying upstream-API integration deterministically and for $0
Integration (Tier B)	Full createOracleApp + real Matrix + real LLM	End-to-end: discovery, routing, multi-turn behaviour

All three live under apps/qiforge-example/test/integration/ — read weather.int.test.ts for a complete example.

Setup file

File: test/integration/setup.ts

import 'reflect-metadata';
import { fileURLToPath } from 'node:url';
import { dirname, resolve } from 'node:path';
import { config as dotenvConfig } from 'dotenv';
import { expect } from 'vitest';
import { langchainMatchers } from '@langchain/core/testing';
import { Logger } from '@nestjs/common';

const __dirname = dirname(fileURLToPath(import.meta.url));
const appRoot = resolve(__dirname, '../..');

dotenvConfig({ path: resolve(appRoot, '.env') });
dotenvConfig({ path: resolve(appRoot, '.env.integration'), override: true });

expect.extend(langchainMatchers);
process.env.LOG_LEVEL ??= 'warn';
Logger.overrideLogger(['error', 'warn']);

The setup file loads .env first (runtime config), then layers .env.integration on top with override: true (test-only credentials). Then it registers LangChain matchers and quiets logs.

Each .int.test.ts file declares its required env up front and throws on missing values — see the next step. No silent skips.

Per-file env gate — fail loud

Every integration test file lists the env it needs and throws at module load if anything is missing.

1. Declare required env at the top of the test file

   const REQUIRED_ENV = [
     'MATRIX_BASE_URL',
     'MATRIX_ORACLE_ADMIN_USER_ID',
     'MATRIX_ORACLE_ADMIN_ACCESS_TOKEN',
     'TEST_USER_MNEMONIC',
     'TEST_USER_DID',
     'ORACLE_DID',
     'OPEN_ROUTER_API_KEY',
   ] as const;
   
   const missing = REQUIRED_ENV.filter((k) => !process.env[k]);
   if (missing.length > 0) {
     throw new Error(
       `weather.int.test.ts requires the following env vars: ${missing.join(', ')}`,
     );
   }
   

   Source: weather.int.test.ts lines 80-94.

2. Do not use describe.skipIf for env gates

   Silent skips hide broken setups. A throw at file load surfaces immediately when .env.integration is missing or incomplete.

Unit test — createTestRuntime

createTestRuntime returns a TestRuntime directly (no { runtime } wrapper). Drive it through that object’s methods — listTools, invokeTool, invokeMiddleware, invokeSubAgent, listCapabilities, getManifest, assertNoCollisions, assertManifestValid.

import { describe, expect, it } from 'vitest';
import { createTestRuntime } from '@ixo/oracle-runtime/testing';
import { WeatherPlugin } from '../src/plugins/weather/index.js';

describe('WeatherPlugin', () => {
  it('registers get_current_weather at boot', async () => {
    const runtime = await createTestRuntime({
      plugins: [new WeatherPlugin()],
      config: { WEATHER_DEFAULT_UNITS: 'celsius' },
    });
    expect(runtime.listTools().map((t) => t.name)).toContain('get_current_weather');
  });

  it('has valid manifests and no registry collisions', async () => {
    const runtime = await createTestRuntime({ plugins: [new WeatherPlugin()] });
    expect(() => runtime.assertManifestValid()).not.toThrow();
    expect(() => runtime.assertNoCollisions()).not.toThrow();
  });
});

createTestRuntime resolves plugins, populates the six registries, and builds a RuntimeContext it hands to tool handlers — but does not boot Nest, talk to Matrix, or call the LLM (every ambient service is a mock). Pass plugin env through config. Use for fast, focused tests: tool registration, manifest validity, middleware hooks, and sub-agent prompts in isolation.

createTestRuntime options take config (plugin env, read as ctx.config) — not env. Its mocks.fetch is exposed for handlers that read it explicitly; it is not auto-installed onto globalThis.fetch. To stub a plugin’s upstream HTTP calls deterministically, use the Tier A createIntegrationRuntime, whose fetch option does replace globalThis.fetch. The full TestRuntime surface lives in create-test-runtime.ts.

Tier A integration — direct invoke

import { afterAll, beforeAll, describe, expect, test } from 'vitest';
import {
  createIntegrationRuntime,
  type IntegrationRuntime,
} from '@ixo/oracle-runtime/testing/integration';
import { WeatherPlugin } from '../../src/plugins/weather/index.js';

describe('Tier A — direct invoke', () => {
  let runtime: IntegrationRuntime | undefined;

  beforeAll(async () => {
    runtime = await createIntegrationRuntime({
      plugins: [new WeatherPlugin()],
      user: { did: process.env.TEST_USER_DID! },
    });
  }, 60_000);

  afterAll(async () => {
    if (runtime) await runtime.close();
  });

  test('get_current_weather({ city: "Berlin" }) returns numeric temperature', async () => {
    const raw = await runtime!.invokeTool('get_current_weather', { city: 'Berlin' });
    const result = JSON.parse(raw as string) as { temp: number; city: string };
    expect(result.city.toLowerCase()).toContain('berlin');
    expect(Number.isFinite(result.temp)).toBe(true);
  });
});

Tier A boots the runtime registries against your plugin list but skips the agent loop entirely. Use it to verify env wiring, upstream-API contracts, and config threading.

createIntegrationRuntime accepts more than plugins + user: config (plugin env, defaults to process.env), features, delegation, capabilities, session, state, identity, fetch (stub upstreams), and ucan.

The default ucan stub throws on mintInvocation / createInvocationFromDelegation. If the plugin under test mints downstream invocations (memory, sandbox, skills, composio, …), pass a real adapter — boot an oracle with createIntegrationOracle() and hand its bootedOracle.app.ambient.ucan through as ucan, after seeding the user’s delegation into that oracle’s UcanService cache.

Tier B integration — full agent loop

import {
  ChatClient,
  allCaps,
  createIntegrationOracle,
  mintAuthInvocation,
  mintUserDelegation,
  waitForMatrixLoaded,
  type IntegrationOracle,
} from '@ixo/oracle-runtime/testing/integration';
import * as sdk from 'matrix-js-sdk';

describe('Tier B — agent loop', () => {
  let oracle: IntegrationOracle | undefined;
  let client: ChatClient | undefined;

  beforeAll(async () => {
    const matrixClient = sdk.createClient({
      baseUrl: process.env.MATRIX_BASE_URL!,
      userId: process.env.MATRIX_ORACLE_ADMIN_USER_ID!,
      accessToken: process.env.MATRIX_ORACLE_ADMIN_ACCESS_TOKEN!,
    });
    oracle = await createIntegrationOracle({
      config: oracleConfig,
      plugins: [new EditorPlugin({ matrixClient }), new WeatherPlugin()],
    });
    await waitForMatrixLoaded(oracle, 90_000);

    // Primary auth: a user-signed invocation (Authorization: Bearer +
    // X-Auth-Type: ucan). Downstream authorization: the delegation.
    const invocation = await mintAuthInvocation({
      userMnemonic: process.env.TEST_USER_MNEMONIC!,
      oracleDid: process.env.ORACLE_DID!,
      userDid: process.env.TEST_USER_DID,
    });
    const delegation = await mintUserDelegation({
      userMnemonic: process.env.TEST_USER_MNEMONIC!,
      oracleDid: process.env.ORACLE_DID!,
      userDid: process.env.TEST_USER_DID,
      capabilities: allCaps,
    });
    client = new ChatClient(oracle.baseUrl, { invocation, delegation });
  }, 180_000);

  afterAll(async () => {
    if (oracle) await oracle.close();
  });

  test('weather request triggers get_current_weather', async () => {
    const sid = await client!.createSession();
    const stream = client!.stream(sid, "What's the weather in Berlin?");
    const calls = [];
    for await (const evt of stream) {
      if (evt.event === 'tool_call') calls.push(evt.data);
    }
    expect(calls.some((c) => c.toolName === 'get_current_weather')).toBe(true);
  });
});

mintAuthInvocation (sent as Authorization: Bearer + X-Auth-Type: ucan) is the primary auth artifact the runtime expects; mintUserDelegation (sent as x-ucan-delegation) carries the capabilities plugins use for downstream authorization. ChatClient sends both when supplied — that’s the production path. A delegation-only new ChatClient(url, { delegation }) still authenticates via the migration fallback, which is why older tests pass without an invocation.

Full reference, including multi-turn scenarios: weather.int.test.ts. Cross-plugin chains: agent-scenarios.int.test.ts. Boot smoke: boot.int.test.ts.

Why the vitest config looks like that

fileParallelism: false

Integration tests share a single Matrix admin user. Two test files booting in parallel collide on Matrix’s one-time key uploads at the homeserver. Run sequentially.

120s timeouts

Real Nest boot, Matrix sync, and LLM round-trips run 5-30s each. 120s leaves headroom for retries; pushing higher means cutting scope, not raising the cap.

Share one session across tests in a describe

client.createSession() is a server-side round-trip. Create once in beforeAll, reuse for every test. Only mint per-test sessions when the test’s whole point is session isolation (first-contact, cross-session recall).

Assert structurally on streamed events

Tier B uses a real model — response wording drifts. Collect tool_call events from the stream and assert which tools fired with which args, not on text output.

createIntegrationOracle swaps in cheaper test models

createIntegrationOracle installs a default resolveModel hook that overrides the main and subagent roles with cheaper test models; every other role falls through to the production resolver. Pass your own hooks to override — caller hooks merge on top per key, so a caller-supplied resolveModel wins.

What not to do

Don’t loosen assertions to mask failures. Broadening a regex, adding “or” clauses, or raising tolerances to make a flaky test pass discards the check that catches the bug. Investigate the real failure.

Don’t edit plugin code to make tests pass. Two test-side retry attempts max per failure, then stop and ask. Plugins are presumed-working production code; tests describe behaviour, not dictate it.

Don’t add skip-real-services flags (skipMatrixInit, skipGracefulShutdown) to integration tests as a speed-up. Integration tests must boot the same way production does — that’s their point.

Where to read next

Write a plugin

The Weather plugin tests live next to its source.

Environment variables

The .env.integration requirements per plugin.

Runtime context

The object handed to your tool handlers in both unit and integration tests.