Skip to main content

Create a New Source

Prerequisites​

Before starting, read these skills:

Source Types​

A source's identity is split into two fields:

FieldMeaningExamples
source.typeThe kind of source (its role / mechanism)browser, dataLayer, cookiefirst, fetch
source.platformThe runtime that hosts the source (web, server, app, ...)web, server
PlatformInputExample types
webDOM events, dataLayerbrowser, dataLayer
serverHTTP requests, webhooksgcp, express, lambda, fetch

Augmenting SourceMap​

Every source registers its type literal and any source-specific source.* fields by augmenting SourceMap from @walkeros/core. This is how the collector and downstream consumers know about the new source kind without loosening the union to string.

Add this to the source's src/types.ts (or src/types/index.ts):

import type { Source, Elb } from '@walkeros/core';

declare module '@walkeros/core' {
  interface SourceMap {
    // Replace `mySource` with the source's package-level identifier.
    mySource: {
      type: 'mySource'; // matches the literal you return from Source.Init
      platform?: 'web'; // 'web' | 'server' | 'app' | ...
      // Add any extra fields the source surfaces in `event.source.*` here.
      // e.g. `version?: string;` is already on the base Source - only add
      // truly source-specific keys.
    };
  }
}

Reference implementations: packages/web/sources/browser/src/types/index.ts, packages/web/sources/demo/src/types.ts. Conflicting declarations cause compile errors on purpose, this surfaces naming collisions early.

Source Categories​

CategoryPurposeExamplesKey Concern
TransformationConvert external format → walkerOS eventsdataLayer, fetchMapping accuracy
TransportReceive events from specific platformgcp, aws, expressPlatform integration

Choose Your Template​

ComplexityTemplateWhen to Use
Simple transformationfetch/Generic HTTP handler, data conversion
Platform transportgcp/, aws/Cloud platform integration
Browser interceptiondataLayer/DOM events, array interception

Process Overview​

1. Research     → Deeply understand external system, SDK, and data format
2. Classify     → Determine source type and integration approach
3. Examples     → Define in/out pairs FIRST (start with the end result)
4. Mapping      → Define input → walkerOS event transformation
5. Scaffold     → Copy template and configure
6. Convention   → Add walkerOS.json metadata and buildDev
7. Implement    → Build using examples as test fixtures
8. Test         → Verify against example variations
9. Document     → Write README

Phase 1: Research​

Goal: Deeply understand the external system before writing any code. Research quality determines implementation quality.

1.1 Find and Install Official SDK​

Always prefer the vendor's official SDK package over raw HTTP API calls. The SDK handles transport, data formatting, and platform specifics - don't reinvent these.

  • Install the SDK - npm install @vendor/sdk and read the actual source
  • Read TypeScript types - Import types from the SDK package directly. Never duplicate type definitions. This ensures IntelliSense completeness and consistency with SDK updates.
  • Understand the full API surface - List every public method and type export. What data structures does the platform provide? What request/event formats exist?
# Search npm for official packages
npm search [vendor-name]
npm search @[vendor]

# Install and inspect actual types
npm install @vendor/sdk
ls node_modules/@vendor/sdk/lib/esm/

1.2 Understand SDK Architecture​

  • Init options - What does the SDK expose? How is the platform connection established?
  • Call ordering - When does data arrive? Is it pushed (webhooks, callbacks) or pulled (polling, intercepting)? What are the timing implications?
  • Data format - What does the raw event/request look like? Headers, body structure, query params, authentication tokens?
  • Identity signals - Does the external system provide user IDs, session IDs, device IDs? How are they delivered (headers, cookies, body fields)?
  • Consent - Does the platform have consent signals? How are they communicated?

1.3 Identify All Data Entry Points​

Go beyond just the primary event payload. Most external systems provide multiple data channels:

Data ChannelExampleswalkerOS Handling
Event payloadRequest body, DOM event dataDefault push()
Headers/metadataAuth tokens, content-type, origincontext or user
Query paramsUTM parameters, tracking IDsdata or context
Platform contextCloud function metadata, Lambda ctxsource or custom
IdentityUser ID, session ID, device IDuser
Consent signalsOpt-in/out flags, consent stringconsent

1.4 Check Existing Patterns​

Review similar sources in the codebase:

# List existing sources
ls packages/web/sources/
ls packages/server/sources/

# Reference implementations
# - dataLayer: DOM-based, array interception
# - express: HTTP middleware
# - fetch: Generic HTTP handler (simplest server pattern)
# - gcp: Cloud Functions specific

Gate: Research Complete​

  • Official SDK installed and types inspected (or HTTP API documented if no SDK exists)
  • All data entry points listed with their format
  • Init options and call ordering understood
  • Identity and consent signal delivery documented
  • Raw event/request structure mapped to walkerOS event fields

Checkpoint: Research Review (Optional)​

If working with human oversight, pause here to confirm:

  • SDK integration approach correct?
  • Data capture strategy makes sense for the use case?
  • Any platform quirks or limitations to handle?

Phase 2: Classify Source Type​

Goal: Understand what the source captures and how it delivers data, which determines implementation complexity.

2.1 Source Categories​

CategoryDescriptionMapping NeededExample Sources
TransformationConverts external event format to walkerOSEssential - must map fieldsdataLayer, fetch
TransportReceives events from a specific platformStructural - platform unwrapgcp, aws, express
InterceptionIntercepts existing data flowsVaries - depends on data formatdataLayer, CMP sources

2.2 Determine Integration Approach​

ApproachWhen to usePattern
Platform SDK as hostSDK provides typed request/responseUse SDK types, wrap handler in walkerOS source
DOM interceptionCapture browser-side eventsListen to DOM events, intercept arrays/globals
HTTP handlerGeneric webhook/API receiverParse request, extract events, forward to collector
Callback/event listenerPlatform provides event emitterRegister listener, transform events, forward to collector

Prefer the vendor SDK - it provides typed request/response objects and handles platform specifics. Raw HTTP parsing is a fallback when no SDK exists.

When using the vendor SDK:

  • Import types from the SDK package directly
  • Use SDK request/response types for handler signatures
  • Let the SDK handle platform-specific parsing (body parsing, header extraction)

Gate: Classification Complete​

  • Source category identified (transformation / transport / interception)
  • Integration approach chosen (SDK / DOM / HTTP / callback)
  • Know what the source captures and how it delivers data

Phase 3: Create Input Examples (BEFORE Implementation)​

Mandatory. Examples are the test fixtures for Phase 8. Define expected trigger / in / out triples FIRST - start with the end result in mind. Without examples, you cannot test. Even for simple sources, step examples are the single source of truth for tests, simulations, and documentation.

Authoritative pattern: See using-step-examples for the Three Type Zones, createTrigger contract, and CI integration. This skill reuses that contract - do not diverge.

3.1 Scaffold Directory Structure​

mkdir -p packages/<platform>/sources/[name]/src/examples
mkdir -p packages/<platform>/sources/[name]/src/{schemas,types}

3.2 Required Files (3-4 files)​

All reference sources in the monorepo use this exact layout in src/examples/. Match it - no inputs.ts, outputs.ts, requests.ts, or standalone mapping.ts.

FileRequired?Purpose
examples/step.tsyesFlow.StepExample entries with trigger / in / out triples
examples/trigger.tsyescreateTrigger implementation following Trigger.CreateFn
examples/index.tsyesBarrel exports: env (if present), step, createTrigger
examples/env.tsif neededMock env for platform deps (browser window/document, express, etc.)

env.ts is included whenever the source touches platform globals or injected deps - all web sources and every server source that wraps a platform SDK ship one. Sources whose tests drive the collector entirely through trigger.ts (e.g. web/sources/session) may omit it. When in doubt, include it.

The old inputs.ts / outputs.ts / requests.ts / mapping.ts files are gone

  • their contents now live inline in each Flow.StepExample entry in step.ts.

3.3 Three Type Zones for Sources​

Sources are the inverse of destinations in the Three Type Zones model:

ZoneSource semantics
triggerHow to simulate the invocation (HTTP method, DOM event type, cloud event)
inExternal trigger content - HTTP request, DOM HTML, SDK payload (NOT a walkerOS event)
outThe walkerOS event(s) the source should emit (WalkerOS.Event)

Where a destination does WalkerOS.Event → vendor output, a source does external content → WalkerOS.Event. Read using-step-examples before authoring entries.

3.4 Typing Rules (strict)​

No any. Every example value must be explicitly typed.

  • trigger uses the local source trigger type or a platform-native type (e.g. 'load' | 'click' for DOM, HTTP method strings for server sources).
  • in uses the vendor / platform SDK types imported from the official package whenever available (Express Request, Fetch Request, API Gateway APIGatewayProxyEvent, Lambda Context, GCP CloudEvent, etc.). Do not invent local request types when the platform publishes them.
  • out uses WalkerOS.Event (or DeepPartialEvent for fragments).
  • Step entries are typed Flow.StepExample from @walkeros/core.
  • Mock env is typed against the source's local Env type from ../types.
  • createTrigger is typed as Trigger.CreateFn<Content, Result> - the Content and Result generics come from the source's own types module.

3.5 Code Template - examples/step.ts​

import type { Flow } from '@walkeros/core';

// One step example per captured trigger / input shape.
// `trigger` tells createTrigger how to simulate the invocation.
// `in` is the platform-specific content (HTTP request, DOM HTML, SDK payload) -
//      typed against the platform SDK's published types where available.
// `out` is the walkerOS event the source is expected to emit.

// Set `title` + `description` for public examples; mark test-only fixtures
// with `public: false`. See
// [walkeros-using-step-examples](../walkeros-using-step-examples/SKILL.md).

export const pageView: Flow.StepExample = {
  trigger: {
    type: 'load',
    options: {
      url: 'https://example.com/docs',
      title: 'Documentation',
    },
  },
  in: '', // no external content - DOM-driven trigger
  out: {
    name: 'page view',
    data: { domain: 'example.com', title: 'Documentation', id: '/docs' },
    entity: 'page',
    action: 'view',
    trigger: 'load',
    source: {
      type: 'browser',
      platform: 'web',
      url: 'https://example.com/docs',
    },
  },
};

// Server example: HTTP POST carrying a walker event payload.
export const orderComplete: Flow.StepExample = {
  trigger: { type: 'POST' },
  in: {
    method: 'POST',
    path: '/collect',
    body: { name: 'order complete', data: { id: 'ORD-123', total: 149.97 } },
  },
  out: {
    name: 'order complete',
    data: { id: 'ORD-123', total: 149.97 },
    entity: 'order',
    action: 'complete',
  },
};

3.6 examples/index.ts (barrel)​

export * as env from './env'; // omit if the source has no env.ts
export * as step from './step';
export { createTrigger, trigger } from './trigger';

3.7 examples/trigger.ts - createTrigger​

Every source exports a createTrigger following the unified Trigger.CreateFn<Content, Result> interface. It simulates real-world invocations from the outside - no source instance access, full blackbox.

import type { Trigger } from '@walkeros/core';
import { startFlow } from '@walkeros/collector';

export const createTrigger: Trigger.CreateFn<Content, Result> = async (
  config,
) => {
  let flow: Trigger.FlowHandle | undefined;

  const trigger: Trigger.Fn<Content, Result> =
    (type?: string) => async (content) => {
      if (!flow) {
        const result = await startFlow(config);
        flow = { collector: result.collector, elb: result.elb };
      }
      // Package-specific: make real HTTP request, inject DOM, dispatch SDK call.
      // Return the Result type declared by this source.
      return /* ... */;
    };

  return {
    get flow() {
      return flow;
    },
    trigger,
  };
};

Reference implementations:

  • Browser: packages/web/sources/browser/src/examples/trigger.ts - DOM injection + native event dispatch
  • Session: packages/web/sources/session/src/examples/trigger.ts - no env.ts, trigger drives collector directly
  • Express: packages/server/sources/express/src/examples/trigger.ts - real HTTP fetch() to running server
  • CMP (Usercentrics): packages/web/sources/cmps/usercentrics/src/examples/trigger.ts - dispatches CMP events, asserts on collector consent state
  • Fetch (function handler): packages/server/sources/fetch/src/examples/trigger.ts - accesses source instance via collector.sources, calls source.push() with platform-native Request
  • AWS Lambda: packages/server/sources/aws/src/lambda/examples/trigger.ts - constructs API Gateway event + Lambda context
  • GCP CloudFunction: packages/server/sources/gcp/src/cloudfunction/examples/trigger.ts - synthesizes mock req/res (matching GCP Functions Framework)

3.8 Test Fixture Contract (hard rule)​

The examples authored here are the Phase 8 test fixtures. No parallel fixtures allowed.

  • src/index.test.ts MUST iterate examples via it.each(Object.entries(examples.step)).
  • Tests must NOT contain hardcoded trigger payloads, HTTP requests, DOM HTML, or expected events.
  • If a test needs a value that is not in examples.step, add it to step.ts first, then consume it from the test.
  • Tests invoke examples.createTrigger(config) and dispatch each example's trigger.type + in content, asserting the collector receives out.

See the canonical source tests under packages/web/sources/browser/src/index.test.ts and packages/server/sources/express/src/index.test.ts.

3.9 Export via dev.ts​

export * as schemas from './schemas';
export * as examples from './examples';

Phase 3 Acceptance Checklist​

  • src/examples/step.ts - one Flow.StepExample per captured trigger / input shape, typed trigger / in / out
  • src/examples/trigger.ts - exports createTrigger typed as Trigger.CreateFn<Content, Result>
  • src/examples/index.ts - barrel exports step, createTrigger, and env (when present)
  • src/examples/env.ts - included whenever the source touches platform globals or injected deps; typed against local Env; no real network
  • No standalone inputs.ts, outputs.ts, requests.ts, or mapping.ts files
  • All platform / SDK types imported from the official package - no any, no reinvented request / response shapes
  • src/index.test.ts iterates examples.step via it.each(Object.entries(...))
  • Tests contain zero hardcoded payloads, requests, or expected events - everything flows from examples.step
  • Edge cases included (minimal input, invalid input)
  • npm run build passes - examples compile against published types
  • Each example traces: trigger + in → source push → matches out

Phase 4: Define Mapping​

Goal: Document transformation from input format to walkerOS events.

Mapping lives inside each Flow.StepExample entry in step.ts - no separate mapping.ts file. Sources typically carry the mapping either in the source's own settings (see dataLayer for an example) or inline via the trigger → in → out relationship: the in content is the raw platform payload; the out is the walkerOS event after the source's transformation.

Verify Mapping Logic​

For each entry in step.ts, trace:

Input: examples.step.pageView.trigger + examples.step.pageView.in
  ↓ createTrigger dispatches the trigger
  ↓ Source receives platform content, runs its transformation
  ↓ Source calls env.push / collector.push
Output: Should match examples.step.pageView.out (a WalkerOS.Event)

Gate: Mapping Verified​

  • Step examples cover the main input event types
  • Each example name transforms to correct walkerOS event name
  • Each example traces correctly from (trigger, in) to out

Phase 5: Scaffold​

Template sources:

  • Web: packages/web/sources/dataLayer/
  • Server: packages/server/sources/fetch/ (simplest pattern)
cp -r packages/server/sources/fetch packages/server/sources/[name]
cd packages/server/sources/[name]

# Update package.json: name, description, repository.directory

Directory structure:

packages/server/sources/[name]/
├── src/
│   ├── index.ts           # Main export
│   ├── index.test.ts      # Tests against examples
│   ├── dev.ts             # Exports schemas and examples
│   ├── examples/
│   ├── schemas/
│   └── types/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md

Transformer Chain Integration​

Sources can wire to transformer chains via next in the init config:

sources: {
  mySource: {
    code: sourceMySource,
    config: { settings: { /* ... */ } },
    next: 'validate'  // Events go through validator before collector
  }
}

Phase 6: walkerOS.json Convention​

Every walkerOS package ships a walkerOS.json file for CDN-based schema discovery.

Add walkerOS field to package.json​

{
  "walkerOS": { "type": "source", "platform": "web" },
  "keywords": ["walkerOS", "walkerOS-source", ...]
}

Use buildDev() in tsup.config.ts​

Replace buildModules({ entry: ['src/dev.ts'] }) with buildDev():

import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),

This auto-generates dist/walkerOS.json from your Zod schemas at build time.

Hints (Optional)​

If your source has capabilities, behaviors, or troubleshooting patterns not obvious from schemas alone, add hints. See walkeros-writing-documentation skill for full guidelines.

Create src/hints.ts:

import type { Hint } from '@walkeros/core';

export const hints: Hint.Hints = {
  'capture-timing': {
    text: 'Describes when events are captured. See settings schema for options.',
    code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
  },
};

Export from src/dev.ts:

export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';

Guidelines:

  • Expand awareness - describe capabilities ("supports X, Y, Z"), don't prescribe one path
  • Reference schemas and examples, don't duplicate them
  • Verify every claim against actual implementation before publishing
  • Key naming: kebab-case, group with prefixes (auth-*, capture-*, troubleshoot-*)
  • Most sources don't need hints - schemas and examples cover the common case

Gate: Convention Met​

  • walkerOS field in package.json with type and platform
  • buildDev() in tsup.config.ts
  • Build generates dist/walkerOS.json
  • Keywords include walkerOS and walkerOS-source

Runtime-only npm dependencies​

If your package wraps a third-party npm dep that cannot be ESM-bundled (uses __dirname, ships a .node binary, etc.), declare it under walkerOS.bundle.external in your package.json. See walkeros-using-cli → Bundle externals for the complete contract.

Phase 7: Implement​

Now write code to produce the outputs defined in Phase 3.

Template Files​

FilePurposeTemplate
types/index.tsType definitionstypes.ts
schemas/index.tsZod validation schemasschemas.ts
index.tsMain sourceindex.ts

Key Patterns​

  1. Context destructuring: Extract config, env, logger, id from context
  2. Schema validation: Use Zod schemas to validate settings and provide defaults
  3. Forward to collector: Call env.push() to send events to the collector
  4. Platform deps via env: All platform dependencies (window, document, express, cors) must go through env with fallback to globals/imports: env.express ?? express. This enables testing without mocking globals.
  5. Error logging: Use logger?.error() for errors only, not routine operations
  6. Return Source.Instance: Return { type, config, push } object
  7. Optional destroy method: Implement if the source holds resources (HTTP servers, timers, connections) that need cleanup on shutdown

Gate: Implementation Compiles​

  • npm run build passes
  • npm run verify:touched -- <source-name> passes (L1: typecheck + lint + test)

Adding setup (optional)​

A source package can implement an optional setup() function to provision external resources idempotently: Pub/Sub subscriptions, webhook registrations on upstream platforms, queue declarations, SQS queues, polling cursors, inbound API keys. Setup runs only when an operator explicitly types walkeros setup source.<name>. The runtime never auto-invokes it from init(), push, or destroy().

The framework provides the slot, the CLI command, and a resolveSetup helper. The package owns: what setup means, idempotency, error handling, return value.

For background on how setup fits the source lifecycle, see understanding-sources.

Types​

// types/index.ts
import type { CoreSource } from '@walkeros/core';

export interface Settings {
  /* runtime push settings */
}
export interface InitSettings {
  /* one-time init settings */
}
export interface Mapping {
  /* event extraction mapping */
}
export interface Env {
  /* injected platform deps (SDK clients, request handlers, etc.) */
}

// The package's own setup options interface.
// Becomes the U slot of Types; surfaces as `config.setup: boolean | Setup` for users.
export interface Setup {
  // package-specific provisioning options
  // e.g. for Pub/Sub source: subscriptionName, ackDeadlineSeconds, filter
  subscriptionName?: string;
  ackDeadlineSeconds?: number;
}

export type Types = CoreSource.Types<
  Settings,
  Mapping,
  Env,
  InitSettings,
  Setup
>;

Implementation​

// setup.ts
import type { CoreSource, SetupFn } from '@walkeros/core';
import { resolveSetup } from '@walkeros/core';
import type { Setup, Types } from './types';

const DEFAULT_SETUP: Setup = {
  ackDeadlineSeconds: 60,
};

export const setup: SetupFn<
  CoreSource.Config<Types>,
  CoreSource.Env<Types>
> = async ({ config, env, logger }) => {
  const options = resolveSetup(config.setup, DEFAULT_SETUP);
  if (!options) return; // config.setup is false or unset

  // Package-specific provisioning, idempotent.
  // Returning a structured object (e.g. { subscriptionCreated: true })
  // makes that data available to operators via `walkeros setup ... | jq`.
};

Wire it in your default export:

// index.ts
import { setup } from './setup';

export default {
  type: 'my-source',
  init: /* ... */,
  setup,
};

When to implement​

Implement setup() when your source needs first-time provisioning of upstream resources before events can be received: Pub/Sub subscriptions bound to a topic, webhook registrations on upstream SaaS platforms (Stripe, GitHub, Shopify), SQS queue declarations, message broker bindings, polling cursors. Skip it when your source only listens on an HTTP port the runtime already owns or intercepts data already present (DOM events, dataLayer pushes).

Contract​

  • Triggered only by walkeros setup <kind>.<name>. Never by runtime push, init, or destroy.
  • Idempotency is your responsibility. Re-running setup against a fully provisioned environment must be a safe no-op. Use try-create-catch-409 on REST APIs, IF NOT EXISTS on SQL, native idempotent operations where available. The framework does not retry, track state, or detect drift.
  • Return structured data from setup() when useful for operator scripting. The CLI emits non-undefined return values as JSON to stdout.
  • For packages where setup: true (boolean form) is meaningless because mandatory fields have no safe defaults (e.g., GitHub webhook webhookUrl, Pub/Sub source topicName), reject the boolean form with a clear runtime error listing required fields:
if (config.setup === true) {
  throw new Error(
    'github-webhook source setup requires explicit options: ' +
      '{ webhookUrl, repo, events }. There is no safe default.',
  );
}

Phase 8: Test Against Examples​

Tests verify implementation against the examples from Phase 3. If examples are incomplete, tests will be incomplete.

See testing-strategy for the shared env / dev-examples conventions this phase depends on.

Verify implementation produces expected outputs.

Test Template​

Use the test template: index.test.ts. Canonical references:

  • packages/web/sources/browser/src/index.test.ts
  • packages/server/sources/express/src/index.test.ts

Key Test Patterns​

  1. it.each(Object.entries(examples.step)) is mandatory - one iteration per step example. Do not write per-feature tests with hand-rolled payloads.
  2. Drive via createTrigger - construct the trigger with startFlow config, then dispatch each example's trigger.type + in content.
  3. Use createSourceContext() helper for any direct context construction.
  4. Zero hardcoded payloads - every trigger type, request body, DOM HTML, and expected event comes from examples.step or examples.env. If you need something new, add it to examples first.
  5. Test error paths - verify graceful error handling and logging for invalid input (add an error example to examples.step if needed).

Gate: Tests Pass​

  • npm run verify:touched -- <source-name> passes (L1)
  • Tests iterate via it.each(Object.entries(examples.step))
  • Tests contain no hardcoded payloads, requests, or expected events
  • Every assertion reads from examples.step[...].out
  • Invalid input handled gracefully (no crashes)

Phase 9: Document​

Follow the writing-documentation skill for:

  • README structure and templates
  • Example validation against apps/quickstart/
  • Quality checklist before publishing

Key requirements for source documentation:

  • Input format table documenting expected fields
  • Event name mapping table (source format → walkerOS format)
  • Configuration options table
  • Working code example with imports
  • Installation instructions

Validation Checklist​

Beyond understanding-development requirements (build, test, lint, no any):

  • dev.ts exports schemas and examples
  • Examples include edge cases (minimal, invalid input)
  • Invalid input returns gracefully (no crashes, clear error)
  • Tests use examples for assertions (not hardcoded values)
  • walkerOS.json generated at build time
  • walkerOS field in package.json

Reference Files​

WhatWhere
Web templatepackages/web/sources/dataLayer/
Server templatepackages/server/sources/fetch/
Source typespackages/core/src/types/source.ts
Event creationpackages/core/src/lib/event.ts
💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)