Stores

Without stores, each event in a flow is processed in isolation: no shared state, no caching, no persisted data between requests. Stores add a shared memory layer that any component in your flow (transformers, destinations) can read from and write to.

A common example: serving the walker.js script from your own infrastructure. The file lives in a store (S3 or filesystem) with file: true, and a transformer reads it on each request: no hardcoded paths, no rebuilds needed to swap the file.

Stores hold one canonical value type: structured data, with binary as a first-class leaf. A shared core codec serializes that value to and from each backing. By default a store is a structured key-value store (sessions, lookups, cached responses). Set file: true on a byte-native backend (filesystem, S3, GCS) to make it persist raw bytes byte-exact instead, the mode for serving assets such as walker.js. The Sheets store is structured-only and rejects file: true. The file flag is documented in the Configuration table below.

When stores are active

Stores are initialized before any other component in the flow. Once running, they're available to transformers and destinations via $store.storeId wiring. A transformer might cache a processed result; a destination might check a store before making an external API call.

Flow startup order:
  1. Stores        ← initialized first
  2. Destinations  ← can reference stores via $store.id
  3. Transformers  ← can reference stores via $store.id
  4. Sources       ← start pushing events

On shutdown the order reverses: stores are destroyed last, after all other components have shut down, so they stay available while every other component shuts down gracefully.

Choosing a store

Store	Platform	Persistence	Speed	Best for
Filesystem	Server only	Disk (survives restarts)	Fast (local I/O)	Local dev, Docker with baked-in assets
S3	Server only	Cloud (always available)	Network I/O	Managed deployments, shared assets, hot-swap
GCS	Server only	Cloud (always available)	Network I/O	Cloud Run / GKE deployments, GCP-native
Sheets	Server only	Cloud (Google Sheets)	Slow (HTTP, rate-limited)	Demos, prototypes with a spreadsheet UI

For ephemeral, in-process caching, use the built-in cache tier on any of the stores above, no separate package needed.

To read from or write to a store from a step without wiring $store and writing $code:, use the declarative state block on a source, transformer, or destination.

Wiring stores to components

Components reference stores in their env configuration. In a bundled flow.json, write $store.storeId; the CLI resolves it, and references to undefined stores are caught at build time. In integrated mode, put the same store definition object in stores and in env; the collector replaces it with the initialized store instance at startup (matched by object identity, so a copy does not work).

Integrated

import { startFlow } from '@walkeros/collector';
import { storeS3Init } from '@walkeros/server-store-s3';
import { transformerFile } from '@walkeros/server-transformer-file';

const assets = {
  code: storeS3Init,
  config: {
    file: true,
    settings: {
      bucket: 'my-assets',
      endpoint: 'https://s3.eu-west-1.amazonaws.com',
      accessKeyId: process.env.S3_ACCESS_KEY,
      secretAccessKey: process.env.S3_SECRET_KEY,
    },
  },
};

await startFlow({
  stores: { assets },
  transformers: {
    file: {
      code: transformerFile,
      config: { settings: { prefix: '/static' } },
      env: { store: assets }, // same object as stores.assets
    },
  },
});

Bundled

{
  "stores": {
    "assets": {
      "package": "@walkeros/server-store-s3",
      "config": {
        "file": true,
        "settings": {
          "bucket": "my-assets",
          "endpoint": "https://s3.eu-west-1.amazonaws.com",
          "accessKeyId": "$env.S3_ACCESS_KEY",
          "secretAccessKey": "$env.S3_SECRET_KEY"
        }
      }
    }
  },
  "transformers": {
    "file": {
      "package": "@walkeros/server-transformer-file",
      "config": { "settings": { "prefix": "/static" } },
      "env": { "store": "$store.assets" }
    }
  }
}

Common patterns

Stores are rarely used alone. They work together with transformers to enable analytics use cases that aren't possible with stateless event processing.

Pattern	Store	Transformer	What it enables
Self-hosted tag delivery	S3 or Filesystem (file: true)	File	Serve walker.js and tracking pixels byte-exact from your own infrastructure
Cookie-free user identification	Built-in cache tier	Fingerprint	Store session hashes server-side, identify users without cookies or PII
Response caching	Built-in cache tier	Cache	Deduplicate identical requests, reduce load on downstream analytics APIs
Quota-friendly Sheets lookups	Sheets + cache tier	Any transformer	Memoize slow API reads, absorb rate limits via TTL
Local development	Filesystem	File	Serve assets from disk during development, no S3 credentials needed

Available stores

Filesystem

Local filesystem store for server flows. Reads and writes files relative to a base path with path traversal protection. Good for local development and container deployments with baked-in assets.

Learn more →

S3

S3-compatible object storage. Works with AWS S3, Cloudflare R2, Scaleway, DigitalOcean Spaces, Backblaze B2, MinIO, and any S3-compatible provider. The recommended store for managed cloud deployments.

Learn more →

GCS

Google Cloud Storage with zero runtime dependencies. Built-in auth supports Application Default Credentials (ADC) on Cloud Run / GKE and explicit service account JSON for non-GCP environments. The recommended store for GCP-native deployments.

Learn more →

Sheets

Google Sheets store with zero runtime dependencies. Row-per-key storage using the Sheets v4 REST API. Designed for demos and small prototypes where the spreadsheet is the operator-facing UI. Rate-limited; pair with the cache tier for any non-trivial throughput.

Learn more →

Caching on stores

Wrap any of the stores above with a read-through, write-through cache tier by setting cache on the store declaration. The default tier uses the collector's built-in in-memory LRU. Compose multi-tier chains via cache.store.

Learn more →

The Store interface

Unlike sources, transformers, and destinations, stores are passive: no event chains, no push functions. Every store implements the same operations:

interface Store.Instance {
  type: string;
  config: Store.Config;
  get(key: string): StoreValue | undefined | Promise<StoreValue | undefined>;
  set(key: string, value: StoreValue, ttl?: number): void | Promise<void>;
  delete(key: string): void | Promise<void>;
  destroy?(): void | Promise<void>;
}

Stores can be synchronous or asynchronous; the interface supports both. In integrated mode, initialized stores are also exposed on the collector instance:

collector.stores.assets.set('key', 'value');
const value = await collector.stores.assets.get('key');

Setup lifecycle

Stores may implement an optional setup() lifecycle for one-time, operator-time provisioning, for example creating an S3 bucket. Setup runs only when an operator explicitly invokes walkeros setup store.<name>; the runtime never auto-invokes it. Opt in via config.setup in the flow config (true or an object).

See Setup lifecycle for the full concept, and the walkeros setup CLI command for the operator-facing command.

Configuration

These fields are available on every store, regardless of package. They wrap the package-specific settings field, which is documented on each store's page.

Property	Type	Description	More
settings	Store.Settings	Implementation-specific configuration
credentials	Store.Credentials	Optional credentials (store-defined shape)
env	Store.Env	Environment dependencies (platform-specific)
id	string	Store instance identifier (defaults to store key)
logger	Logger.Config
setup	boolean | object	One-time setup options applied during store registration (boolean enables defaults, object configures specifics)
file	boolean	Persist values as raw bytes (byte-exact), bypassing the structured codec. Default false (structured). Set true only on byte-native stores for asset serving; sheets rejects it.