Observability

Every Lunora function call produces telemetry across all three OpenTelemetry signals: an RPC event (one per query/mutation/action — path, duration, ok/error, shard, fan-out), log events (ctx.log), spans (ctx.trace), and metrics (ctx.metrics). A sink receives them.

Locally, none of it needs setup: the Studio groups error events into an Issues view, streams log lines in Logs, and renders ctx.trace waterfalls in Traces. To watch a deployed app you point a sink at an OpenTelemetry collector — your own, a vendor's, or the Lunora cloud — and the worker and any container ship the same telemetry over one protocol.

Sinks

A sink is passed to createWorker as the observability option. Lunora ships several; combine them with combineSinks.

import { analyticsEngineSink, combineSinks, otlpSink } from "@lunora/runtime";

export default createWorker({
    // …schema, functions…
    observability: combineSinks(
        otlpSink({ endpoint: env.LUNORA_OTLP_ENDPOINT, token: env.LUNORA_OTLP_TOKEN }),
        analyticsEngineSink({ dataset: env.ANALYTICS }),
    ),
});

Sinks that carry RPC events accept onlyErrors: true to drop successful RPCs and export only failures (pipelineLogSink is log-only, so it has no such option; log events always pass through regardless). All four network/log sinks — otlpSink, webhookSink, pipelineLogSink, plus sentrySink when you wire its captureLog — forward ctx.log lines; webhookSink takes a transformLog redactor mirroring its RPC transform.

:::caution[Upgrading a webhookSink] webhookSink now ships ctx.log lines (message and structured fields, which may carry user input) in addition to RPC events — previously it shipped neither. Two consequences for an existing config:

• onlyErrors does not gate log lines (only RPC events); every ctx.log call egresses to the endpoint.
• The RPC transform redactor does not cover log lines — add a transformLog (same fail-closed contract) if you scrub PII before it leaves the worker.

Set transformLog (or point the sink at a trusted endpoint) before upgrading if your handlers log user data. sentrySink (opt-in via captureLog) and otlpSink don't have this expansion. :::

otlpSink

otlpSink({
    endpoint: env.LUNORA_OTLP_ENDPOINT, // required — the collector base URL
    token: env.LUNORA_OTLP_TOKEN, // optional — sent as `Authorization: Bearer <token>`
    headers: { "x-lunora-deployment": env.LUNORA_DEPLOYMENT_ID }, // optional correlation headers
    serviceName: "my-app", // optional — the `service.name` resource attribute (default `"lunora"`)
    onlyErrors: false, // optional — export only error spans
});

Read endpoint/token from the environment so the platform can inject them at deploy time with no code change. Each event is one fire-and-forget POST; on a Worker it is registered with waitUntil so it outlives the response.

pipelineLogSink

Persist every ctx.log line durably to a Cloudflare Pipeline → R2, giving you a queryable log store (read back with R2 SQL) in your own account — no cloud required. It is the durable counterpart to otlpSink (which streams to a collector). Only log lines are stored; RPC-span metrics belong in analyticsEngineSink.

import { pipelineLogSink } from "@lunora/runtime";

pipelineLogSink({ pipeline: env.LOG_PIPELINE });

Each record carries message, level, functionPath, fields, traceId, spanId, shardKey, userId, and ts. The send is registered with the request's waitUntil so it survives isolate teardown.

Reading the durable archive back

pipelineLogSink only writes. To read those records back — no cloud involved — use createPipelineLogReader, which builds a safe, keyset-paginated R2 SQL query over the Iceberg table the Pipeline lands in, or run the lunora logs --durable CLI command.

This needs operator setup — it is not code-only. The reader queries a table that only exists once you have wired the Pipeline to an R2 Data Catalog (Iceberg) table whose schema matches the written columns below, and provided read credentials. Until then the CLI fails closed with a clear message.

1. Create the destination table with a column per written field. The reader owns these names (its DEFAULT_LOG_COLUMNS), matching exactly what the sink writes:

If your table types fields as a string column (so R2 SQL can query it), turn on serializeFields so the sink stores it as a JSON string; the reader parses it back to an object on read:

pipelineLogSink({ pipeline: env.LOG_PIPELINE, serializeFields: true });

Renamed a column in your Iceberg schema? Pass a columnMap to realign the reader without touching the writer.

2. Provide the R2 SQL credentials as env vars (also used by ctx.r2sql):

• R2_SQL_ACCOUNT_ID — the Cloudflare account that owns the bucket/catalog
• R2_SQL_TOKEN — an API token scoped to R2 SQL read + R2 Data Catalog + R2 storage
• R2_SQL_BUCKET — the R2 bucket (warehouse) the catalog runs against

3. Read it back, from code:

import { createPipelineLogReader } from "@lunora/runtime";
import { createR2Sql } from "@lunora/bindings/r2sql";

const reader = createPipelineLogReader(createR2Sql({ accountId, apiToken, bucket }), { namespace: "default", table: "logs" });

const page = await reader.query({ minLevel: "warn", sinceTs: Date.now() - 3_600_000, limit: 200 });
// page.rows — newest first; page.nextCursor — pass as `cursor` for the next page

or from the CLI:

# newest 200 warn+ lines from the last hour, as a table
lunora logs --durable --table logs --namespace default --min-level warn --limit 200

# one JSON object per line, filtered to a function-path prefix (pipeable to jq)
lunora logs --durable --table logs --function-prefix "messages:" --ndjson

# resume the next page with the ts the previous run printed
lunora logs --durable --table logs --cursor 1737460000000

The reader paginates by keyset on ts DESC (WHERE ts < cursor), not OFFSET, so deep pages stay cheap over Iceberg. Filters (sinceTs/untilTs, level, minLevel, functionPathPrefix, traceId, shardKey, userId) are all inlined as escaped SQL literals — user values can never inject.

Structured logging with ctx.log

ctx.log spans the full OpenTelemetry severity ramp — trace, debug, info (and its log alias), warn, error, fatal — and takes either console-style values or a structured message + fields object:

export const placeOrder = mutation({
    handler: async (ctx, args) => {
        // Console-style: any number of values, joined into the message.
        ctx.log.debug("placing order", args);

        // Structured: a message plus a fields object. The fields become
        // filterable/indexable log-record attributes.
        ctx.log.info("order placed", { orderId: order._id, total: order.total });

        // Bind context once with `.with(...)`; every line inherits it
        // (per-call fields win on a key clash).
        const log = ctx.log.with({ orderId: order._id });
        log.warn("inventory low", { sku });
        log.fatal("charge failed", { code: err.code });
    },
});

The (string, object) shape is the structured form; every other shape is console-style. The rendered message and the structured fields reach the dev terminal, Workers Logs, the Studio Logs panel, and any sink; the raw positional args of a console-style call reach only the in-process onLog sink you control.

:::caution[Behavior change] A two-argument call whose second argument is a plain object — ctx.log.info("saved", user) — is now the structured form: the message is "saved" and user becomes fields, instead of being rendered into the message as saved {…}. Calls with a non-object second argument, or three or more arguments, are unchanged. If you relied on the object being folded into the message text, pass it as a third argument (ctx.log.info("saved", "-", user)) or pre-render it. :::

Tracing sub-operations with ctx.trace

Every dispatch is already one span, named after the function path. That tells you a request took 900ms; it doesn't tell you which part took 900ms. ctx.trace wraps a sub-operation so it becomes its own span nested under the request:

export const checkout = action({
    handler: async (ctx, args) => {
        const cart = await ctx.trace("cart.load", () => loadCart(args.cartId));

        // Attributes are structured like log fields, and become span attributes.
        const charge = await ctx.trace("stripe.charge", () => stripe.charges.create({ amount: cart.total }), { cartId: cart._id });

        // Nesting is explicit: the body receives a tracer bound to its own span,
        // and calling that is what makes a child.
        await ctx.trace("fulfil", async (trace) => {
            await Promise.all([trace("reserve.stock", () => reserve(cart)), trace("email.receipt", () => sendReceipt(charge))]);
        });
    },
});

The body's value is returned unchanged, and a throw is recorded as an error span and then re-thrown — this is instrumentation, never flow control. Recording is best-effort: a failing sink can't turn a working handler into a broken one.

:::note[Why the tracer is passed in, rather than nesting being implicit] It would read nicer if a bare ctx.trace inside another span's body were automatically its child. That can't be done correctly here: with Promise.all([trace("a", …), trace("b", …)]), b starts while a is still open, so an "innermost currently-open span" rule records b as a child of a rather than its sibling — and parallel fan-out is one of the main things a tracer is for. Telling "called inside a" apart from "called concurrently with a" needs AsyncLocalStorage, which Lunora's Durable Objects deliberately don't require. Passing the parent is correct in every case, and visible where it happens.

Calling ctx.trace inside a body instead of the passed tracer isn't an error — that span is just parented to the request rather than the enclosing span. :::

A span created inside a function invoked via ctx.runQuery / runMutation / runAction is attributed to the outer entrypoint's function path, since the composed call reuses its context — the same rule ctx.log follows.

Spans share the dispatch's trace id with its ctx.log lines and with any container the handler calls (the same traceparent is propagated), so one trace stitches together worker, shard, and container.

:::tip[Keep span names low-cardinality] Put the varying part in the attributes, not the name — ctx.trace("stripe.charge", …, { orderId }), never a name interpolated from the order id. A name built from an id makes every span its own group in a collector, which is exactly what attributes exist to avoid. :::

Locally, the Traces panel in the Studio renders recent waterfalls from an in-memory ring on the shard — recent activity on this instance, reset on hibernation. It is a development readout, not a trace store: for retention and cross-instance search, point otlpSink at a real collector, where each span is exported as an OTLP INTERNAL span carrying its parentSpanId.

Application metrics with ctx.metrics

The third signal, alongside logs and traces. A trace tells you what one request did; a metric tells you what a million requests did.

export const checkout = action({
    handler: async (ctx, args) => {
        const started = Date.now();

        // "How many" — summed over time.
        ctx.metrics.count("orders.placed", 1, { plan: user.plan });

        // "How many right now" — replaces the previous reading.
        ctx.metrics.gauge("cart.items", cart.items.length);

        // "What's the distribution" — percentiles, not just a mean.
        ctx.metrics.record("checkout.latency_ms", Date.now() - started);
    },
});

Each becomes an OTLP metric at POST {endpoint}/v1/metrics: a monotonic Sum, a Gauge, or a Histogram.

:::caution[Keep attributes low-cardinality] Attributes are the metric's dimensions, and every distinct combination is a separate time series. { plan: user.plan } is a handful of series; { userId } is one per user, which is how a metrics bill gets out of hand. Identifiers belong on a log line or a span — that's what they're for. :::

Nothing is pre-aggregated: one call is one exported measurement, with counter and histogram values carrying delta temporality for the collector to aggregate. That keeps the sink model identical to logs and spans, at the cost of one export per call — so in a hot loop, sum locally and record once at the end rather than calling per iteration.

Metrics have no local buffer and no Studio panel: their value is in the aggregate over time, which an in-memory ring on a hibernating instance can't represent. consoleSink prints them in dev; for anything real, point otlpSink at a collector. (The Studio's Metrics page shows framework-level per-function metrics, which are collected separately and always on.)

Container telemetry

Code running inside a container can't use a worker sink — it is a separate process. @lunora/container/otel gives it a zero-config exporter that speaks the exact same wire contract, so container spans and worker spans land in the same collector side by side.

import { createContainerTelemetry } from "@lunora/container/otel";

// Reads LUNORA_OTLP_ENDPOINT / LUNORA_OTLP_TOKEN from the container env.
const telemetry = createContainerTelemetry();

// Time a unit of work — records an ok span, or an error span if it throws.
const result = await telemetry.trace("transcode", () => transcode(job), { jobId: job.id });

telemetry.emitLog({ level: "info", message: "done", attributes: { jobId: job.id } });

// Before the process exits, flush any in-flight sends.
await telemetry.flush();

With no endpoint resolvable the exporter is a silent no-op (telemetry.enabled === false) — trace still runs your work, it just records nothing. The same code runs unchanged locally and in the cloud.

Each POST is bounded by timeoutMs (default 10s), so a hung collector aborts instead of pinning a send in flight and stalling flush(); failures are handed to the optional onError callback and never break the container.

Thread the endpoint/token into the container the same way any other config reaches it — declare them on defineContainer:

defineContainer({
    name: "transcoder",
    // …
    env: { LUNORA_OTLP_ENDPOINT: env.LUNORA_OTLP_ENDPOINT },
    secrets: ["LUNORA_OTLP_TOKEN"],
    // The collector host must be reachable from the container egress allow-list.
    allowedHosts: ["collector.example.com"],
});

The wire contract

Both the worker otlpSink and the container exporter conform to one contract, so any OTLP-compatible collector — and the Lunora cloud ingest — accepts either without special-casing.

Transport. OTLP over HTTP with JSON encoding (not protobuf). Two endpoints, derived from the configured base endpoint (trailing slashes are tolerated):

Headers.

The x-lunora-* headers are a convention the cloud ingest reads to route telemetry; pass them through the headers option. A collector that ignores them still accepts the payload.

Encoding. Bodies are standard OTLP ExportTraceServiceRequest / ExportLogsServiceRequest JSON. Per the OTLP/JSON spec:

• traceId is 16 random bytes as 32 lowercase hex chars; spanId is 8 bytes as 16 hex chars (the documented exception to proto3 JSON's base64 bytes).
• timeUnixNano fields are the nanoseconds since the epoch as a decimal string (Lunora works in millis, so this is the millisecond value followed by six zeros — exact).
• resource carries a service.name attribute; the instrumentation scope.name is @lunora/runtime (worker) or @lunora/container (container).
• Attribute values follow the OTLP AnyValue union — strings as stringValue, booleans as boolValue, integers as intValue (a decimal string), floats as doubleValue.

Span shape. One span per RPC event (worker) or per trace/emitSpan (container):

Worker spans carry these attributes:

Container spans carry whatever attributes you pass, plus error.type when the work throws.

Log record shape. body.stringValue is the message; severityText is the upper-cased level; severityNumber maps as:

Worker log records also carry lunora.function_path, and lunora.shard_key / lunora.user_id when known. Structured fields (below) become additional log-record attributes, and each record carries its dispatch's trace_id / span_id so a line links back to its RPC span.

Privacy

Spans carry error.type and error messages, and log records carry the rendered message — either can include user-supplied input. Point endpoint only at a collector you trust, and use onlyErrors to narrow what leaves the deployment.