defuss-dbsonl

defuss-dbsonl

v0.6.0MIT licensed, ESM and CommonJS, TypeScript declarations

Transactional key-value storage for DSON

defuss-dbsonl runs on Node.js, Bun and in dedicated browser workers. Batches and transactions publish all of their changes or none, snapshots keep reading the state they opened on, and several processes or workers can share one database.

DSON is typed JSON that keeps Map and SetDate and RegExpbiginttyped arraysundefined and NaNcyclic references

defuss-dbsonl adds atomic batchesserializable transactionsnon-blocking snapshotsmulti-process writesper-key conflict checksexplicit compaction

The demos run the packaged library in your tab. Three use memory storage; one keeps a counter in your browser's origin private file system (OPFS).

import { openDbsonl } from 'defuss-dbsonl';
import { createFileBackend } from 'defuss-dbsonl/server';
const db = await openDbsonl<number>(
  await createFileBackend('./data'), 'accounts', { create: true });
const before = await db.beginRead(); // old state
await db.transaction(async tx => {
  const alice = (await tx.get('alice')).value!;
  const bob = (await tx.get('bob')).value!;
  tx.put('alice', alice - 20); // staged only
  tx.put('bob', bob + 20);
}); // resolves once both are committed
await db.get('alice'); // new balance from here on
await before.get('alice'); // still the old balance
await before.close();
Both writes become visible together when the awaited transaction resolves; the snapshot before keeps reading the old balances.
defuss-dbsonl against Redis 8.10

Faster than Redis in 12 of 16 pairings

Four workloads, each on Node.js and on Bun, flushed to disk and in memory. defuss-dbsonl runs in the measuring process; Redis 8.10.3 answers over loopback TCP through the node-redis client. A Redis transaction is WATCH, MULTI and EXEC; a defuss-dbsonl transaction is transaction().

12 of 16pairings in which defuss-dbsonl took less time than Redis; 0.4.0 did in 4
18×faster with four writers committing to disk on Bun: 0.17 ms against 3.0 ms per commit
3.6×faster point reads in memory on Node.js: 12 µs against 42 µs
4 of 16went to Redis, by up to 2.2×: a lone writer flushing to disk on Node.js, and atomic batches except on disk under Bun
Runtime
Storage
1.2 times faster
Point read32 µs against 39 µs
8.8 times faster
Transaction428 µs against 3.8 ms
18 times faster
Four writers at once166 µs against 3.0 ms
2.0 times faster
Batch of 1000 writes2.7 ms against 5.3 ms

Right of 1× defuss-dbsonl was faster; each step on the scale doubles. Times are defuss-dbsonl's, then Redis's.

Flushed to disk

Durable commits against Redis with appendfsync always. The line spans the gap between the two engines; the scale is logarithmic.

In memory

The memory backend against Redis without persistence. Same scale, so the two charts compare.

All 16 pairings as tables
Durable, per operationdbsonl, Node.jsRedis, Node.js clientdbsonl, BunRedis, Bun client
Point read27 µs47 µs32 µs39 µs
Transaction, read 2 keys and write 27.6 ms3.8 ms0.43 ms3.8 ms
Same, 4 writers at once, per commit1.9 ms3.0 ms0.17 ms3.0 ms
Atomic batch of 1000 writes13.4 ms6.6 ms2.7 ms5.3 ms
In memory, per operationdbsonl, Node.jsRedis, Node.js clientdbsonl, BunRedis, Bun client
Point read12 µs42 µs20 µs44 µs
Transaction, read 2 keys and write 20.082 ms0.16 ms0.093 ms0.21 ms
Same, 4 writers at once, per commit0.063 ms0.076 ms0.069 ms0.084 ms
Atomic batch of 1000 writes5.4 ms2.5 ms2.2 ms1.4 ms

Median of three make bench-redis runs on an Apple M5 Max with Node.js 24.22.1 and Bun 1.4.3, each the median of seven rounds over 1000 records, under a load average of 5.5 to 10.9. Raw samples.

What you get

Values keep their types

Values are stored as DSON, typed JSON from defuss-dson: Map, Set, Date, RegExp, bigint, typed arrays, undefined, NaN and cyclic references come back as they went in. JSON turns a Map into {} and a date into a string, and throws on a bigint.

Readers never block writers

A snapshot pins one immutable root of the tree. Commits keep publishing while it reads, and it sees none of them until you open a new one.

All changes or none

A batch or a transaction becomes visible as one revision. A transaction fails with CONFLICT when a key it read changed, appeared or disappeared after its snapshot, which rules out write skew; writers on other keys commit without retrying.

Opens fast, however much it holds

A log-based store replays its log on open, so startup slows as data grows. defuss-dbsonl reads a small header that points at the current tree, not the stored data; the benchmark reopened a database in 0.62 ms on Node.js. Memory is capped by the page cache budget, not by the number of keys.

Shared by processes and tabs

Servers coordinate through kernel file locks, browser workers through Web Locks. A process that dies releases its locks, and no lock expires on a timer, so a paused writer never loses ownership.

Corruption fails closed

Every frame carries a CRC32C checksum, and verify() walks every reachable value and index page. Nothing repairs data by guessing, because a guessed repair can turn damage into silent data loss.

Where it fits

The storage and API layer for defuss-dataview

defuss-dbsonl is designed as the backing store and API of defuss-dataview. It turns the byte I/O of defuss-storage into a transactional key-value store and encodes every value with defuss-dson.

defuss-dataviewFilters, sorting, paging and tree traversal for table and tree grids
defuss-dbsonl this packageTransactions, snapshots, the B+tree index, locks and compaction
defuss-dsonTyped JSON in frames with CRC32C checksums
defuss-storageBounded byte I/O, one API for every medium
  • Native file system
  • OPFS and directory handles
  • Memory
Live demo

Four guarantees you can test here

Each card runs defuss-dbsonl in this tab, built from the same package entries you install. The demos send no network requests.

A snapshot keeps its root

Isolation without read locks

The left side reads the latest commit. The right side reads a snapshot, opened when the page loaded or when you last opened a new one. Each transfer is a transaction that moves 20 from alice to bob in one commit.

Latest, revision 0

alice
0
bob
0

Snapshot, revision 0

alice
0
bob
0

A stale transaction fails

Optimistic concurrency, no silent retry

Transaction A reads visits and waits. Writer B commits in between. When A publishes, the key it read has changed, so it fails with CONFLICT. The library does not retry, because a retry would run your callback and its side effects again.

0visits

    Values come back as they went in

    DSON instead of JSON

    The button stores one object with a date, bytes, a bigint, an undefined property and a reference to itself, then reads it back and checks each field.

    FieldRead backResult
    Not stored yet.

    It survives a reload

    OPFS in a dedicated worker

    This counter lives in a database in your browser's origin private file system. Reload the page and it keeps its value. Open the page in a second tab: both tabs' workers write the same database, and Web Locks order their commits.

    0revision 0

    Starting the worker.

    Building blocks

    Six parts, one store

    Copy-on-write B+tree

    Fanout 32. A commit rewrites the path from leaf to root and never changes a written page.

    Immutable roots

    Every revision is a root. A snapshot pins one and reads it while new roots are published.

    Two head slots

    Two checksummed 4 KiB slots name the current root. Writers alternate between them; the newest valid slot wins.

    DSON frames

    Values and index pages are DSON frames with a length, a header CRC32C and a payload CRC32C.

    Bounded page cache

    Decoded index pages stay in a second-chance cache limited by bytes and page count. Values are always read fresh.

    Lock order

    Commit lock, then generation pin, then I/O lock. Kernel locks on servers, Web Locks in browsers, no timeouts.

    How it works

    One commit, crash-safe at every step

    A commit writes next to the old data, never over it, and switches with one small write at the end.

    1. Lockone writer at a time
    2. Appendvalues and index pages
    3. Flushbytes reach the device
    4. Publishthe older of two head slots
    Crash before step 4 The previous head still points at a complete tree.
    Design choices

    What each choice costs

    Each card names the design in use, the obvious alternative, why the alternative lost and what the choice costs.

    Finding keys

    Persistent B+tree with fanout 32 and a bounded page cache

    Instead of an in-memory key map rebuilt from a log

    Why
    Memory does not grow with the number of keys, and opening needs no replay.
    Cost
    A point read whose page is not cached reads from disk.

    Readers and writers

    Immutable roots, pinned per snapshot

    Instead of read locks that block commits

    Why
    A slow reader cannot stall a writer.
    Cost
    Old generations stay on disk until their readers close.

    Atomic publication

    Two checksummed head slots, written alternately

    Instead of writing a new file, then renaming it

    Why
    OPFS and Storage offer no uniform atomic rename.
    Cost
    Every commit writes one 4 KiB slot.

    Several writers

    OS whole-file locks or Web Locks, held per commit group

    Instead of one process that owns the database

    Why
    Independent processes and workers share one database, and commits queued on one handle share one write and its flushes.
    Cost
    Commits serialize, so revisions keep a single order.

    Transactions

    Recheck every key the transaction read

    Instead of one revision check for the whole database

    Why
    Writers on disjoint keys commit without retrying, and write skew and phantoms are still ruled out.
    Cost
    A transaction remembers each key it read; after a scan it falls back to the whole-database check.

    Space reuse

    Explicit compact() and collect()

    Instead of in-place updates or a background vacuum

    Why
    No hidden I/O: the application decides when to pay.
    Cost
    Compaction needs disk space for one live copy.

    Stale locks

    Kernel locks, released when a process dies

    Instead of lock files that expire after a timeout

    Why
    A paused process never loses ownership to a second writer.
    Cost
    Servers need the native lock addon.
    Benchmarks

    What an operation costs

    defuss-dbsonl 0.5.0 with native asynchronous I/O on an Apple M5 Max under macOS, Node.js 24.22.1 with V8 and Bun 1.4.3 with JavaScriptCore (JSC). Each figure is the median of three runs; every run takes the median of seven rounds over 1000 records after two warmups and checks every returned value.

    Point read

    get(), head checked on every call

    Node.js (V8)
    21.3 µs
    Bun (JSC)
    44.4 µs

    Snapshot read

    get() on an open snapshot

    Node.js (V8)
    11.7 µs
    Bun (JSC)
    18.5 µs

    Scan

    per row, in key order

    Node.js (V8)
    4.4 µs
    Bun (JSC)
    6.7 µs

    Atomic batch

    1000 writes in one flushed commit

    Node.js (V8)
    12.7 ms
    Bun (JSC)
    6.7 ms

    Compaction

    1000 live records copied

    Node.js (V8)
    19.7 ms
    Bun (JSC)
    10.7 ms

    Reopen

    indexed open of an existing database

    Node.js (V8)
    0.62 ms
    Bun (JSC)
    0.44 ms
    All modes, in milliseconds per 1000 operations
    RuntimeStoragePoint readSnapshot readScanBatchCompactionReopen
    Node.js (V8)native async21.2811.724.3512.7219.670.62
    Node.js (V8)native sync10.723.612.5511.7117.960.56
    Node.js (V8)memory9.774.083.854.055.580.04
    Bun (JSC)native async44.3818.516.706.6710.710.44
    Bun (JSC)native sync19.143.182.863.896.270.29
    Bun (JSC)memory21.064.533.144.045.570.04

    Synchronous I/O is opt-in and blocks its thread, including during flushes; use it in a worker or CLI. Batch, compaction and reopen are per workload, not per operation.

    Against v0.3.0 in the same three runs, no figure was slower in all of them, and atomic batches took 1.6 times less time on Node.js and 2.4 times less on Bun. Measured under load: other work kept the machine busy through all three runs (load average 8.3 to 11.0), so every figure includes contention for CPU and I/O. The file cache was warm; cold-disk latency was not measured. Raw samples are in validation/bench-macos-arm64.json, and make bench repeats the run on your machine.

    The comparison with Redis is below the introduction.

    Tests

    Tested on real files, locks and workers

    The tests run against real memory and native storage, real kernel locks and real browser workers, with no mocks: the defects that matter here, such as partial reads, torn slots and lock lifetimes, live inside the subsystems a mock would replace.

    46
    tests pass on macOS
    94%
    statements covered, 85% required
    91%
    branches covered, 75% required
    97%
    functions covered, 85% required
    98%
    lines covered, 85% required

    Tree against a Map

    Every tree result is compared with an independent Map, alongside atomic batches, snapshots, conflicts, phantoms and quota failures.

    mvcc.test.ts

    Commit groups and per-key conflicts

    Disjoint writers commit, changed read keys, phantoms and write skew conflict; queued commits share one publication and keep their own receipts.

    commit-groups.test.ts

    API contracts

    Transaction scans over staged writes, the read-set fallback, argument errors, close, lock protocol, newer manifests and denied reads.

    contracts.test.ts

    Batch scans

    Batches match the row scan for every range and limit on memory and native storage, keep their budgets, and stay owned, cancellable and safe to close.

    batches.test.ts

    Crashes and corruption

    Torn head slots, truncated and damaged data, publication boundaries and an interrupted create, written as real byte images.

    v2-recovery.test.ts

    Real kernel locks

    Independent native backends contend for real OS locks and hit real permission failures.

    native-v2.test.ts

    Native I/O edges

    Descriptor limits, stale roots after another client compacts, partial reads and writes, large binary values.

    native-performance.test.ts

    Index-page writer

    2000 generated leaf and branch pages per run must encode to the same bytes as the general DSON encoder and decode back.

    page.test.ts

    Web Locks

    Shared pins, try-locks, cancelled waits and pinned generations through the browser backend.

    web-locks.test.ts

    The packed package

    Node.js, Bun, ESM, CommonJS and types from the tarball; v0.2.0 and current writers killed mid-batch; Chromium workers; every demo on this page.

    make e2e
    Contract

    Guarantees

    The platforms these guarantees were tested on, and the ones they were not, are listed in the README.

    Get started

    Setup

    Install it from npm with Bun or npm. The package needs Node.js 20.19 or newer, Bun, or a dedicated worker in a browser with OPFS and Web Locks. The native lock addon ships prebuilt binaries for Linux x64, macOS and Windows x64, so nothing compiles on install.

    bun add defuss-dbsonl

    npm install defuss-dbsonl

    The /server entry adds kernel file locks and directory sync. Every process that opens the same directory joins the same lock protocol.

    import { openDbsonl } from 'defuss-dbsonl';
    import { createFileBackend } from 'defuss-dbsonl/server';
    const db = await openDbsonl(await createFileBackend('./data'), 'orders', { create: true });
    await db.batch([
      { operation: 'put', key: 'order:1', value: { placed: new Date(), total: 4200n } },
      { operation: 'delete', key: 'cart:1' },
    ]);
    await db.close();

    Read the protocol behind these guarantees

    ARCH.md documents the lock order, the crash cases and reclamation. The test suite runs them against real files, kernel locks, killed processes and browser workers.

    MIT license.