Data product

Bitcask · Search, transactions, and graphs for Erlang/OTP

Basho Bitcask's log-structured design, with our libbitcask as the storage and search engine, wired into Erlang/OTP through a C++23 NIF. To an Erlang program it is still the familiar bitcask:open / put / get.

On top of that, configure an embedder at open and writing text generates vectors automatically; query with a sentence and full-text and vector results fuse into one ranking. Transactions, graph traversal, and graph analytics all happen inside OTP — no separate service.

One OTP app: store, search, compute, transact

Erlang: isolated transactions (pessimistic 2PL + deadlock detection + automatic restart)
1> R = bitcask:open("/tmp/db", [read_write]).
2> bitcask:put(R, <<"a">>, <<"100">>), bitcask:put(R, <<"b">>, <<"0">>).
3> bitcask_txn:transaction(R, fun(Tx) ->
       {ok, A} = bitcask_txn:read(Tx, <<"a">>),
       {ok, B} = bitcask_txn:read(Tx, <<"b">>),
       ok = bitcask_txn:write(Tx, <<"a">>, integer_to_binary(binary_to_integer(A) - 30)),
       ok = bitcask_txn:write(Tx, <<"b">>, integer_to_binary(binary_to_integer(B) + 30)),
       moved
   end).
{atomic,moved}
4> [bitcask:get(R, K) || K <- [<<"a">>, <<"b">>]].
[{ok,<<"70">>},{ok,<<"30">>}]

Features

What it can do

  • Search: full text, vectors, fusion

    Open with {analyzer, whitespace | ngram | jieba} and BM25 is on; put indexes automatically. Vector engines: hnsw / ivfrq / diskann; RRF hybrid search fuses both rankings. Structured queries combine key conditions, full-text conditions, and metadata filters in one call — the planner picks indexes automatically, with pagination and secondary indexes.

  • Automated embedding

    Pass an embedder at open and the collection dimension aligns automatically; writing text and running queries then generate vectors on the fly. Works with any OpenAI-compatible endpoint (llama.cpp server, vLLM) or the optional local llama.cpp backend with CUDA / Vulkan acceleration, weights shared across libraries.

  • Atomic batches and isolated transactions

    put_batch_atomic / txn_commit are all-or-nothing across crashes; batches and transactions can write documents with automatic embeddings. bitcask_txn adds isolation: pessimistic two-phase locking, deadlock detection, automatic restarts — prefix locks make range scans inside a transaction phantom-free.

  • Idempotency keys

    If the caller is killed mid-commit, retry with the same idem_key: a key commits at most once, and a retry returns the first result.

  • Graph processing

    graphdb stores vertices and edges in the KV layer, walking adjacency with ordered prefix scans: BFS, k-hop, bidirectional shortest paths, filtered traversals. graphdb_analytics materializes an in-memory CSR for PageRank, connected components, and single-source shortest paths.

  • Ordered range queries

    range / range_fold take [Lo, Hi) in key order at O(range) cost; prefix scans are one line. Underneath, libbitcask 7.0 compacts in background tiers with immutable read views.

Implementation

How it's built

  1. bitcask: the Erlang facade

    bitcask.erl is a thin facade; every operation goes through bitcask_cpp_nifs into priv/bitcask_cpp.so — the libbitcask 7.0 compiled into the NIF. Slow paths run as dirty NIFs so the BEAM schedulers never block; open returns {CaskRef, EmbedderCtx} as the overall handle.

  2. bitcask_txn: transaction coordination

    Pure OTP with zero engine changes: write buffers live in the calling process and commit expands into a single txn_commit atomic batch in ascending key order. bitcask_txn_locker manages pessimistic two-phase locks in shards, detects deadlocks with a wait-for graph, picks the youngest transaction as victim and restarts it; prefix locks prevent phantoms and commit tokens keep locks held even if the caller is killed mid-flight.

  3. Embedding (embedder)

    bitcask_embedder is a behaviour with an OpenAI-compatible implementation built in; the optional local backend is a second, separate NIF (priv/bitcask_llama.so). bitcask_embedder_server holds the model weights shared across libraries, slots configure concurrent inference, and a proxy balances by shortest queue.

  4. graphdb: graph processing

    Vertices and edges live in the KV layer under fixed-width keys; edges write both directions in an atomic batch. Adjacency walks use ordered prefix ranges: BFS, k-hop, bidirectional shortest paths. graphdb_analytics materializes an in-memory CSR for PageRank, connected components, and single-source shortest paths.

  5. The OTP supervision tree

    bitcask_sup supervises the global merge scheduler (time windows, deduplication, rate limiting) and the transaction-lock manager shards with one_for_one. Embedder service processes appear only when an embedder is configured in application env. A standard OTP application that starts and stops with your system.

Specifications

Technical specifications

RuntimeErlang/OTP 27+, built with rebar3
Version7.0.2, bundling libbitcask 7.0.2
ImplementationA single C++23 NIF (libbitcask) + a thin Erlang facade
EmbeddingOpenAI-compatible HTTP endpoints; optional local llama.cpp (not built by default; CUDA / Vulkan when enabled)
Testing400+ C++ GoogleTest cases; the full Erlang eunit suite passes
LicenseApache License 2.0

FAQ

You might be wondering

How does it relate to Basho's original bitcask?

It keeps the original log-structured design and the bitcask:open / put / get usage, swapping the storage and search engine for C++23 libbitcask and adding full text, vectors, transactions, and graphs.

Which OTP version?

Erlang/OTP 27 minimum, built with rebar3; the C++ side needs a C++23 compiler and ICU development headers.

Does embedding require a GPU?

No. Any OpenAI-compatible endpoint works (llama.cpp server, vLLM); you can also build the local llama.cpp backend and accelerate with CUDA or Vulkan when a discrete GPU is present, otherwise CPU.

What do transactions guarantee?

put_batch_atomic / txn_commit guarantee atomicity across crashes; bitcask_txn adds isolation with two-phase locking, deadlock detection, and automatic restarts. Note the transaction function may run more than once and must be side-effect free; the pass-through bitcask:put / get bypass locks — don't mix them with transactions over the same keys.

What is the license?

Apache License 2.0, fine for commercial projects.

Open source

Apache 2.0 open source

The engine, tests, benchmarks, and design docs are all in the repository — read them, change them, file issues.

The same engine

libbitcask · Embedded KV and search engine

A Bitcask append-only KV store in C++23, layered with BM25 full text, vector nearest-neighbor, and RRF hybrid search. A C++ library to use directly, plus a C ABI for bindings in other languages.