Data product
libbitcask · Embedded KV and search engine
Building retrieval for an LLM application usually means gluing together a KV store, a full-text engine, and a vector database, then fusing rankings in your application layer. libbitcask puts all of that into one embedded library: write a document once and both the full-text and vector indexes are built; query once and get the fused result.
The core is Bitcask's append-only design: writes only append; reads locate via the in-memory keydir and fetch with a single pread. On top of that you can enable index mode when needed — leave it off and you have a pure, fast KV store.
One library: KV, full text, vectors, fusion
CaskOptions opts;
opts.read_write = true;
opts.enable_search = true;
opts.search_config = SearchLayerConfig{
.analyzer_config = AnalyzerConfig{.type = AnalyzerType::Ngram,
.min_n = 2, .max_n = 3},
};
opts.vector_dim = 128;
opts.vector_metric = meta::VectorMetric::kCosineNormalized;
IndexRegistry registry;
auto c = Cask::open("/tmp/db", opts, ®istry);
DocInput doc{
.text = std::as_bytes(std::span{"Beijing weather is clear today"}),
.vector = query_vec,
};
(*c)->put_doc(key, doc);
(*c)->flush_index();
auto bm25 = (*c)->search_text("Beijing weather", 10);
auto knn = (*c)->search_vector(query_vec, /*k=*/10);
auto hyb = (*c)->search_hybrid("Beijing weather", query_vec, /*k=*/10);Features
What it can do
Full-text search
BM25 scoring with phrase, proximity, boolean (AND / OR / NOT), per-field weighting, fuzzy (edit distance), wildcard, highlighting, and pagination; filter results by document metadata. Tokenizers include whitespace, n-gram, jieba Chinese segmentation, and stemming; queries expand through synonym dictionaries.
Vector search, three engines
hnsw in-memory graph (default, int8 quantization cuts vector memory by roughly 80%); ivfrq, an IVF-RaBitQ disk-based engine for 10M–100M vectors; diskann (Vamana on-disk graph) is experimental. The engine is chosen at database creation; offline tools migrate between engines.
Hybrid retrieval
BM25 and vector each produce candidates, fused with Reciprocal Rank Fusion in a single call. Batch retrieval runs multiple queries concurrently on a shared search pool.
Reliable writes
Crash-safe atomic batches: after a power cut, a batch either fully applies or not at all. Batches may include deletes, and you can write documents with full-text and vector indexes directly. Write offsets anchor to the last complete record, so a torn tail is never mistaken for valid data.
Ordered indexes and tiered compaction
The OKI ordered key index supports [lo, hi) lexicographic scans at O(range) rather than O(table): measured 8.0 ms → 0.53 ms at 1/256 selectivity. From 7.0, compaction runs in the background over latest contiguous suffixes, and read views publish immutably.
Keys at scale
OKI supports disk residency: only hot keys stay cached in memory, and a cold key lookup costs at most two preads. Resident memory for 100M keys drops from ~11 GB to ~1.1 GB with no regression on hot paths.
Implementation
How it's built
C API
libbitcask.so exposes opaque extern "C" handles, binary-safe slices, and error details; a PIMPL holds the C++ Cask and no C++ exception crosses the ABI boundary. Signatures stay stable across versions, keeping FFI bindings easy.
Cask: KV and search facade
data files remain the only durable truth, and in a healthy directory OKI is the authority for lookups and commits. IndexAuthority keeps the ord / file-id / fstats ledger; DocMap maintains ord↔key and document metadata; data-file read handles cache under approximate LRU; metadata files carry version, CRC, and epoch gating.
Search plugins
The full-text plugin: inverted index + WAND pruning + k-way intersection + query syntax tree, with n-gram, jieba, whitespace, and stemming tokenizers. The vector plugin: HNSW / IVF-RaBitQ / DiskANN behind one contract, chosen by creation metadata, with int8 quantization and mmap. HybridSearcher fuses both rankings with RRF.
Async indexing
Writes persist as soon as they enter the append log; indexes build asynchronously in an IndexPool MapReduce pipeline. A read barrier drains in-flight writes before queries, so results are near-real-time.
Merge and recovery
merge runs under its own merge.lock concurrently with online reads and writes, CAS-relocating live records and broadcasting to search plugins. OKI compaction runs tiered in the background with immutable read views; after a crash, replay is bounded from the checkpoint and torn tail records are truncated rather than misread.
Specifications
Technical specifications
| Language | C++23 with no Boost / abseil; third parties vendored as submodules, build needs no network |
|---|---|
| Version | 7.0.3; the C API product version releases in lockstep with SOVERSION 7 |
| Form | C++ library (bitcask::Cask) and extern "C" ABI (libbitcask.so) for FFI bindings from Python / Rust / Go / Node and more |
| Concurrency | One handle is safe across threads: true concurrent reads, internally serialized writes; merge concurrent with reads and writes |
| Platforms | Linux (GCC 13+ / Clang 17+), Windows (native MSVC x64), libc++ (FreeBSD / macOS) |
| Text | UTF-8 internally; normalization and transcoding (including GB18030) via ICU |
| License | Apache License 2.0 |
FAQ
You might be wondering
Can I use it as a plain KV store?
Yes. With index mode off it is a pure Bitcask KV: writes append only, and a healthy directory answers via OKI plus one pread. When you later need search, Cask::upgrade migrates an existing directory offline into index mode.
How do I call it from Python, Rust, Go, or similar?
The build produces libbitcask.so, and bitcask_c.h provides a complete C API (opaque handles, explicit *_free, error codes + details). Any language that can call C functions can bind through FFI.
How do I choose a vector engine?
Up to a few million vectors use the default hnsw (in-memory graph, optional int8 quantization); for 10M–100M use ivfrq (disk-based); diskann is experimental. The engine is chosen at creation; offline tools migrate later.
How does multithreading work?
One Cask handle is shareable across threads: reads and searches run truly concurrently, writes serialize internally (an append-only log is serial anyway), and merge runs concurrently with both. Iterators are per-thread; use parallel_scan to traverse in parallel.
Will a crash or power cut lose data?
put_batch_atomic provides crash-safe atomic batches: a batch fully applies or not at all. Recovery anchors the write position to the last complete record; a torn tail is never treated as valid data.
What is the license?
Apache License 2.0, fine for commercial projects.
Open source
Apache 2.0 open source
The engine, tests, benchmarks, and design docs are all in the repository — read them, change them, file issues.
The same engine
Bitcask · Search, transactions, and graphs for Erlang/OTP
libbitcask brought into Erlang/OTP via a NIF: KV, full text, vectors, hybrid search, and structured queries — plus transactions, idempotency keys, graph processing, and local embeddings.