bun:arrow

In-memory columnar tables, computes, and Arrow IPC reader/writer — wire-compatible with apache-arrow, pyarrow, polars, duckdb.

import arrow from "bun:arrow";

Apache Arrow's columnar model, in-process, with no npm dep on apache-arrow. RecordBatches are typed-array views with optional validity bitmaps; tables are sequences of batches sharing a schema. The Arrow IPC streaming + file formats round-trip both directions against the canonical implementations.

Building tables

recordBatch({ ... })

Takes a map of column name → values and infers per-column type. Values can be:

• A typed array — Int32Array → int32, BigInt64Array → int64, Float32Array → float32, Float64Array → float64, Uint8Array → bool.
• A string[] → utf8.
• A T[][] (array of arrays) → list<T> with the child type inferred from the flattened first non-empty row.

const batch = arrow.recordBatch({
  age:    new Int32Array([25, 30, 35]),
  score:  new Float64Array([0.95, 0.82, 0.71]),
  name:   ["alice", "bob", "carol"],
  tags:   [["a", "b"], [], ["c", "d", "e"]],   // list<utf8>
});

batch.numRows;                    // 3
batch.column("age").get(0);       // 25
batch.column("tags").get(2);      // ["c", "d", "e"]

table(batches)

Concatenates batches sharing a schema. Table has a .column(name) that returns a ConcatColumn — a virtual view across batches. Pass it to any compute function for a table-wide aggregate.

fromRows(rows, opts?) / toRows(source)

Bridge between row-shaped JS data and the columnar form. fromRows is the typical entry point from bun:csv output:

import csv from "bun:csv";
import arrow from "bun:arrow";

const rows: any[] = [];
for await (const r of csv.parseCsv(Bun.file("data.csv"), { header: true })) rows.push(r);
const tbl = arrow.fromRows(rows);

Compute primitives

All take a Column or ConcatColumn. Numeric reductions return a single scalar; predicate-style return a column or new batch.

filter(batch, predicate)

Returns a new RecordBatch keeping rows where predicate(row) is truthy. Predicate sees a row-shaped object keyed by column name.

const adults = arrow.filter(batch, row => row.age >= 30);

groupBy(batch, keys, aggs)

Hash group-by. keys is a string or array of column names; aggs is a map of output-name → { column, op }. Supported ops: sum, mean, min, max, count, variance, stddev, distinct.

const result = arrow.groupBy(batch, "city", {
  rows:    { column: "name",  op: "count" },
  avgAge:  { column: "age",   op: "mean"  },
  topScore:{ column: "score", op: "max"   },
});

sort(batch, by, opts?)

Stable sort by one or more keys. by is string | string[] | { name, descending?: boolean }[]. Returns a new batch with rows reordered.

Arrow IPC

Streaming format

const bytes = arrow.toIPC(table);          // Uint8Array
const restored = arrow.fromIPC(bytes);     // Table

Continuation-prefixed Schema + RecordBatch messages, FlatBuffers metadata (hand-rolled builder/reader; no npm dep), 8-byte-aligned body buffers, EOS marker. DictionaryBatch decode is implemented for round-tripping apache-arrow's default Dictionary<Utf8> for string columns.

File format

Pass "file" as the second arg to write the ARROW1-bracketed file format:

const fileBytes = arrow.toIPC(table, "file");   // ARROW1 + messages + EOS + Footer + len + ARROW1
const restored = arrow.fromIPC(fileBytes);      // auto-detects via head/tail magic

The Footer flatbuffer carries a redundant copy of the schema plus a list of Block { offset, metaDataLength, bodyLength } entries pointing at each RecordBatch / DictionaryBatch — random-access on read.

fromIPC auto-detects: if the bytes start with ARROW1\0\0 and end with ARROW1 the file path is taken (Footer's schema and Block list drive the decode); otherwise it falls through to the streaming reader. Same callsite, both formats.

Type coverage

Date / Time / Timestamp from upstream Arrow streams are coerced to int32 / int64 on read (unit and timezone metadata dropped). Round-tripping re-emits them as plain ints.

Wire compat

bench/parabun-arrow-ipc-interop/ round-trips both directions against apache-arrow@21.1.0:

• Parabun encodes streaming + file → apache-arrow decodes.
• apache-arrow encodes streaming + file (including default Dictionary<Utf8> strings + Date64) → Parabun decodes.

Mixed type table (Int8, Uint16, Uint32, Int32, Float64, Date64, Dictionary<Utf8>, List<Float64>) round-trips bit-for-bit through both formats.

Output you can read elsewhere

The bytes Parabun produces are the same wire format pyarrow, arrow-rs, nanoarrow, polars, and duckdb consume on the streaming + file paths. Save with .arrow:

await Bun.write("data.arrow", arrow.toIPC(table, "file"));

Then in Python:

import pyarrow.feather as feather
df = feather.read_table("data.arrow")

Parquet

fromParquet(bytes) reads and toParquet(source, opts?) writes Apache Parquet files. Hand-rolled Thrift compact-protocol codec, Snappy compressor + decompressor, dictionary + RLE + bit-pack hybrid decoders, RLE writer for definition levels — no npm dep.

// Read
const bytes = new Uint8Array(await Bun.file("rows.parquet").arrayBuffer());
const tbl = arrow.fromParquet(bytes);

// Write
const out = arrow.toParquet(tbl, { compression: "snappy" });
await Bun.write("rows.parquet", out);

toParquet options:

Verified end-to-end against pyarrow output:

• Read: round-trips 6-column fixtures (int32 / int64 / float32 / float64 / utf8 / bool) under uncompressed + snappy, plus a 10,000-row fixture with nulls at 1/5, 1/7, 1/13 ratios across 4 row groups under all three compression codecs.
• Write: pyarrow reads Parabun's output bit-for-bit (uncompressed, snappy, gzip); 100-row fixture with scattered nulls (1/7 id, 1/11 score, 1/13 name) → null counts match pyarrow's 15 / 10 / 8 exactly.

What's not here yet

• Dictionary encoding on write — strings emit as PLAIN today. pyarrow's default RLE_DICTIONARY for low-cardinality columns is an opt-in encoding pass that lands when there's a workload that needs the density.
• Multi-row-group writes — large tables write as one giant row group today. Splitting at ~1M rows would match pyarrow's defaults.
• Struct / Map / FixedSizeList / Union / Decimal128 / FixedSizeBinary — nested + decimal types. The List<T> shape proves out the recursive FieldNode + buffer walk; the others reuse it.
• Dictionary delta batches (isDelta=true) — apache-arrow's default is non-delta, so this is a long-tail follow-up.
• uint64 — no lossless 64-bit unsigned representation in JS Number / BigInt without losing range.
• Lossless narrow-type round-trip — Parabun reads int8 by widening to int32, then writes int32. Lossless on values, lossy on the type tag. A typed wrapper that remembers wire types can land if there's a use case.