README.md

PyBCSV — Python Bindings for BCSV Library

High-performance Python bindings for the BCSV (Binary CSV) library — fast, compact time-series storage with pandas integration.

Features

• High Performance: Binary format with optional LZ4 compression and delta encoding
• Pandas Integration: Columnar DataFrame read/write via numpy zero-copy
• Type Safety: Preserves column types and data integrity (10 numeric types + strings)
• Cross-platform: Linux (x86_64, ARM64), macOS (x86_64, ARM64), Windows (AMD64)
• Context Managers: All readers/writers support with statements
• Streaming I/O: Row-by-row read/write, never loads entire file into memory
• Direct Access: Random-access reads by row index via ReaderDirectAccess
• Sampler: Bytecode VM for server-side row filtering and column projection
• CSV Interop: Convert between CSV and BCSV via from_csv() / to_csv()

Installation

pip install pybcsv

# With pandas support
pip install pybcsv[pandas]

Developing from a checkout

Install into a virtualenv, never the user site:

python -m venv .venv
.venv/bin/python -m pip install -e python

Two things to know about the editable install:

• After a release bump, reinstall it. The compiled extension auto-rebuilds on import, but its version string does not, so it keeps reporting the previous release while carrying the current code — which then labels benchmark results with the wrong version:

  .venv/bin/python -m pip install -e python --no-deps --force-reinstall

• A stray editable install in ~/.local shadows the venv for every interpreter outside it and puts broken bcsv2parquet / parquet2bcsv scripts on PATH. The symptom is an import error naming DEFAULT_COMPRESSION_LEVEL.

Verify what is actually installed, and see VERSIONING.md for the details of both failure modes:

scripts/check_versions.py --skip-manifests --python .venv/bin/python

Quick Start

Write and Read

import pybcsv

# Define schema
layout = pybcsv.Layout()
layout.add_column("id", pybcsv.INT32)
layout.add_column("name", pybcsv.STRING)
layout.add_column("value", pybcsv.DOUBLE)

# Write rows (context manager auto-closes)
with pybcsv.Writer(layout) as writer:
    writer.open("data.bcsv")
    writer.write_row([1, "Alice", 123.45])
    writer.write_row([2, "Bob", 678.90])

# Read all rows
with pybcsv.Reader() as reader:
    reader.open("data.bcsv")
    for row in reader:          # iterator protocol
        print(row)
    # or: all_rows = reader.read_all()

Pandas Integration

import pybcsv
import pandas as pd

df = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'value': [123.45, 678.90, 111.22]
})

# Write DataFrame (columnar path, numpy zero-copy for numerics)
pybcsv.write_dataframe(df, "data.bcsv")

# Read back as DataFrame
df_read = pybcsv.read_dataframe("data.bcsv")

CSV Conversion

import pybcsv

pybcsv.from_csv("input.csv", "output.bcsv")   # CSV → BCSV
pybcsv.to_csv("output.bcsv", "output.csv")    # BCSV → CSV

Polars Integration

Zero-copy Polars DataFrame I/O via the Arrow C Data Interface:

import pybcsv

# Read BCSV → Polars DataFrame (zero-copy via Arrow)
df = pybcsv.read_polars("data.bcsv")

# Write Polars DataFrame → BCSV
pybcsv.write_polars(df, "output.bcsv", row_codec="delta")

Install with the optional Polars dependency:

pip install pybcsv[polars]

Random Access

import pybcsv

with pybcsv.ReaderDirectAccess() as da:
    da.open("data.bcsv")
    print(f"Total rows: {len(da)}")
    row = da[42]         # read row 42 directly (O(1) seek)
    print(da.read(100))  # alternative syntax

Bundled Native CLI Tools

The wheel ships the core BCSV command-line tools as native, version-matched binaries — installed into the environment's scripts directory, so they are on PATH in an activated venv with no compiler or CMake required:

Tool	Purpose
csv2bcsv	CSV → BCSV with validated type inference (see the main CLI docs)
bcsv2csv	BCSV → CSV with row/column selection
bcsvHeader	Show a file's schema
bcsvHead / bcsvTail	Show the first/last N rows
bcsvCast	Change/narrow column types
bcsvSampler	Filter/project rows with a condition expression
bcsvValidate	Integrity-check a file
bcsvRepair	Recover data from truncated/corrupted files
bcsvCompare	Compare two files (with float tolerance)
bcsvGenerator	Generate synthetic test datasets

csv2bcsv data.csv data.bcsv        # from an activated environment

import pybcsv

pybcsv.tools.run("csv2bcsv", "data.csv", "data.bcsv", "--overwrite")
print(pybcsv.tools.run("bcsvHeader", "data.bcsv").stdout)
pybcsv.tools.path("csv2bcsv")      # absolute path, e.g. for custom pipelines

Source builds can opt out with -DPYBCSV_BUILD_TOOLS=OFF; the full tool suite (and docs) live in the main repository under src/tools/.

Parquet Conversion Tools (CLI)

Installing pybcsv provides two streaming command-line converters (require the arrow extra: pip install pybcsv[arrow]). They stream in bounded batches, so they handle files larger than memory.

# Parquet → BCSV
parquet2bcsv input.parquet -o output.bcsv
#   --row-codec {delta,zoh,flat}          row codec (default: delta)
#   --file-codec {packet_lz4_batch,...}   file codec (default: packet_lz4_batch)
#   --chunk-size N                        rows per streamed batch (default: 512000)
#   --null-policy {reject,nan,zero}       Parquet nulls (default: reject)
#   --no-metadata2json                    skip <output>.meta.json
#   --no-source-hash                      omit the source Parquet's SHA-256
#   --no-bcsv-hash                        omit the output BCSV's SHA-256 (weakens the
#                                         companion's binding to a heuristic)
#   -f/--force                            overwrite an existing output

# BCSV → Parquet
bcsv2parquet input.bcsv -o output.parquet
#   --columns "a,b,c"       select/reorder columns (order is honored)
#   --slice 10:100          Python-style row slice
#   --unflatten (default)   reconstruct nested structs from dotted/bracketed names
#   --no-unflatten          keep flat columns (names like 'a.b', 'vals[0]')
#   --no-json2metadata      ignore <input>.meta.json instead of restoring it
#   --parquet-compression {none,snappy,gzip,zstd,lz4}

Schema mapping. Parquet structs and fixed-size lists are flattened to BCSV columns using dotted (location.lat) and bracketed (vals[0]) names; bcsv2parquet --unflatten reverses this. Notes and limitations:

• Nulls (--null-policy): BCSV has no null representation, so you pick what a null becomes:

  policy	float16/32/64	integers, bool, string
  reject (default)	abort, naming column and row	abort
  nan	NaN	abort
  zero	0.0	0 / False / ""

  nan is the honest choice for float columns: NaN is a real IEEE-754 value BCSV round-trips bit-exactly, and it stays visibly not a measurement. It aborts on other types because there is no such value for an integer or a bool. zero fills every type with the BCSV default — what an unset cell already holds — and is the only policy that can carry integer, bool or string nulls.

  Both filling policies lose information, differently. nan only loses it when the column also holds genuine NaNs, in which case the two collapse and bcsv2parquet cannot separate them. zero is unconditionally lossy: a filled zero is indistinguishable from a measured zero, so nothing downstream can tell the sample was missing. Choose it only when a consumer genuinely treats 0/False/"" as absent.

• File-level metadata (--no-metadata2json / --no-json2metadata): BCSV's header has no key/value section, so parquet2bcsv writes the source's Parquet key/value metadata to <output>.meta.json (metadata2json=True by default) and bcsv2parquet restores it into the output footer (json2metadata=True by default). pyarrow's internal ARROW:schema key is excluded. The file is optional in both directions: suppress its generation with --no-metadata2json, and bcsv2parquet works fine without one — the output simply carries no key/value metadata.

  It records the BCSV file's SHA-256 and refuses to be applied to a file whose digest does not match, so a document left over from an earlier conversion to the same output name cannot stamp unrelated data with someone else's provenance. --no-bcsv-hash skips that read pass, but then the binding rests on byte size and row count alone — a heuristic two recordings of the same shape can both satisfy, not an identity check.

  read_metadata_json and BcsvMetadata.ReadCompanion return only the document's key_value_metadata object — the Parquet footer pairs, verbatim. The document level around it (metadata_json_version, source_path, source_sha256, bcsv_sha256, bcsv_bytes, bcsv_rows) describes the conversion itself and is not returned; parse the JSON if you need it. Mind that the two levels share one namespace: source_sha256 at the document level is the digest of the Parquet input, while a pipeline that stamped its own source_sha256 into the Parquet footer leaves a different digest under the same name inside key_value_metadata. Same name, two levels, two different links of the chain — source_path, bcsv_sha256, bcsv_bytes, bcsv_rows and metadata_json_version behave the same way.

  An in-format metadata channel is planned for 1.6.0; this one will keep working when it lands.

• Random access: the packet* file codecs write a footer index, so ReaderDirectAccess / BcsvReader.Read(index) work on the output. The stream and stream_lz4 codecs write no packets and no footer — the result is sequential-only and parquet2bcsv warns when you ask for one.

• Type widening: float16/bfloat16 widen to float32; large_string maps to string.

• Unsupported types (variable-length lists, maps, timestamps, decimals, dictionaries) are rejected with a clear error.

• Column names ending in _ are rejected (the unflatten escape protocol reserves trailing underscores).

• Lists of structs round-trip: FixedSizeList<struct<x, y>> flattens to field[0].x, field[0].y, field[1].x, … and --unflatten rebuilds the nested type, including structs and fixed-size lists nested inside the element.

• Ambiguous names: --unflatten fails loudly rather than guessing when the flat names cannot describe one nested schema — a literal dotted column (a.b) colliding with a struct path, a name used as both a leaf and a parent (a plus a.b), a name used as both a list and a struct (a[0] plus a.b), list indices with a gap (x[0] and x[2] but no x[1]), or list elements that disagree on type. Use --no-unflatten to keep the flat columns.

Available Types

Constant	Description
pybcsv.BOOL	Boolean
pybcsv.INT8 / pybcsv.UINT8	8-bit integers
pybcsv.INT16 / pybcsv.UINT16	16-bit integers
pybcsv.INT32 / pybcsv.UINT32	32-bit integers
pybcsv.INT64 / pybcsv.UINT64	64-bit integers
pybcsv.FLOAT	32-bit float
pybcsv.DOUBLE	64-bit float
pybcsv.STRING	Variable-length string

API Reference

Layout

layout = pybcsv.Layout()                              # empty layout
layout = pybcsv.Layout([ColumnDefinition("x", INT32)]) # from list

layout.add_column(name: str, type: ColumnType)
layout.add_column(col: ColumnDefinition)
layout.column_count() -> int
layout.column_name(index: int) -> str
layout.column_type(index: int) -> ColumnType
layout.has_column(name: str) -> bool
layout.column_index(name: str) -> int
layout.get_column_names() -> list[str]
layout.get_column_types() -> list[ColumnType]
layout.get_column(index: int) -> ColumnDefinition
len(layout)           # column count
layout[i]             # ColumnDefinition at index i

Writer

writer = pybcsv.Writer(layout: Layout, row_codec: str = "delta")
writer.open(filename: str, overwrite: bool = False,
            compression_level: int = 6, block_size_kb: int = 8192,
            flags: FileFlags = FileFlags.BATCH_COMPRESS)  # raises RuntimeError on failure
writer.write_row(values: list)
writer.write_rows(rows: list[list])     # batch write
writer.flush()
writer.close()
writer.is_open() -> bool
writer.row_count() -> int
writer.row_codec() -> str
writer.compression_level() -> int
writer.layout() -> Layout

# Context manager
with pybcsv.Writer(layout) as w:
    w.open("out.bcsv")
    w.write_row([...])

Row codec options: "flat", "zoh" (zero-order hold), "delta" (default).

Reader

reader = pybcsv.Reader()
reader.open(filename: str)              # raises RuntimeError on failure
reader.read_next() -> bool              # advance to next row
reader.read_row() -> list | None        # read+advance, None at EOF
reader.read_all() -> list[list]         # read remaining rows
reader.close()
reader.is_open() -> bool
reader.layout() -> Layout
reader.row_pos() -> int                 # current row index
reader.row_value(column: int) -> Any    # typed value from current row
reader.row_dict() -> dict               # current row as {name: value}
reader.file_flags() -> FileFlags
reader.compression_level() -> int
reader.version_string() -> str
reader.creation_time() -> str
reader.count_rows() -> int              # total row count

# Iterator protocol
for row in reader:
    print(row)

# Context manager
with pybcsv.Reader() as r:
    r.open("data.bcsv")
    for row in r:
        print(row)

ReaderDirectAccess

Random-access reader — reads any row by index without scanning.

da = pybcsv.ReaderDirectAccess()
da.open(filename: str, rebuild_footer: bool = False)
da.read(index: int) -> list             # read row at index
da.row_count() -> int
da.layout() -> Layout
da.close()
da.is_open() -> bool
da.file_flags() -> FileFlags
da.compression_level() -> int
da.version_string() -> str
da.creation_time() -> str

len(da)               # row count
da[i]                 # read row at index i

CsvWriter / CsvReader

Native CSV I/O with the same Layout-based schema.

# Write CSV
csv_w = pybcsv.CsvWriter(layout, delimiter=',', decimal_sep='.')
csv_w.open(filename, overwrite=False, include_header=True)
csv_w.write_row(values)
csv_w.write_rows(rows)
csv_w.close()

# Read CSV
csv_r = pybcsv.CsvReader(layout, delimiter=',', decimal_sep='.')
csv_r.open(filename, has_header=True)
for row in csv_r:       # iterator support
    print(row)
csv_r.close()

Sampler

Bytecode VM for filtering and projecting rows from an open Reader.

reader = pybcsv.Reader()
reader.open("data.bcsv")

sampler = pybcsv.Sampler(reader)
sampler.set_conditional("col_a > 10")    # filter expression
sampler.set_selection("col_a, col_b")    # column projection

result = sampler.output_layout()         # SamplerCompileResult (bool-testable)
if result:
    for row in sampler:                  # iterate matching rows
        print(row)

FileFlags

An enum.IntFlag, so members combine and the result is still a FileFlags:

flags = pybcsv.FileFlags.BATCH_COMPRESS | pybcsv.FileFlags.NO_FILE_INDEX
pybcsv.FileFlags(10)                       # reconstructs the same combination
pybcsv.FileFlags.NO_FILE_INDEX in flags    # True

Combining flags did not work before 1.5.17 — the binding produced a bare int, which every write function then rejected as the wrong type. Any single flag worked; nothing else did. Upgrade if you need more than one.

Three of the five are settings; two are outputs.

flag	value	settable at open()?
NO_FILE_INDEX	2	yes
STREAM_MODE	4	yes
BATCH_COMPRESS	8	yes
ZERO_ORDER_HOLD	1	no — comes from row_codec
DELTA_ENCODING	16	no — comes from row_codec

The row-codec bits describe how the rows were actually encoded, so they are not a request: a writer replaces whatever you pass for them with its own codec's value, because a header claiming one codec while the rows use another is a file no reader can trust. Choose the codec where it is actually chosen:

writer = pybcsv.Writer(layout, "zoh")      # here — not in open()'s flags
writer.open(path, flags=pybcsv.FileFlags.BATCH_COMPRESS)
writer.file_flags()                        # what actually reached the header

Writer.file_flags() is new in 1.5.17; Reader.file_flags() has always reported the same thing for a file it has open. See docs/API_OVERVIEW.md for the full account.

Utility Functions

# Pandas integration (requires pandas)
pybcsv.write_dataframe(df, filename,
                       compression_level=6,
                       row_codec="delta",
                       type_hints=None)  # dict[str, ColumnType]
pybcsv.read_dataframe(filename, columns=None)  # -> pd.DataFrame

# CSV conversion (requires pandas)
pybcsv.from_csv(csv_file, bcsv_file, compression_level=6, type_hints=None)
pybcsv.to_csv(bcsv_file, csv_file)

# Columnar I/O (numpy arrays)
pybcsv.read_columns(filename) -> dict[str, np.ndarray | list[str]]
pybcsv.write_columns(filename, columns, col_order, col_types,
                     row_codec="delta", compression_level=6)
# Type utilities
pybcsv.type_to_string(column_type) -> str

Testing

pip install pybcsv[test]
python -m pytest tests/ -v

File Structure

python/
├── pybcsv/
│   ├── __init__.py           # Public API and exports
│   ├── __version__.py        # Version (setuptools-scm)
│   ├── bindings.cpp          # C++ nanobind bindings
│   └── pandas_utils.py       # Pandas/CSV integration
├── examples/
│   ├── basic_usage.py        # Core BCSV operations
│   ├── pandas_integration.py # DataFrame examples
│   ├── advanced_usage.py     # DirectAccess, Sampler, CSV, columnar I/O
│   └── performance_benchmark.py
├── tests/                    # 17 test modules (pytest)
├── benchmarks/               # Python benchmark runner
├── pyproject.toml
└── README.md

Known Limitations

• Arrow string columns: 2 GB per batch. The Arrow C Data Interface uses utf8 format ("u") with int32 offsets, limiting the total byte size of any single string column within one batch to ~2 GB. An OverflowError is raised at runtime if this limit is exceeded. For most workloads this is not an issue. If you hit this limit, consider splitting data into smaller batches.

• No native null/missing value support. BCSV is a fixed-width binary format without a null bitmap. When writing a pandas DataFrame with NaN/None values, they are coerced to zero, False, or empty string by default (with a warning). Use strict=True in write_dataframe() to reject NaN values instead.

Compatibility

• Python: 3.11, 3.12, 3.13
• Platforms: Linux (x86_64, ARM64), macOS (x86_64, ARM64), Windows (AMD64)
• Compilers: GCC 13+, Clang 16+, MSVC 2022 17.4+, Apple Clang (Xcode 15.4+)
• C++ Standard: C++20
• Dependencies:
  • numpy >= 1.19.0 (required)
  • pandas >= 1.0.0 (optional — pip install pybcsv[pandas])

License

MIT — see LICENSE for details.

Publishing

Wheels are built automatically via GitHub Actions (cibuildwheel) and published using Trusted Publisher (OIDC) — no API tokens required.

• TestPyPI: every push to main/master or version tags
• PyPI: only on v* tags (e.g. git tag v1.4.0 && git push origin v1.4.0)

3. Trigger the publish workflow:

• The workflow triggers on pushes to the release branch or via manual workflow_dispatch.

4. Install from TestPyPI for verification:

# in a fresh virtualenv
python -m venv venv && source venv/bin/activate
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pybcsv
python -c "import pybcsv; print(pybcsv.__version__)"

If the import and version check succeed the wheel is good for release.