stream-sorting is a streaming sort plus a suite of operations that consume or produce sorted object-mode Readable streams. The headline operation is a disk-backed external merge sort designed to sort a billion records on a laptop without OOMing. On top of the sort, the package builds:
join/leftJoin/fullJoin— key-based join of sorted streams (inner and outer)union/intersection/difference— set operations on sorted streamsmerge— k-way sorted merge without deduplication
stream-sorting is part of the stream-chain / stream-json family. Scope: anything that requires or preserves sortedness. Combinators that know nothing about sortedness (zip, select, race, concat) live in stream-join; 1→N split / route / filter ops live in stream-fork. Node-only — the package targets server / CLI contexts; browser is out of scope (node:fs is load-bearing for the disk-backed sort).
Runtime dependencies: stream-chain and stream-join. The k-way-merge phase of the sort composes stream-join's mergeSorted (built on select + sortedInsert); polyphaseSort and the join family use custom group-aware merges where a flat merge can't express their sub-stream structure. Distributed under New BSD license.
Early development. All operations are implemented — both sort algorithms (sort, polyphaseSort), the wrapper protocol, the join family, aggregate, the key-based filters, and the set operations (listed below). The package has not yet published its first npm release. See the wiki for usage and ARCHITECTURE.md for the design.
npm i stream-sortingObject-mode in, object-mode out: every operation accepts AsyncIterable<T> | Iterable<T> and returns an AsyncIterable<T>. Options, examples, and design notes live in the wiki.
import sort from 'stream-sorting/sort.js';
for await (const item of sort(input, {compare: (a, b) => a.id - b.id, tmpDir: '/var/sort'})) {
// items in ascending id order
}sort— disk-backed external (k-way) merge sort. The headline operation.polyphaseSort— polyphase merge sort; a fixed file budget, for bounded or heterogeneous storage.- Wrapper protocol —
ObjectStreamWrapperwith built-inMemoryWrapperandLocalFileWrapper. The storage abstraction the sorts read and write through, so a sort can exceed any single disk. join/leftJoin/fullJoin— key-based join of two or more sorted streams; named-map inputs, Cartesian product on equal keys,combineover a named bag, inner + outer shapes.aggregate— fold sorted child streams under a master by key (SQL-style group/aggregate); the master is a{input, key}stream or akey => objectfunction, children fold viainit/fold/finalize, one row per key.matching/unmatched— key-based filters: rows of the primary stream whose key is present / absent in the other (semi / anti-join).merge— k-way sorted merge, duplicates preserved; the suite's foundational op.union/intersection/difference— set operations on sorted streams.sortJsonl— text-JSONL convenience: parse lines →sort→ stringify (stream-sorting/utils/sort-jsonl.js).
| Package | Scope |
|---|---|
stream-chain |
Pipeline plumbing: chain, readableFrom, final. |
stream-json |
JSON streaming parser / generator. |
stream-join |
N→1 combinators that know nothing about sortedness. |
stream-fork |
1→N split / route / filter. |
stream-sorting |
Sort + sorted-stream operations. |
Together, the family closes the billion-row pipeline story end-to-end in pure Node streams: sort each input with stream-sorting, join the sorted streams, fork the output downstream.
- AI agent rules:
AGENTS.md. - Architecture:
ARCHITECTURE.md. - AI-facing reference:
llms.txt(short) andllms-full.txt(long). - Wiki (usage docs, design notes): https://github.com/uhop/stream-sorting/wiki.
Initial scaffold.