Zercher Log Viewer

A web-based log viewer for browsing and searching large log files. Files are streamed into a persistent SQLite index on the server, and the UI uses virtual scrolling so it stays responsive on files with many thousands of lines.

Features

• Streaming upload & indexing — pick a file in the browser; it is streamed straight into the index on the server. The raw file is never written to disk — only the index is kept. Because the file is uploaded, this works even when the browser and server are on different machines.
• Content-addressed indexes — each file is indexed into its own database named by a hash of its contents, so multiple files can be open at once (in different tabs) without clobbering each other. Reopening the same file is instant (a stat fast-path skips re-reading it), and two copies of identical content — even under different names — share a single index. A bounded LRU reclaims the oldest unused indexes from disk.
• Parallel, cancellable search — queries run in a pool of worker threads, so the UI stays responsive. Regex scans are sharded across CPU cores for speed, and a runaway pattern (catastrophic backtracking) is hard-killed on a deadline (ZERCHER_QUERY_TIMEOUT_MS, default 5s) instead of stalling the server.
• Multi-session lifetime management — each tab holds a session id and heartbeats the corpus it's viewing. A corpus stays pinned (never evicted) while any session needs it, and is reclaimed once no session has referenced it for a while — so concurrent sessions on different files coexist safely and idle indexes are cleaned up automatically.
• Admin dashboard — open ?admin for a live, read-only operational view: connected sessions (session id, file, content hash, filter/highlight counts, number of searches run — the term itself is not exposed — and last-heartbeat age), indexed files (source, hash, refcount, line count, longest line, on-disk size, trigram, in-use/idle state), the worker pool (busy/queued), GC configuration, and totals. Refreshes every few seconds.
• Substring or regex search with match navigation (prev/next; the current match is highlighted darker than the others). Toggle the .* button to interpret the input as a JavaScript regular expression instead of literal text.
• Filters with AND/OR, plus per-term enable/disable, case sensitivity, a per-term literal/regex (.*) toggle, and a per-term exclude (≠) toggle that hides lines matching the term (e.g. keep timeout but drop ERROR).
• Highlights — colour-code terms (literal or regex) using an auto-rotating palette.
• Log-level awareness — a level (ERROR/WARN/INFO/DEBUG/TRACE/ FATAL, with common synonyms) is parsed per line at index time. Each line's gutter is tinted by severity, and a levels disclosure on the Filters header reveals one-click level chips with per-level counts; hovering a chip shows the detection regex behind it. Toggling levels scopes search/filter/counts/export (unlevelled continuation lines, e.g. stack traces, stay visible).
• Timestamp histogram & time-range filter — a timestamp is parsed per line (ISO-8601, YYYY/MM/DD, Apache CLF, BSD syslog, epoch). When a file has timestamps, a volume-over-time strip appears below the header, stacked by severity (ERROR/FATAL anchored at the baseline so error spikes are easy to spot over time); hovering a bar shows its per-level breakdown. Drag across it to brush a time range that scopes the view (and search/filter/counts/export). Toggle it in the Display panel.
• JSON pretty-print — lines that are (or contain) a JSON object/array get a { } toggle that expands them in place as indented, readable JSON; any leading prefix (timestamp/level) is preserved.
• Recent searches — the last searches are remembered and offered in a dropdown under the search box (filtered as you type; individually removable).
• Bookmarks & Visible Ranges — pin lines to keep them on screen, listed together in one Visible Ranges panel. Click a line number to bookmark it (shown in yellow), or add a line/range (e.g. 3-5) as a plain "visible range" that stays on screen even when filters would otherwise hide it. In the panel, bookmarked entries are marked with *; the star toggles an entry between bookmark (yellow) and plain visible range.
• Gap indicators & context expansion — non-adjacent lines are separated by a thin strip showing how many lines are hidden, with buttons to reveal the lines just above/below.
• Virtual scrolling — only the lines near the viewport are fetched and rendered, regardless of file size.
• URL state — file, filters, highlights, bookmarks, search term, and the selected line all persist in the URL so a view can be shared.
• Copy as shell command — from the ⋮ menu, copy a grep/awk command equivalent to the current view (filters + AND/OR + case/regex/exclude, plus the bookmarked and Visible-Range lines), so you can reproduce the same filtered output on the command line or on another machine.
• Export text — from the ⋮ menu, download the current filtered view as a text file (raw lines, or with N: line-number prefixes). The download is streamed, so it works on large result sets.
• Minimap — a semi-transparent density strip over the scrollbar showing where search matches, bookmarks, visible ranges, and the selected line fall in the current view, plus a box for the viewport; click to jump. It overlays the scrollbar gutter, so it never narrows the text. Toggle in the Display panel.
• Keyboard shortcuts — / focus search, n/N next/previous match, g/G jump to top/bottom of the view, and ? for an in-app shortcut cheat-sheet (also in the ⋮ menu).
• Display options — zebra striping, directional gap expansion, the time histogram, and an opt-in trigram index for faster search on large files.

Screenshots

Overview — timestamps, level tinting, histogram	Time histogram — severity-stacked; drag to brush a range
Log levels — chips with counts (regex on hover)	JSON pretty-print — expand a JSON line in place
Search & match navigation (regex mode)	Filters & highlights
Recent searches	Bookmarks & visible ranges
Gap indicators & context expansion	Minimap — density strip over the scrollbar
⋮ menu — open / copy as shell command / export	Keyboard shortcuts (?)
Admin dashboard (?admin)

Features not pictured are non-visual: streaming upload/indexing, content-addressed indexes, the parallel/cancellable worker-thread search, virtual scrolling, multi-session lifetime (visible indirectly in the admin dashboard), and URL state.

Screenshots are generated from the e2e fixtures. To regenerate them: SHOTS=1 npx playwright test e2e/screenshots.spec.ts (writes to docs/images/).

Tech Stack

• Backend: Node.js + TypeScript + Express + better-sqlite3
• Frontend: React + TypeScript + react-window + Vite
• Single server — the backend serves both the API and the built frontend.

Installation

cd /Users/azaimi/zercher
npm install

Usage

Start the application

npm start

This builds the frontend and the server, then runs the compiled server on port 3002, serving both the web app and the API. Then open:

http://localhost:3002

The server is run compiled (plain Node) so the search worker threads load as .js on any platform. There is also npm run start:tsx, which runs the server straight from TypeScript via tsx — convenient, but on some setups tsx's TypeScript loader doesn't reach worker threads (you'll see "Unknown file extension .ts" and a log line about falling back to main-thread query execution). If you hit that, use npm start (compiled). The app still works under the fallback — just without parallel/cancellable search.

Development mode (auto-reload)

npm run dev

Rebuilds the frontend and watches the server for changes.

How to Use

1. Load a log file:

   • Open the ⋮ menu (top right) and choose Open File, then pick a log file from your machine.
   • It streams to the server and indexes on the fly; the progress bar shows real indexing progress.
   • Each file is indexed into its own content-addressed database, so several files can be open at once in different tabs. Reopening a file you've already loaded is instant, and identical content (even under another name) reuses the same index. The server keeps a bounded number of recent indexes and reclaims the oldest from disk.

2. Search:

   • Type a term and press Enter (or click Search). Matching is exact substring (e.g. 15T10:33 matches that literal text).
   • Use the ‹ / › buttons to jump between matches; the current match is a darker yellow, the others light yellow.
   • Selecting text in the log highlights its other occurrences in teal.

3. Build filters:

   • Type a term and click Filter to add it to the filter list.
   • Toggle AND/OR; AND shows lines matching all enabled filters, OR shows lines matching any.
   • Per filter: enable/disable with the checkbox, toggle case sensitivity (aa/Aa), literal/regex (.*), exclude (≠ — hide lines matching the term instead of keeping them), or remove with ×.
   • Exclude composes with AND/OR: in AND mode timeout + excluded ERROR shows lines with timeout that don't contain ERROR.

4. Highlight terms:

   • Type a term and click Highlight to colour every occurrence (colours rotate automatically through a 20-colour palette).

5. Bookmarks & context:

   • Click a line number to bookmark it (shown in yellow); it appears in the Visible Ranges panel marked with *.
   • Where displayed lines aren't contiguous, a thin gap strip shows how many lines are hidden, with ↓ 5 below / ↑ 5 above to reveal the lines adjacent to the neighbouring visible lines. With Directional Gap Expand on, those revealed lines are added to Visible Ranges (plain content), not bookmarked.
   • Shift-click a line number toggles ±5 lines of context around it.

6. Visible Ranges (right panel) — bookmarks and visible ranges share this one list:

   • Type a line or range (e.g. 5, 3-5, 3-5,8) and click + to keep those lines always on screen as ordinary (zebra-striped) content rather than as yellow bookmarks or dimmed context.
   • Each entry has a * toggle: starred entries are bookmarks (yellow); unstarred entries are plain visible ranges. Click the star to switch an entry between the two.
   • Useful on large files: filter/regex away everything you don't want, then pin the regions you need to watch. Click an entry to jump to it; × to remove. Bookmarks persist in the URL as bookmarks=4,7 and visible ranges as visible=3-5,8.

7. Copy as shell command (⋮ menu):

   • Choose Copy as Shell Command to put a grep/awk pipeline equivalent to the current view on your clipboard. A simple AND of literal filters becomes a grep -F pipeline; anything using OR, regex, or forced bookmark/Visible-Range lines becomes a single awk command. Run it against your own copy of the log to reproduce the same lines. (Regex is translated to awk's dialect best-effort — a note is included when it may differ.)
   • Choose Export text to download the current view's lines as a .txt file, or Export with line numbers to prefix each line with its original line number (N:line). Same scope as the shell command (filters plus bookmarked and Visible-Range lines); with no filters it exports the whole file. The download is streamed, so large result sets work.

8. Filter by level & time (when the file carries them):

   • Click the small levels ▾ disclosure on the Filters header to reveal the level chips (ERROR/WARN/INFO/…) with per-level counts. Toggle chips to scope the view; hover a chip to see the detection regex behind it; reset restores all. The disclosure is tinted while a level filter is active.
   • When the file has timestamps, a volume-over-time strip appears below the header. Drag across it to brush a time range (shown as a selection, with a × to clear); the view, search, counts, and export all scope to that window.

9. Pretty-print JSON:

   • Hover a line that is (or contains) a JSON object/array and click the { } button to expand it as indented JSON in place; click again to collapse.

10. Select a line:

• Double-click a line to mark it (recorded as selected=<n> in the URL). Opening a URL with selected scrolls to that line.

11. Display options (right panel):

• Zebra Stripes — alternating row shading.
• Directional Gap Expand — when on, the gap buttons reveal only the 5 lines in the chosen direction; when off, they reveal ±5 around the neighbouring line.
• Minimap — show/hide the density strip over the scrollbar (match, bookmark, visible-range, and selected-line markers; click to jump).
• Time histogram — show/hide the volume-over-time strip (only offered when the file has parsed timestamps).
• Trigram index (faster search) — off by default. Turning it on (after a confirmation) builds a trigram index so Search/Filter/counts use it instead of scanning. It speeds up selective searches on large files but uses substantial extra disk (~4× the file) and takes time to build; turning it off drops it. Can be defaulted on with ZERCHER_TRIGRAM=1 or the --trigram flag.

Keyboard Shortcuts

These work whenever you're not typing in a text box:

Key	Action
/	Focus the search box
n / N	Jump to the next / previous search match
g / G	Scroll to the top / bottom of the current view
?	Show / hide the keyboard-shortcut cheat-sheet

Sharing Views

The URL updates automatically with the current view, so copying it shares the exact state:

http://localhost:3002/?file=app.log&filters=ERROR,timeout&enabled=1,1&mode=AND&highlights=Connection&search=pool&bookmarks=4,7&selected=120

Note: uploaded files live only in the index (the raw file isn't stored). The session pins to the loaded file's content-addressed id, so reopening a shared URL works as long as that index is still resident; if it has been reclaimed by the LRU, a server-side path is re-indexed automatically, while an uploaded file must be re-uploaded via the ⋮ menu.

Log Format

Any text file works. Each line is stored and indexed as-is; searching, filtering, and highlighting all operate on the full line text (literal substring match), so no particular log format is required. As a bonus, a timestamp and a log level are parsed per line best-effort during indexing — when present they enable the time histogram/range filter and the level chips/tinting, and when absent the line is simply treated as plain text.

Both text-prefixed and JSON lines are understood:

• Text prefix — a timestamp at the start of the line (ISO-8601/RFC 3339, YYYY/MM/DD, Apache CLF, BSD syslog, or bare epoch sec/ms) followed by a level token (ERROR/WARN/…, level=info, [INFO]). Recommended: 2026-06-22T14:03:30.123Z INFO message….
• JSON objects (one per line; zap, logrus, pino, bunyan, Elastic, MongoDB) — the timestamp is read from ts/time/timestamp/@timestamp/t and the level from level/severity/lvl/s. Values may be an RFC 3339 string, a numeric epoch (sec/ms), MongoDB extended JSON ({"$date":…}), and the level may be a word ("info"), a single letter ("s":"E"), or a number (pino 10–60 / syslog 0–7). Such lines also get the { } pretty-print.

Sub-millisecond precision is preserved in the displayed text but rounded to milliseconds for histogram bucketing and time-range filtering.

Time zones. Timestamps that carry an explicit offset (Z, +00:00, -0700) are absolute. Offset-less ("naive") timestamps — e.g. 2026/06/22 16:47:39 — are ambiguous, so by default they're interpreted as the host's local time zone. That's what lets a file mix local-time lines (a Datadog-style prefix) with UTC lines (JSON …Z) and have them land on one correct timeline. Override with ZERCHER_NAIVE_TZ: utc (treat naive as UTC), local (default), or a fixed offset like +02:00 / -0700. The histogram axis is labelled in UTC.

Roadmap / Future Ideas

Things deliberately left out of the current design, kept here so the intent isn't lost:

• Timestamp "jump to time" & gap visualization — the per-line timestamp is now parsed and powers the histogram and time-range filter (see Features); a natural next step is a "jump to time" input and showing real time gaps between non-adjacent lines.
• Live tail / follow mode — a tail -f equivalent: a Follow toggle that incrementally indexes appended bytes and auto-scrolls to the tail (pausing when you scroll up, with a "↓ N new" resume pill). The marquee feature, but it cuts against the current immutable-file design and needs its own pass:
  • Corpus identity is the content hash with a (path, size, mtime) fast-path, and indexing reads the whole file once — both assume the file never changes. A growing file would look brand-new on every write and re-index from scratch.
  • It needs a separate append-only corpus mode keyed by path+inode (exempt from content-dedup) that tracks the last byte offset and indexes only new bytes, continuing the line numbering — plus fs.watch/polling to detect growth and SSE (or polling) to push new lines to the client.
  • Correctness corners: rotation/truncation (logrotate, inode change), a partial trailing line with no newline yet, incremental filter/histogram/level updates so only new matches stream in, and one shared watcher across sessions following the same file.
  • Build it behind a flag so it can't destabilise the immutable-file path the rest of the app relies on.
• Authentication & per-user isolation — there is no auth or access control; any client can load any server-side path the process can read, and corpora are shared by content across sessions. Multi-session lifetime management is implemented (see Features / Architecture); multi-user security (logins, per-user namespaces, quotas) is the next layer if this is ever exposed beyond a trusted network.

Notes & Limitations

• Search/filter/highlight are literal substring by default, implemented with SQLite instr() (so %/_ are matched literally, and the aa/Aa toggle controls case). Toggle the .* button to interpret the term as a regular expression instead — evaluated per line via a custom SQLite REGEXP function (JavaScript RegExp semantics; the aa/Aa toggle maps to the i flag). Each filter and highlight remembers its own literal/regex mode, and the mode is preserved in the shareable URL (frx/hrx/srx).

• Default search scans the index, which is fine up to ~1 GB. Searches run in a worker-thread pool, and regex scans are sharded across cores, so a scan is parallelised and never blocks the server. For faster search on large files, enable the Trigram index (Display panel) — literal, case-insensitive terms of ≥3 chars then use it; case-sensitive, shorter, broad-result, or regex queries still scan (the trigram index can't accelerate arbitrary regex). A broad query returns all matching line numbers regardless of index, so very large result sets are limited by result size, not scan speed.

• Regex timeout — a query exceeding ZERCHER_QUERY_TIMEOUT_MS (default 5s) is cancelled by terminating its worker thread and returns 408, so a catastrophically backtracking pattern can't stall the server. The threshold is configurable via that env var.

• Multiple files coexist — each file is indexed into its own content-addressed database, so different tabs can view different files at once. A corpus is pinned while any session is viewing it (tracked by heartbeats) and is reclaimed once it has been idle past ZERCHER_IDLE_TTL_MS (default 10 min); a bounded LRU (ZERCHER_DB_MAX, default 8) is the hard cap over idle corpora. A request for a reclaimed index transparently re-indexes the file (for server-side paths) — uploads would need re-uploading. Sessions are reaped after ZERCHER_SESSION_TIMEOUT_MS (default 2.5 min) without a heartbeat.

• Uploads always re-index — an uploaded stream is hashed and indexed on arrival; if its content matches an existing index it's deduped, but there's no way to skip the upload itself.

• No authentication — every endpoint, including the ?admin dashboard and /api/admin (which exposes file paths, content hashes, and session ids), is unauthenticated. Only run this on a trusted network.

• Restrict which files can be loaded with ZERCHER_ROOT — by default /api/load will index (and then serve back) any path the server process can read, which is an arbitrary file read if the server is reachable by others. Set ZERCHER_ROOT to one directory, or several separated by :, to allow loads only from those trees:

  ZERCHER_ROOT=/var/log:/srv/applogs npm start

  Requests for paths outside the allowlist get 403. The check resolves symlinks (via realpath) so a link inside an allowed root can't point outside it. Uploads are unaffected (they stream bytes, not a path). Unset = unrestricted (fine for local single-user use); the server logs which mode it's in at startup.

Project Structure

zercher/
├── src/
│   ├── server/            # Backend (Node.js + Express)
│   │   ├── server.ts      # Express server (serves both API & frontend)
│   │   ├── corpus.ts      # CorpusRegistry: content-addressed DBs + LRU disk cap
│   │   ├── indexer.ts     # Streaming indexer + SQLite storage (per corpus)
│   │   ├── search.ts      # Search engine (main-thread; literal/regex)
│   │   ├── logParse.ts    # Per-line timestamp + log-level parsing (indexing)
│   │   ├── pathPolicy.ts  # ZERCHER_ROOT allowlist for /api/load (file-read guard)
│   │   ├── regexpFn.ts    # Shared regexp_match + WHERE builder (stack/time/level)
│   │   ├── searchPool.ts  # Worker-thread pool (sharded regex + deadline kill)
│   │   ├── searchWorker.ts# Worker entry: read-only query executor
│   │   ├── sessions.ts    # SessionTracker: per-tab refcounts + corpus lifetime
│   │   └── types.ts       # TypeScript types
│   └── client/            # Frontend (React)
│       ├── App.tsx        # Main app component
│       ├── main.tsx       # React entry point (renders App, or AdminView at ?admin)
│       ├── lib/           # Pure helpers (search history, JSON/level parsing, …)
│       └── components/    # React components (LogViewer, TimeHistogram,
│                          #   HelpOverlay, AdminView, MiniMap, …)
├── dist/                  # Built frontend (served by Express)
├── index.html             # HTML entry point
├── tsconfig.json          # TypeScript config (client)
├── tsconfig.server.json   # TypeScript config (server)
└── package.json           # Dependencies & scripts

Available Scripts

• npm start — build frontend + server, then run the compiled server
• npm run start:tsx — build frontend + run the server from TS via tsx (see the worker-threads note under Usage)
• npm run dev — development mode with auto-reload (tsx watch)
• npm run build — build both frontend and backend
• npm run build:frontend — build frontend only
• npm run build:backend — type-check/compile the server (tsconfig.server.json)

Architecture

The application runs as a single Express server on port 3002:

• /api/* — REST API for upload, search, line retrieval, stats, the time histogram (/api/histogram) and level counts (/api/levels), session heartbeats, and the admin summary (/api/admin)
• all other routes — serve the React frontend (including ?admin, the dashboard)

No separate frontend/backend servers or proxies needed.

Storage & query layers

• Corpus identity is explicit. A CorpusRegistry owns every indexed file as its own database under zercher-dbs/, named by the sha256 of the file's bytes (DB_DIR overrides the location). A (path, size, mtime) stat fast-path resolves a reopened file with no read; otherwise the file is indexed into a temp DB while hashing, then deduped by content hash. Each load returns a corpusId, which the client sends on every request so the server routes to the right index — there is no implicit "currently loaded" file. A bounded LRU (ZERCHER_DB_MAX) evicts the oldest unused corpus and deletes its files. The single indexing pass also parses a per-line timestamp and log level into indexed columns, which power the histogram, time-range filter, and level chips.
• Queries are composed from independent scopes. composeWhere builds one WHERE from the search stack plus optional time-range and level predicates, so the same SQL is shared by the main engine and the worker pool, and search, filter, counts, and export all honour the active time window and level selection identically.
• Queries run off the main thread. A SearchPool of cores − 2 worker threads executes search/count SQL against read-only connections. Regex stacks are sharded by line-number range across workers (results concatenate already sorted); literal/trigram queries run on one worker. A per-query deadline terminates and respawns the worker(s) running a runaway query — the only reliable way to stop catastrophic regex backtracking — and the request returns 408.
• Sessions own corpus lifetime. A SessionTracker maps each browser tab's session id to the corpora it is viewing, derived from its heartbeats (the heartbeat set is the source of truth). It keeps each corpus's refcount in sync: pinning while referenced, unpinning when a session switches files, detaches (navigator.sendBeacon on unload), or is reaped after a heartbeat timeout. Every client request carries its sessionId and refreshes the session's liveness, so an actively-used tab stays alive between heartbeats — the periodic heartbeat is really just the idle keepalive. A periodic sweep reaps dead sessions and then deletes corpora that have been idle (refcount 0) past the idle TTL. Pinned corpora are never evicted — an active session's data can't be reclaimed underneath it. Server restarts start clean (the DB directory is wiped and sessions are in-memory), so there are no orphans to recover.

Everything is keyed by corpus and by session, so the remaining step to a true multi-user deployment is access control (auth, per-user namespaces, quotas) — the storage, query, and lifetime layers are already isolation-ready.