Skip to content

Commit cbbd28b

Browse files
smypmsaclaude
andauthored
viewer: make addresses, tables and charts usable for analysis (#6)
* cli: name the missing snapshot and the stray key in errors A dataset in an ingest project whose snapshot apply does not write (a lookup table filled in by a script afterwards) passed validation, since ingest snapshots appear at apply time, and then failed the build with an internal error carrying the stat call's ENOENT. It is now a validation error naming the dataset, its path and what to do. Schema errors for an additional property now name the key. "must NOT have additional properties" alone sent a reader hunting; the key usually gives the cause away, as when an unquoted comma in a flow-mapping title splits it into a second key. The missing-snapshot pointer for dataset-only projects also narrows from /datasets to the entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * viewer: copy, explorer links, column labels, SQL, CSV and filter A dashboard over wallets, contracts and transactions could not be read or checked: every hex value was shortened with the full one only in a tooltip, nothing linked anywhere, headers were SQL aliases, and an unexplained dot marked amount columns. - Every hex cell has a copy button for the full value; a panel's columns.<name>.full shows it unshortened. - chain_sources[].explorer_url links address, transaction and block cells. Addresses are detected from their values, tx_hash and block_number by name; a 32-byte value is never guessed to be a transaction, since it may be any id. - columns.<name>.label, .description and .kind on a panel. A header with a description is underlined and explains itself on hover; an amount column explains its decimals. The dot is gone. - Each panel links to its SQL in the source bundle, tables and charts download their rows as CSV with exact values, and a table over ten rows gets a filter. - Bar charts with up to 30 bars label every bar, tilted when crowded; past that the axis is a scale and labels thin out as before. - A KPI beside a table or chart keeps its own height instead of stretching into an empty card; two KPIs side by side still level. explorer_url and the SQL paths are written into the dashboard doc, not release.json, so they count toward the content digest. Releases built before this render as they did, without the new links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: list panel columns and the layout rules The Presentation paragraph listed the panel options without columns, and capabilities did not say how bar labels and KPI height now behave. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * viewer: link only web explorers and the release's own SQL Validation holds explorer_url to http(s), but the viewer renders whatever dashboard JSON it is served, and a release's JSON can be edited after the build: a javascript: base would have become a clickable script. explorerHref now refuses a base that is not http(s), and a panel's SQL link renders only for a path inside the release's source/ directory. security.md says what the viewer turns into links and why a cell value cannot carry a scheme into one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: viewer/dist is built, not committed compatibility.md still described the bundle as committed under viewer/dist/, which .gitignore has excluded since the bundle and its sources could drift. Match the README: pnpm build produces it, and build refuses to write a release without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent a2f2022 commit cbbd28b

15 files changed

Lines changed: 971 additions & 118 deletions

File tree

‎README.md‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ chain_sources:
7878
- id: mainnet
7979
chain_id: 1
8080
rpc_secret: RPC_URL # the env var name, never the value
81+
explorer_url: https://etherscan.io # optional; links addresses, txs, blocks
8182
finality: { policy: finalized }
8283

8384
event_sources:
@@ -150,15 +151,39 @@ from a closed set — `validation`, `policy_refused`, `missing_credentials`,
150151

151152
## Presentation
152153

153-
Panels are declarative. `title`, `description`, `span`, `hide_columns` and
154-
`unit` shape the page; `decimals`, `symbol` and `label` on a raw amount column
155-
shape the numbers. Charts are `line`, `bar`, `area`, `kpi`, `table` — an
154+
Panels are declarative. `title`, `description`, `span`, `hide_columns`,
155+
`unit` and `columns` shape the page; `decimals`, `symbol` and `label` on a raw
156+
amount column shape the numbers. Charts are `line`, `bar`, `area`, `kpi`, `table` — an
156157
allowlist, not an embedded plotting language.
157158

158159
Display metadata never alters stored values. `decimals: 6` renders
159160
`983644533552` as `983,644.533552 USDC`; the exact integer stays in the result
160161
JSON and in the hover title.
161162

163+
`columns` on a panel names, explains and types what a query returns:
164+
165+
```yaml
166+
panels:
167+
- query: largest_sells
168+
chart: table
169+
columns:
170+
wallet: { label: Seller, description: "The wallet that sold." }
171+
tx_hash: { label: Transaction, full: true }
172+
order_id: { kind: text } # a bytes32 id, not a transaction
173+
```
174+
175+
`label` replaces the header text and `description` appears when hovering it; a
176+
header with a description is underlined. `kind` is `address`, `tx`, `block` or
177+
`text`. Left out, a 20-byte hex value is read as an address, a column named
178+
`tx_hash` as a transaction and one named `block_number` as a block. A 32-byte
179+
value is never guessed to be a transaction, since it may as well be any id.
180+
With an `explorer_url` on the chain source, those cells link to the explorer.
181+
Every hex cell has a copy button that copies the full value; `full: true` also
182+
shows it unshortened.
183+
184+
Every panel links to the SQL behind it, tables and charts download their rows
185+
as CSV with exact values, and a table longer than ten rows gets a filter box.
186+
162187
## What a release weighs
163188

164189
A release is a static page plus the answers, and that part is about 800 KB

‎docs/capabilities.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,12 @@ Machine-readable source of truth: `chainplot capabilities --json`.
3434
- Publish root: beside `latest.json`, `publish` writes an `index.html` that forwards to the release the pointer names. A release directory is content-addressed, so its URL moves whenever the release does — a change to the viewer bundle is enough, since the bundle is part of the digest. The root is the URL that stays put, and `publish` returns it as `entry_url`. The page is written both at `<prefix>/index.html` and at `<prefix>/`: a host that resolves directories finds the former, an object store that serves keys needs the latter, and writing both keeps the bare URL working on either without a rewrite rule at the CDN. That second key is best effort: a prefix-less target has no directory key, a `directory` target cannot name a file that way, a store may refuse a key ending in a separator, and a foreign object already there is left alone. `entry_url` then names `index.html` explicitly, so the URL returned resolves either way. An `index.html` already at that key without chainplot's marker is left alone and `entry_point_written` comes back false, so publishing into a bucket that serves a site of its own does not replace that site's front page. On `s3` that refusal is enforced by a conditional write and holds against a concurrent writer; on `directory` it is a check followed by a rename, so a foreign file written into the same key mid-publish is overwritten. The page reads `latest.json` in the browser rather than naming a release, so its bytes depend only on the prefix: every publish writes the same page, and the entry point cannot fall behind the pointer whatever order concurrent publishers finish in. It needs script, as does the viewer it forwards to. Both the page and `latest.json` are written `no-cache, must-revalidate`, since a cache that serves either without asking would show an older release.
3535
- Charts: `line`, `bar`, `area`, `kpi`, `table` (allowlisted encodings only)
3636
- Dataset modes: `results_only` (default), `dataset_referenced`, `dataset_included`. The default publishes the page and its results only. `dataset_referenced` uploads the parquet beside the release and records a release-relative path and checksum, so `fork` fetches and verifies it on demand. `dataset_included` copies it into the release. Set per build (`--mode`) or per project (`policy.release_mode`).
37-
- Panel presentation: `title`, `description`, `span` (`half`/`full`), `hide_columns`, `unit`
37+
- Panel presentation: `title`, `description`, `span` (`half`/`full`), `hide_columns`, `unit`, `columns`
3838
- Column display: `raw_amount_columns[].decimals` / `.symbol` / `.label` (display only; stored values are never rewritten)
39+
- Panel columns: `columns.<name>.label` / `.description` (header tooltip) / `.kind` (`address`, `tx`, `block`, `text`) / `.full` (hex unshortened). Addresses are detected from their values, `tx_hash` and `block_number` by name; 32-byte values are never guessed to be transactions.
40+
- Explorer links: `chain_sources[].explorer_url` links address, transaction and block cells to `<url>/address/…`, `/tx/…`, `/block/…`. The first chain source that declares one is used.
41+
- Viewer affordances: copy button on every hex cell, a link to each panel's SQL in the source bundle, CSV download of a table's or chart's rows (exact values, query column names), a filter on tables over ten rows
42+
- Layout: a bar chart with up to 30 bars labels every bar, tilted when crowded; past that its axis thins labels like a line's. A KPI beside a table or chart keeps its own height; two KPIs side by side are levelled
3943

4044
## SQL admission control
4145

‎docs/compatibility.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,14 +50,14 @@ Full static build on `main`: SELECT model graph (topo order, cycles refused at
5050
`validate`, SELECT-only enforced in the isolated worker), extended `build`
5151
(full §16.1 layout minus `latest.json`: `index.html`, `assets/`,
5252
`dashboards/`, `results/`, `datasets/<id>/{manifest.json,tables/*.parquet}`,
53-
`source/` allowlist), viewer bundle (React + Vite + ECharts, committed under
54-
`viewer/dist/`, no CDN), `serve` (127.0.0.1 by default, `--host` to change), `plan --intent build`
53+
`source/` allowlist), viewer bundle (React + Vite + ECharts, built into
54+
`viewer/dist/` by `pnpm build` and not committed, no CDN), `serve` (127.0.0.1 by default, `--host` to change), `plan --intent build`
5555
(no RPC). A2 + A9 groundwork pass offline.
5656

5757
| Piece | Pin |
5858
| --- | --- |
5959
| viewer deps | react 19, echarts 6, vite 7 (`viewer/package.json`, own lockfile) |
60-
| viewer bundle | committed `viewer/dist/`; rebuild with `pnpm --dir viewer build` |
60+
| viewer bundle | built into `viewer/dist/` by `pnpm build`, not committed; `build` refuses to run without it |
6161
| release id | `local-<ts>` staged then renamed to `dist/releases/local` (M4 adds real ids + `latest.json`) |
6262

6363
Notes:

‎docs/security.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,17 @@ the statement count, so `SELECT 1; DROP TABLE t` is refused as two statements
5050
rather than passing a check aimed at the first. The statement is parsed, not
5151
run, by this call.
5252

53+
### Links the viewer renders
54+
55+
A dashboard turns project-supplied strings into links: `explorer_url` on a
56+
chain source, and each panel's SQL path. Validation holds `explorer_url` to
57+
`http(s)://`, and the viewer checks again on what it is actually served, since
58+
a release's JSON can be edited after the build: an explorer base that is not
59+
http(s) produces no link, and a SQL path renders only when it stays inside the
60+
release's `source/` directory. Cell values reach a link only after matching
61+
the shape their kind requires (20-byte address, 32-byte hash, decimal block),
62+
so a value cannot carry markup or a scheme into an `href`.
63+
5364
## What is *not* defended
5465

5566
- **Authenticity of a published release.** `release.json` lists a SHA-256 for

‎schemas/project.schema.json‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,24 @@
107107
"type": "array",
108108
"items": { "type": "string" }
109109
},
110-
"unit": { "type": "string", "maxLength": 16 }
110+
"unit": { "type": "string", "maxLength": 16 },
111+
"columns": {
112+
"description": "Per-column presentation, keyed by the column name the query returns.",
113+
"type": "object",
114+
"additionalProperties": {
115+
"type": "object",
116+
"additionalProperties": false,
117+
"properties": {
118+
"label": { "description": "Header text, in place of the column name.", "type": "string", "maxLength": 60 },
119+
"description": { "description": "Shown when hovering the header.", "type": "string", "maxLength": 240 },
120+
"kind": {
121+
"description": "What the values are. address, tx and block link to the chain's explorer_url. Left out, a 20-byte hex value is read as an address, a column named tx_hash as a transaction and one named block_number as a block; text turns that off.",
122+
"enum": ["address", "tx", "block", "text"]
123+
},
124+
"full": { "description": "Show hex values in full instead of shortened. Long hashes widen the table.", "type": "boolean" }
125+
}
126+
}
127+
}
111128
}
112129
}
113130
}
@@ -132,6 +149,12 @@
132149
"id": { "type": "string" },
133150
"chain_id": { "type": "integer" },
134151
"rpc_secret": { "type": "string" },
152+
"explorer_url": {
153+
"description": "Block explorer base URL, without a trailing slash. Table cells holding addresses, transactions and blocks link to <url>/address/<a>, <url>/tx/<hash> and <url>/block/<n>, the Etherscan and Blockscout layout.",
154+
"type": "string",
155+
"pattern": "^https?://[^\\s]+[^/\\s]$",
156+
"maxLength": 200
157+
},
135158
"finality": {
136159
"oneOf": [
137160
{ "type": "object", "additionalProperties": false, "required": ["policy"],

‎src/project/types.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,15 @@ export interface Query {
2929

3030
export type ChartKind = "line" | "bar" | "area" | "kpi" | "table";
3131

32+
export type ColumnKind = "address" | "tx" | "block" | "text";
33+
34+
export interface PanelColumn {
35+
label?: string;
36+
description?: string;
37+
kind?: ColumnKind;
38+
full?: boolean;
39+
}
40+
3241
export interface DashboardPanel {
3342
query: string;
3443
chart: ChartKind;
@@ -37,6 +46,7 @@ export interface DashboardPanel {
3746
span?: "half" | "full";
3847
hide_columns?: string[];
3948
unit?: string;
49+
columns?: Record<string, PanelColumn>;
4050
}
4151

4252
export interface Dashboard {
@@ -61,6 +71,7 @@ export interface ChainSource {
6171
id: string;
6272
chain_id: number;
6373
rpc_secret: string;
74+
explorer_url?: string;
6475
finality: Finality;
6576
}
6677

‎src/project/validate.ts‎

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -31,16 +31,33 @@ function error(
3131
};
3232
}
3333

34+
function strayProperty(err: ErrorObject): string | null {
35+
if (err.keyword !== "additionalProperties") return null;
36+
const prop = (err.params as { additionalProperty?: string }).additionalProperty;
37+
return typeof prop === "string" ? prop : null;
38+
}
39+
3440
function pointerFromAjv(err: ErrorObject): string | null {
35-
if (err.keyword === "additionalProperties") {
36-
const prop = (err.params as { additionalProperty?: string }).additionalProperty;
37-
if (typeof prop === "string") {
38-
return `${err.instancePath}/${prop}`;
39-
}
40-
}
41+
const prop = strayProperty(err);
42+
if (prop !== null) return `${err.instancePath}/${prop}`;
4143
return err.instancePath || null;
4244
}
4345

46+
/**
47+
* Ajv's text, with the stray key named. "must NOT have additional properties"
48+
* alone sends a reader hunting; the key usually gives the cause away, as when
49+
* an unquoted comma in a flow-mapping title splits it into a second key.
50+
*/
51+
function messageFromAjv(errors: ErrorObject[]): string {
52+
return errors
53+
.map((err) => {
54+
const text = `project${err.instancePath} ${err.message ?? "is invalid"}`;
55+
const prop = strayProperty(err);
56+
return prop === null ? text : `${text}: ${JSON.stringify(prop)}`;
57+
})
58+
.join(", ");
59+
}
60+
4461
export function validateProject(
4562
doc: unknown,
4663
projectDir: string,
@@ -66,8 +83,8 @@ export function validateProject(
6683
if (!validateSchema(doc)) {
6784
const first = validateSchema.errors?.[0];
6885
const pointer = first ? pointerFromAjv(first) : null;
69-
const message = first
70-
? ajv.errorsText(validateSchema.errors, { dataVar: "project" })
86+
const message = validateSchema.errors?.length
87+
? messageFromAjv(validateSchema.errors)
7188
: "invalid project document";
7289
return {
7390
ok: false,
@@ -154,15 +171,15 @@ export function validateProject(
154171
// Snapshot files are authored inputs for dataset-only projects. Ingest
155172
// projects materialize them at apply time, so their absence is not an error.
156173
const isIngest = (project.event_sources ?? []).length > 0;
157-
for (const dataset of project.datasets ?? []) {
174+
for (const [index, dataset] of (project.datasets ?? []).entries()) {
158175
if (isIngest) break;
159176
const filePath = path.resolve(projectDir, dataset.snapshot);
160177
if (!fs.existsSync(filePath)) {
161178
return {
162179
ok: false,
163180
error: error("validation", `missing snapshot file: ${dataset.snapshot}`, {
164181
resource_id: dataset.id,
165-
pointer: "/datasets",
182+
pointer: `/datasets/${index}/snapshot`,
166183
}),
167184
};
168185
}

‎src/publish/writeRelease.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,21 @@ export async function buildRelease(
150150
// lets someone fork the release and recompute, and it is also what turns an
151151
// 800 KB page into hundreds of megabytes.
152152
const mode = opts.mode ?? project.policy?.release_mode ?? "results_only";
153+
// A snapshot chainplot did not write itself — a lookup table a script fills
154+
// in after `apply`, say — may simply not exist yet. Say which one, rather
155+
// than surfacing the stat call's ENOENT as an internal error.
156+
for (const [index, dataset] of datasets.entries()) {
157+
if (!fs.existsSync(path.resolve(projectDir, dataset.snapshot))) {
158+
throw {
159+
...error(
160+
"validation",
161+
`dataset ${dataset.id} has no snapshot at ${dataset.snapshot}`,
162+
{ resource_id: dataset.id, pointer: `/datasets/${index}/snapshot` },
163+
),
164+
suggested_next: "produce the snapshot (`apply` writes an event source's own), then build again",
165+
} satisfies CommandError;
166+
}
167+
}
153168
let copiedBytes = 0;
154169
for (const dataset of datasets) {
155170
const parquetPath = path.resolve(projectDir, dataset.snapshot);
@@ -307,6 +322,18 @@ export async function buildRelease(
307322
// Dashboards data. Panel headings are resolved here — panel title, then
308323
// the query's title, then its id — so the viewer needs no query registry.
309324
const queryTitles = new Map(queries.map((q) => [q.id, q.title ?? q.id]));
325+
// Each panel links to the SQL behind it, which the source bundle ships
326+
// under source/. Only files inside the bundle's allowlist get a link.
327+
const querySql = new Map(
328+
queries
329+
.map((q) => [q.id, path.posix.normalize(q.file.replace(/\\/g, "/"))] as const)
330+
.filter(([, file]) => file.startsWith("queries/"))
331+
.map(([id, file]) => [id, `source/${file}`]),
332+
);
333+
// Lives here rather than in release.json so that it counts toward the
334+
// content digest: it changes what the page renders.
335+
const explorerUrl =
336+
project.chain_sources?.find((c) => c.explorer_url)?.explorer_url ?? null;
310337
const dashboardIds: string[] = [];
311338
for (const dashboard of project.dashboards ?? []) {
312339
const rel = path.join("dashboards", `${dashboard.id}.json`);
@@ -315,10 +342,12 @@ export async function buildRelease(
315342
dashboard_id: dashboard.id,
316343
title: dashboard.title,
317344
description: dashboard.description ?? null,
345+
explorer_url: explorerUrl,
318346
panels: dashboard.panels.map((panel) => ({
319347
...panel,
320348
title: panel.title ?? queryTitles.get(panel.query) ?? panel.query,
321349
span: panel.span ?? "half",
350+
sql: querySql.get(panel.query) ?? null,
322351
})),
323352
});
324353
files.push(rel);

‎tests/cli/build.test.ts‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,9 @@ describe("build", () => {
5151
fs.readFileSync(path.join(dist, "dashboards/overview.json"), "utf8"),
5252
);
5353
expect(dash.title).toBe("Amounts");
54+
// Each panel links to its SQL in the source bundle; no chain, no explorer.
55+
expect(dash.panels[0].sql).toBe("source/queries/raw_amounts.sql");
56+
expect(dash.explorer_url).toBeNull();
5457

5558
const raw = JSON.parse(
5659
fs.readFileSync(path.join(dist, "results/raw_amounts.json"), "utf8"),
@@ -63,6 +66,80 @@ describe("build", () => {
6366
expect(fs.existsSync(path.join(dist, "source/.env"))).toBe(false);
6467
expect(fs.existsSync(path.join(dist, "source/.chainplot"))).toBe(false);
6568
});
69+
70+
// An ingest project may declare a snapshot that `apply` does not write — a
71+
// lookup table a script fills in afterwards. Validation lets it through,
72+
// since ingest snapshots appear at apply time, and the build used to fail on
73+
// it with an internal error carrying the stat call's ENOENT.
74+
it("names a dataset whose snapshot is missing", async () => {
75+
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "chainplot-build-missing-"));
76+
fs.cpSync(template, dir, { recursive: true });
77+
fs.mkdirSync(path.join(dir, "abis"));
78+
fs.writeFileSync(
79+
path.join(dir, "abis/ERC20.json"),
80+
JSON.stringify([
81+
{
82+
type: "event",
83+
name: "Transfer",
84+
anonymous: false,
85+
inputs: [
86+
{ name: "from", type: "address", indexed: true },
87+
{ name: "to", type: "address", indexed: true },
88+
{ name: "value", type: "uint256", indexed: false },
89+
],
90+
},
91+
]),
92+
);
93+
const yaml = path.join(dir, "chainplot.yaml");
94+
fs.writeFileSync(
95+
yaml,
96+
fs
97+
.readFileSync(yaml, "utf8")
98+
.replace(
99+
"datasets:\n",
100+
[
101+
"chain_sources:",
102+
" - { id: mainnet, chain_id: 1, rpc_secret: RPC_URL, finality: { policy: finalized } }",
103+
"event_sources:",
104+
" - id: usdc",
105+
" chain: mainnet",
106+
' addresses: ["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"]',
107+
" abi: abis/ERC20.json",
108+
" events: [Transfer]",
109+
" start_block: 100",
110+
" end: { mode: pinned, block: 110 }",
111+
"datasets:",
112+
" - id: lookups",
113+
" snapshot: .chainplot/lookups/blocks.parquet",
114+
"",
115+
].join("\n"),
116+
),
117+
);
118+
fs.mkdirSync(path.join(dir, ".chainplot"));
119+
fs.writeFileSync(
120+
path.join(dir, ".chainplot/coverage.json"),
121+
JSON.stringify({
122+
schema_version: 1,
123+
chain_id: 1,
124+
sources: [
125+
{
126+
source_id: "usdc",
127+
segments: [{ start_block: 100, end_block: 110, status: "complete_empty", row_count: 0 }],
128+
},
129+
],
130+
}),
131+
);
132+
133+
const result = await runCliJson(["build", "--json"], dir);
134+
expect(result.ok).toBe(false);
135+
expect(result.error).toMatchObject({
136+
code: "validation",
137+
resource_id: "lookups",
138+
pointer: "/datasets/0/snapshot",
139+
});
140+
expect(result.error?.message).toContain(".chainplot/lookups/blocks.parquet");
141+
expect(result.error?.suggested_next).toBeTruthy();
142+
});
66143
});
67144

68145
// Models are materialized into every query's session. While each session

0 commit comments

Comments
 (0)