Repository navigation
Feat/notion renderer - #229
Merged
Merged
Conversation
Recovers the docs.giveth.io design from the live Super site while the subscription is still active, and ports it onto the existing Docusaurus theme. Salvaged into super-salvage/: - TeX Gyre Adventor, the heading font. Super hosted this as a one-off upload on their CDN, so it would have been lost when the subscription lapses. It is GUST Font License (free/libre), so self-hosting is fine. - Navbar and footer logos, light and dark variants. - super_nav.json: the full sidebar (12 sections, 36 pages) and navbar tree, decoded from the streamed RSC payload. Super renders navigation client-side, which is why a wget mirror captures the labels as bare text with no links. - The four served stylesheets, sitemap (131 URLs) and raw homepage. - DESIGN-TOKENS.md documenting the extracted design system. Theme port in src/css/custom.css: - Headings use TeX Gyre Adventor, body copy uses Red Hat Display, matching the Super site. Super uploaded only the Bold cut and declared it at weight 400; that mis-weighting is preserved deliberately so headings render as they do on the live site. - Brand purple #5326ec with generated Infima shades, plus the dark theme (#1b1657) that the Super site already shipped. - Super's layout measurements: 56px navbar, 20px radius, 241px sidebar, 1300px container. - Existing utility classes are preserved. Image borders move from the old pink to the brand purple, and --ifm-color-secondary is kept defined so .formatLink does not fall back to Infima's near-white default. Builds clean across all four locales. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Homepage: - Full-width cover banner, centred title block, and the four-card grid that the Super site used: What Is Giveth?, Donors & Project Guide, The GIVeconomy and DeVouch, each with its banner image and icon. - Card banners and the cover image were served from Super's CDN (images.spr.so) and are now salvaged into static/img/home/. Like the heading font, these would have been lost when the subscription lapses. - Cards carry short descriptions, which the Super originals did not have. Drop the description field for exact parity. - Layout uses the salvaged measurements: 1300px content width, 20px radius, 96px page padding (24px mobile), 35vh cover. redirects.js: Super served flat, all-lowercase URLs (/givbacks); Docusaurus serves nested, camelCase routes (/giveconomy/givbacks). Only 3 of the 36 navigated URLs are identical, so nearly every inbound link needs a redirect. The map covers 32 of them; /what-is-giveth/the-giveconomy/archived-staking-pools has no Docusaurus equivalent and still needs a decision. The file is inert until wired up — @docusaurus/plugin-client-redirects is not yet a dependency. Builds clean across all four locales. Every card link, image and font resolves against the built site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Super dashboard's Custom Code is empty ("css": "", "styles": ""), so none
of the design was hand-written CSS. It is Super's theme settings plus how Super
renders Notion blocks — which means porting it is a matter of restyling
Docusaurus components, not copying a stylesheet.
src/css/notion-blocks.css (new, wired into customCss):
- Admonitions restyled as Notion callouts: tinted panel, icon left and content
right, 1px border, 18px radius, no accent bar and no type label. Colours are
Super's two-tier Notion palette, light and dark.
- Page titles at Super's 3.5rem/700, tunable via --giveth-title-size.
- Image, code and table radii, and the 3px blockquote rule, from Super's
--image-border-radii, --code-border-radii and --quote-border.
- 1300px content column.
Measured from a full crawl of all 132 live pages rather than guessed: 100
images, 44 collections, 21 callouts across 12 pages, 42 pages with an icon and
10 with a cover.
Salvage:
- All 111 unique content images from images.spr.so, named by UUID, with
image-manifest.json mapping live page paths to the images they use.
- pages-html.tar.gz: raw HTML of all 132 pages.
Still unported: per-page covers and icons, the 44 collection card grids, and
the sidebar's Lucide section icons. Those are content-level rather than CSS.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Notion is the source of truth for docs.giveth.io — Super was only a renderer — so content should be regenerated from the API rather than reconstructed from the rendered HTML. scripts/notion_export.py walks the page tree from the Giveth Docs root, renders blocks to Docusaurus markdown, downloads images and writes an export report. Standard library only, so there is nothing to install. The tree is not a plain nest of sub-pages: the root holds an inline database whose four entries are the top-level sections, and those pages mix child_page blocks with further inline databases. Both are followed; walking only child_page blocks finds nothing. The full tree is 45 pages. Block coverage matches what the live site actually uses, measured from a crawl: paragraphs, headings (shifted down one so h1 stays the page title), lists, to-dos, callouts, quotes, code, images, video, tables, toggles, columns, bookmarks and equations. Callout colours map to the same admonition types as src/css/notion-blocks.css. Unhandled block types and unresolved Notion links are recorded in the report rather than silently dropped. .gitignore covered .env.local but not .env, where the Notion secret lives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Super stores a Notion pageId against each nav entry, but all 38 are stale — none match the current tree, because the pages were recreated in Notion since the Super site was built. Titles still correspond, so the mapping is by title. This lets the rebuilt site keep the live URLs rather than redirect to new ones, which removes the need for most of redirects.js. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Super's sidebar is hand-curated in its dashboard — 12 sections, 36 pages — and does not match the Notion tree, which has only 4 top-level sections. A faithful rebuild follows Super's arrangement, so the sidebar is generated from the recovered super_nav.json rather than from the Notion hierarchy. Reports both directions of mismatch: Super entries with no exported page, and exported pages absent from Super's sidebar (the Causes section, which postdates the Super build). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
45/45 pages, 100 images, 9 covers, no failures and no unhandled block types. The image count independently matches the notion-image count measured by crawling the live Super site. Staged rather than applied: adopting this replaces dapps/, whatisgiveth/ and devouch/, and the i18n/ translations for Spanish, Portuguese and Turkish have no Notion equivalent and would be lost. That is a call for the maintainers. Markdown only — the 100 images (~36MB) land in static/img/notion/ and can be regenerated from Notion at any time. The images that actually needed rescuing were Super's CDN copies, already in super-salvage/content-images/. Two exporter fixes since the first run: - Escape asterisks. Some pages contain literal markdown markers typed by hand that are also bold-annotated, which produced "****text****". - Section pages emit <DocCardList />. Their Notion body is an inline database, not prose, so they exported nearly empty; on Super they rendered as card grids, and DocCardList is the Docusaurus equivalent. gen_sidebar.py also handles top-level bare pages — Super's sidebar allows a page at the top level rather than a category, which FAQ uses, and treating every top-level entry as a category dropped it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The markdown in this repo predates the Notion/Super site and had drifted. Notion is authoritative, so content is now generated from it. docs/ holds 45 pages exported by scripts/notion_export.py, with 100 images in static/img/notion/. Every page keeps the URL the Super site served — the slug is written into frontmatter from scripts/slug_map.json — so all 45 indexed URLs survive the move and no redirects are needed. Verified: 45/45 expected doc URLs present in the build, none missing. docusaurus.config.js collapses the three content plugins into one docs plugin at the site root, and reproduces Super's navbar and footer (recovered into super-salvage/). sidebarsNotion.js reproduces Super's 12-section sidebar, which is hand-curated and differs from Notion's 4-section tree. Locales are reduced to English. The Super site serves English only — no locale switcher, no locale URLs in its sitemap — so the es/pt/tr translations cover content that is no longer published. They remain in git history. Three exporter fixes found by building: - Videos emit a plain iframe. The previous ReactPlayer output needed an import that generated markdown had no way to declare, which broke SSR. - Section pages link their children explicitly instead of using DocCardList, which only works on pages that are a sidebar category index. These are not, so it failed at render with an undefined read. - download() rejects empty responses. Two images had been written as zero-byte files, and the exists() check then made that failure permanent across runs. Known content issues, in Notion rather than the export: four links point nowhere resolvable, including a ./angelVault.md left over from an earlier import, which is the one broken link the build still reports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rebuilt site still looked like the old one because of two things. The navbar title was wrong. i18n/en/docusaurus-theme-classic/navbar.json was overriding it with the old "Giveth Documentation", along with stale footer and item labels. I had described the leftover directories as inert; they were not. The homepage had no sidebar. Super's sidebar was global chrome, present on every page including the home page, whereas Docusaurus only renders one on docs that appear in a sidebar file. The homepage is now docs/index.mdx served at /, with its cover, title and card grid moved into src/components/HomeCards. Because of the same rule, every doc has to be placed in the sidebar or it renders bare. The four Notion section pages are now the category link for the Super section they correspond to, which also makes those category headers clickable, and Causes — live at /donation-agents but never in Super's sidebar — is added as its own section. Also removed, now that nothing references them: dapps/, whatisgiveth/, devouch/, glossary/, downloads/, i18n/, notion-export/, the four old sidebar files, redirects.js, _niceTokenList.mdx and src/pages/jobs.md, which was still building a /jobs route the Super site does not have. Build is clean. The only remaining broken link is the ./angelVault.md left over in the Notion content itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Docusaurus work on this branch is a snapshot pipeline, not a renderer: new Notion pages land nowhere and edits need a script run plus a commit. The goal is a self-hosted app that renders Notion directly, so this plan hands the rebuild to a fresh session. Captures the findings that are expensive to re-derive: the Notion tree is built from inline databases rather than sub-pages, Super's slugs are hand-set and not derivable from titles, Super's stored page ids are all stale so matching must be by title, and the design tokens and fonts recovered from the live site. Records the decision to keep the 12-section sidebar hand-maintained, and marks which files are irreplaceable now that Super is being cancelled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs.giveth.io was served by super.so, a paid service rendering the Giveth Notion workspace. This replaces it with a Next.js app that renders the same Notion tree directly, so Notion stays the source of truth and nobody has to run a script or land a commit to publish an edit. Renderer is react-notion-x over Notion's API. Verified early that the docs tree is public and needs no credentials, and that the inline-database card grid -- the load-bearing block type on the homepage -- server-renders correctly, since the architecture depended on it. URLs are preserved exactly. Super's slugs were hand-set in its dashboard and are not derivable from titles, so scripts/slug_map.json stays the override table, keyed by title because the Notion page ids Super stored are all stale. Titles not in it fall through to slugify, so a page created in Notion gets a working URL with no code change. Navigation stays hand-maintained in config/navigation.ts, generated once from the salvaged Super config. Super's 12-section sidebar does not match Notion's 4-section tree and deriving nav from Notion would restructure the site. A page does not need a nav entry to be reachable. Design tokens, the mis-weighted TeX Gyre Adventor heading font and the icon set all come from super-salvage/, which cannot be regenerated once the subscription lapses. Edits land via ISR (5 min for pages, 15 for the tree) with an authenticated /api/revalidate for an immediate push. Images go through /api/image rather than hotlinking Notion. Search is a real index over page text, replacing the stale Algolia index which was built against the old markdown URLs. The Docusaurus site it replaces is removed. Verified: all 46 URLs return 200, every page carries sidebar, navbar and footer, and per-page titles are correct -- the two regressions called out in the handoff plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three separate causes, found by fetching every image on every page: Custom emoji page icons rendered broken. Notion stores them as `notion://custom_emoji/<space>/<id>`, a scheme neither notion-utils nor react-notion-x resolves and which Notion's image endpoint 404s on. They have to be looked up through syncRecordValues, which yields a plain public URL -- and which answers 403 unless the request carries a browser user agent. Affects the GIV token and DeVouch icons. Notion's image endpoint intermittently times out. The proxy turned any such blip into a 502, which the browser shows as a permanently broken image on an otherwise cached page. It now retries transient failures, gives up on 4xx since that is an answer rather than a hiccup, and as a last resort redirects to the upstream URL so a proxy failure degrades to hotlinking instead of a broken image. Images the authors embedded from outside Notion were being sent through the proxy and rejected by its host allowlist. They now go to the browser untouched: proxying arbitrary third-party URLs would make this route an open relay and buys nothing, since the point of the proxy is caching Notion's CDN. 122 of 124 images across all 46 pages now load. The two that do not are hotlinks to boostyourcampaign.com on /fundraisingguide, which that site now refuses to serve to anyone. That is a content fix for Notion, not a renderer bug; super-salvage/content-images/ has the copies the live site was showing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The site was rendering Notion at request time. That is why pages arrived with the chrome but no body: when loadPageChunk fails -- and it fails often, timing out for whole runs from some networks -- the layout still renders because it is local, while the content silently comes back empty. Notion answered in 1.3s one day and 9s the next during this work. Content is now fetched once by scripts/fetch-notion.mjs and committed: content/site-map.json every page, its title, breadcrumb and URL content/pages/<id>.json a page's record map plus its image map public/notion-assets/ every image and video public/search-index.json page text for search The site is a static export, which also matches how this repo already deploys: GitHub Pages off the gh-pages branch, not a Node server. Nothing in the build or the served page contacts Notion, so a slow Notion cannot produce a half-empty page. Assets are all local now, which fixes two things beyond reliability. Video, audio and file blocks bypass mapImageUrl entirely and read recordMap signed_urls, and those URLs expire about an hour after they are issued -- in a static build the videos would have broken shortly after every deploy. Fetching through the signed URL also fixed the four assets that were failing: they are videos, and Notion's image endpoint answers 422 for anything that is not an image. All 116 assets now download. Assets are hashed on the raw Notion URL rather than the signed one, so a re-run reuses what is already on disk instead of rewriting every filename. Record maps are pruned before writing: leftover signed URLs carry access tokens, and every byte is serialised into the HTML of the page. Refreshing is automatic. refresh-content.yml re-fetches every three hours, commits any change, and the push triggers deploy-docs.yml. It can also be run by hand or poked by a Notion automation via repository_dispatch. It refuses to publish a suspiciously small tree and leaves committed content alone if Notion cannot be read. scripts/check-export.mjs runs in CI and asserts what was actually broken: that every live URL exists and that pages carry real content rather than an empty shell. It allows for pages whose substance is not prose -- the homepage is a card grid, /brandbook is a sentence and a Figma embed. Search is now client-side over a static index, since there is no server to query. The ISR, revalidation and image-proxy routes are gone with the server. History note: the four Notion video files this commit refers to were removed from its tree before the branch was first pushed, to keep 102MB of video out of the repository. From "Point videos at the giveth-docs Space" onward they are hosted on the Space. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Videos made up 102MB of the 136MB of committed assets, and Notion's own links to them expire about an hour after they are issued, so they cannot simply be left pointing at Notion either. fetch-notion.mjs no longer downloads videos or other non-image files. It names each one from its Notion URL -- Notion keeps the original filename, extension included, so nothing has to be fetched to know it -- points the page at the file's URL on the Space, and records it in content/media.json. The upload is a CI step, not code in the site: scripts/sync-media.sh lists the Space folder once, downloads only what is missing and uploads it with the aws CLI, then re-lists to confirm. It runs in the same job as the fetch because the signed download URLs expire within the hour, which is also why that list is written to .cache/ and never committed. A failed upload stops the commit, so the site never references a file the Space does not have. Files are named by a hash of their Notion URL, so each is uploaded once and a re-run is a no-op. --verify checks every hosted file is publicly reachable without credentials; deploy runs it as a warning so a missing video does not hold back a text fix. Tested against a local stand-in for the Space with three of four videos already present: one download and upload, three skipped, the uploaded object a complete MP4 with the right content type and a public ACL; a second run did nothing; a run without credentials failed clearly. Not tested against a real Space -- config/media.json has no bucket yet. Until config/media.json names a bucket and region, fetch-notion stops with an error rather than publishing pages with broken videos. The committed content is unchanged by this commit and still serves the videos locally; the first refresh after the Space is configured moves them. History note: the four Notion video files this commit refers to were removed from its tree before the branch was first pushed, to keep 102MB of video out of the repository. From "Point videos at the giveth-docs Space" onward they are hosted on the Space. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
config/media.json now names the Space (giveth-docs, fra1, served from its origin endpoint), so the four Notion-hosted videos are referenced there and their local copies are gone from public/notion-assets. The files are not uploaded yet. That happens in CI, where the Space keys live: refresh-content runs the fetch and scripts/sync-media.sh in one job, and nothing it commits references a file the upload did not confirm. Until the first run after this lands on master, the four videos will not play. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.