Skip to content

Attach exported highlights to their original Reader documents - #41

Merged
Endle merged 16 commits into
mainfrom
feat/reader-highlights-split
Oct 12, 2026
Merged

Endle merged 16 commits into
mainfrom
feat/reader-highlights-split

Conversation

@Endle

@Endle Endle commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Last of the PRs that replace #37. #38 (transport refactor), #39 (201 / POST-retry fixes) and #42 (skip local cleanup when export fails) are merged.

Highlights exported from downloaded Reader articles currently become separate entries through the Readwise v2 API. This change sends them to Reader's POST /api/v3/save/ with the original document's parent_id and selected passage, so they attach to that document. Other books keep the existing v2 export.

  • Export history: Reader highlight IDs and the notes sent with them are saved in the reader_highlights setting once per book, including when the book fails. Each passage is exported once: later syncs skip it, and notes added or edited after export are not sent. Repeated highlights of the same passage become one highlight carrying all their notes.

  • Rejected passages: when Reader answers 400 (e.g. the passage is not found in its copy of the article) or 404 (article deleted), the passage is sent through the v2 highlights API instead and recorded, so Reader is not asked again. The exception is a 400 that names notes or saved_using but not content or parent_id: that is a bug in the request, so it is an export error retried on the next sync instead of silently moving passages to v2. Export continues past a rejected passage, and rejected passages are sent even if a later passage fails. Error bodies from both APIs are logged (first 500 characters). The warning for a rejected passage names its page; the passage text is logged only at debug level, since users attach logs to public issues.

  • Highlights without text (e.g. image selections) are skipped and logged instead of failing the book.

  • Blank notes: a KOReader log on Linux showed Reader's reason for rejecting highlights: 400 {"notes": ["This field may not be blank."]}. The plugin sent notes: "" for every highlight without a note. Create now leaves notes out when it is empty or whitespace only. This probably explains most of the rejections in the Attach exported highlights to their original Reader documents #37 device test (3 of 4 passages).

Addresses #20, following the API update reported in this comment. API contract: Reader create documentation.

Validation: 31 LuaJIT tests pass with KOReader and HTTP stubs (routing, payloads, saved history across restarts, note edits not resent, repeated passages, partial failures, 400 classification, 400/404 fallback including fallback failure and later failures, rate-limit retries, malformed responses, error logging). Luacheck reports 0 warnings. Against a real account (KOReader on Linux), Reader accepted highlights both with and without a note after the blank-notes fix, and one with a note was checked inline in the article. Saved history with quotes, newlines, backslashes, NUL bytes and long passages round-trips through KOReader's own settings serializer (dump.lua).

Limits (the main ones are in the README): highlights exported by earlier versions are not migrated; the first sync after upgrading sends those still on the device to Reader, so they can appear twice in Readwise. Export tracking is local and keyed by document and exact passage. Notes added or edited after export are not sent, a highlight text change creates a new highlight, and local deletions do not propagate. Passages Reader rejects appear in Readwise but not inline in Reader. Resetting settings, losing a create response, or a crash in the middle of a book can allow a later export to be resent.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W8hfPEJmjGWSdSHeaku17w

@Endle
Endle force-pushed the fix/reader-request-safety branch from bbc2a8c to 6917f01 Compare September 28, 2026 03:00
Base automatically changed from fix/reader-request-safety to main October 1, 2026 02:13
@Endle
Endle force-pushed the feat/reader-highlights-split branch from ea03443 to 0e0f594 Compare October 1, 2026 02:30
@Endle
Endle changed the base branch from main to fix/stop-sync-on-export-failure October 1, 2026 02:45
@Endle
Endle force-pushed the feat/reader-highlights-split branch 2 times, most recently from 0c229ff to 64d9aa8 Compare October 3, 2026 20:13
@Endle
Endle force-pushed the fix/stop-sync-on-export-failure branch from 43a4276 to be55205 Compare October 3, 2026 20:27
@Endle
Endle force-pushed the feat/reader-highlights-split branch from 64d9aa8 to 3b8cf6a Compare October 3, 2026 20:29
Base automatically changed from fix/stop-sync-on-export-failure to main October 3, 2026 22:49
Endle and others added 2 commits October 3, 2026 18:50
Highlights in downloaded Reader articles are sent to Reader's
POST /api/v3/save/ with the article's parent_id, so they appear on the
original document. Other books keep the v2 highlights export.

Successful Reader exports are recorded per document and passage and
saved after each request. Repeated syncs skip unchanged passages, and
note edits or removal update the existing highlight via PATCH.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reader answers 400 when a passage does not match its copy of the
article, and may answer 404 for a deleted article. Neither succeeds on
retry, and with sync now keeping local files after a failed export, one
such passage would hold back cleanup on every sync.

Those passages are now sent through the v2 highlights API and recorded,
so Reader is not asked again; later note edits are resent through v2.
Export continues past a rejected passage, rejected passages are sent
even if a later one fails, and highlights without text are skipped
instead of failing the book. The transport returns Reader's error body
so a rejection is logged with the passage and Reader's explanation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Endle
Endle force-pushed the feat/reader-highlights-split branch from 3b8cf6a to c3eaedb Compare October 3, 2026 22:51
Endle and others added 11 commits October 3, 2026 19:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route rejected passages back through exportBook as a one-chapter book
without a document ID, instead of moving the v2 payload code into a new
function. The existing v2 code keeps its original form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhenbo Li <lizhenbo@yahoo.com>
Reader answers 400 "This field may not be blank" when notes is an empty
string, so every highlight without a note was rejected and sent through
the highlights API instead. Found in a KOReader log on Linux.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fallback used to reach the highlights API by calling exportBook again
with a metadata table that left out document_id. Passing the full metadata
through would have sent the rejected passages back to Reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reader trims notes before validating them, so a note of " " or "\n" got the
same 400 as an empty one and the passage was permanently sent to the
highlights API instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every 400 from /save/ was recorded as a passage Reader could not match, so a
validation error on a field the plugin chooses (like the blank-notes bug) sent
the passage to the highlights API permanently. A 400 that names only notes or
saved_using is now an export error that is retried on the next sync. Anything
else, including unrecognized bodies, still falls back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A note edit on a highlight deleted in Reader got 404 on every sync. That
failed the book, which blocked local cleanup and the export of its later
passages. The highlight is now recorded as deleted and left alone, rather
than recreated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reader history is keyed by document and passage text, so two highlights of the
same text shared one record. The second highlight's note overwrote the first,
and every later sync PATCHed the note back and forth. Repeats now become one
highlight whose note joins the distinct non-blank notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every created or updated highlight rewrote and flushed the whole settings file,
which holds the full text of every exported passage. On a large first export
that is quadratic I/O on the e-reader's flash. History is now saved once at the
end of each book, including when the book fails. The trade-off: a crash in the
middle of a book can resend that book's passages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
requestReader logged the whole error body at err level, which can be a
multi-KB HTML page from a proxy. It now logs the first 500 characters and
still returns the full body. Rejected passage text moves from warn to dbg,
because users attach these logs to public issues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Endle
Endle marked this pull request as draft October 4, 2026 03:03
Endle and others added 3 commits October 12, 2026 10:42
Notes edited after export were sent with PATCH /update/<id>/, which has
not been tried on a real account, and any failure there blocked local
cleanup on every sync. A passage that is already recorded is now skipped
whatever its note, including passages sent through the highlights API.
Records still keep the note that was sent, as a baseline for a future
note sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The highlights API path logged only the HTTP status line, so a failure
when sending a rejected passage there would reach a bug report without
the server's explanation. It now logs the first 500 characters of the
body, as requestReader does. A passage Reader rejects is never sent to
Reader again, so its warning now names the page, which identifies the
highlight without logging its text.

The highlights API failure test had been exercising the Reader path
since Reader articles started going to /save/; it now uses a non-Reader
book.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plugin has no record of highlights exported by earlier versions, so
the first sync after upgrading sends those in articles still on the
device to Reader too. Readwise then holds both the old unlinked copy and
the one attached in Reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Endle
Endle marked this pull request as ready for review October 12, 2026 19:18
@Endle
Endle merged commit ecde35c into main Oct 12, 2026
1 check passed
@Endle
Endle deleted the feat/reader-highlights-split branch October 12, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant