Skip to content

feat: freehand polygon drawing (press-and-drag) behind feature flag - #9820

Open
ozymandiashh wants to merge 3 commits into
HumanSignal:developfrom
ozymandiashh:feat/freehand-polygon-drawing
Open

feat: freehand polygon drawing (press-and-drag) behind feature flag#9820
ozymandiashh wants to merge 3 commits into
HumanSignal:developfrom
ozymandiashh:feat/freehand-polygon-drawing

Conversation

@ozymandiashh

@ozymandiashh ozymandiashh commented Jul 14, 2026

Copy link
Copy Markdown

Problem

The accepted community PRD in #8315 describes the current limitation:

Drawing polygons by clicking with the mouse for every point of the polygon is less optimal compared to simply being able to draw a polygon by holding down the mouse. Also the clicking technique does not work well with touchscreen devices or drawing pens, because the hovering feature is not easily available to these devices.

Solution

  • Add a pointer-events drawing path for primary mouse, pen, and touch input.
  • Simplify sampled points with a small in-house Ramer-Douglas-Peucker implementation.
  • Convert coordinates at sample time while keeping simplification in zoom-independent screen space.
  • Require a deliberate 5-pixel displacement before freehand semantics suppress compatibility mouse events.
  • Reuse the stock canvas target checks so existing regions, anchors, transformers, and read-only annotations keep their normal interactions.
  • Preserve the existing Cmd/Ctrl override for intentionally drawing over another region.
  • Scope touch-action suppression to the selected Polygon tool.
  • Commit each completed contour as one undoable history action.
  • Release the history transaction safely when completion is cancelled by a tool switch or teardown.
  • Guard the behavior with fflag_feat_front_polygon_freehand, default off.
  • Preserve the existing click-to-place Polygon workflow with the flag both off and on.
  • Add no runtime or development dependencies.

User Stories

  • As an annotator, I want to draw polygons by simply dragging the mouse, so that I can annotate faster.
  • As an annotator, I want to draw polygons via touchscreen or pen devices, so that I can annotate faster and more comfortably.

Acceptance Criteria

  1. Given fflag_feat_front_polygon_freehand is disabled, when an annotator uses the Polygon tool, then the existing click-to-place workflow is unchanged and a pointer drag does not create a contour.
  2. Given the flag is enabled and the Polygon tool is active, when an annotator presses, drags through at least three distinct points, and releases with a primary mouse pointer, then one simplified polygon is created.
  3. Repeat criterion 2 with a pen, then with primary touch; each input creates exactly one polygon.
  4. Given the flag is either enabled or disabled, click four vertices and close the polygon; exactly one stock click-to-place polygon is created.
  5. Given a pointer moves less than 5 pixels before release, the gesture continues through the stock click path and creates no freehand contour.
  6. If the stage transform changes during a drag, each committed vertex retains the coordinate transform active when that sample was collected.
  7. After any completed freehand contour, one Undo action removes the whole contour and one Redo action restores it.
  8. If the active pointer is cancelled, leaves the stage in a capture-less environment, the tool changes, or the component unmounts, then the in-progress trace is discarded without leaving drawing or history state stuck.
  9. A freehand trace produces no additional network request and requires no new package.

Testing

  • yarn lsf:unit --runInBand --silent: 129 test suites passed; 3,331 tests passed and 5 were skipped.
  • yarn lsf:unit --runInBand --silent --testPathPatterns='components/ImageView/__tests__/ImageView.test.jsx|tools/__tests__/Polygon.test.js|utils/__tests__/freehand.test.js': 3 suites and 85 tests passed.
  • BABEL_ENV=test MODE=standalone yarn nx run editor:build:production --skip-nx-cache: production build compiled successfully.
  • Added Cypress coverage for flag-off drag behavior, click-to-place coexistence with the flag off and on, mouse, pen, touch, compatibility mouse events, and single-undo behavior.
  • The integration spec passes locally on macOS with Cypress 14.5.0 and Electron against the production standalone build: all 7 tests passing.
  • Regression coverage verifies that freehand does not start from existing regions, transformer anchors, or read-only annotations, while the Cmd/Ctrl override remains available.

Credit and context

Thank you to @cozeybozey for the PRD and proof of concept in #8315 and the earlier exploration in #8388. This implementation is intentionally scoped to the generic freehand Polygon workflow and uses pointer events plus an in-house simplifier without the extra dependency or unrelated changes from the prior attempt.

The interaction model has been production-tested with a medical imaging annotation team using tablets and styluses.

Closes-PRD: #8315

Acceptance criteria:
- When the flag is off, Polygon retains its existing click-to-place behavior and pointer drags create nothing.
- When the flag is on, primary mouse, pen, and touch drags create one simplified polygon while ordinary clicks still place vertices.
- Each sample keeps the coordinate transform active when it was collected, including during mid-drag zoom changes.
- One undo action removes the entire completed contour.
- Pointer cancellation, tool switches, teardown, and capture-less fallback leave no in-progress drawing or frozen history.
- Unit and integration coverage exercises simplification, flag gating, input types, click coexistence, transform changes, and undo.

Closes-PRD: HumanSignal#8315
References: HumanSignal#8315
@netlify

netlify Bot commented Jul 14, 2026

Copy link
Copy Markdown

👷 Deploy request for heartex-docs pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 589b10f

@netlify

netlify Bot commented Jul 14, 2026

Copy link
Copy Markdown

👷 Deploy request for label-studio-docs-new-theme pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 589b10f

@netlify

netlify Bot commented Jul 14, 2026

Copy link
Copy Markdown

Deploy Preview for label-studio-playground ready!

Name Link
🔨 Latest commit 589b10f
🔍 Latest deploy log https://app.netlify.com/projects/label-studio-playground/deploys/6a582c076c7b560008520719
😎 Deploy Preview https://deploy-preview-9820--label-studio-playground.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@netlify

netlify Bot commented Jul 14, 2026

Copy link
Copy Markdown

Deploy Preview for label-studio-storybook ready!

Name Link
🔨 Latest commit 589b10f
🔍 Latest deploy log https://app.netlify.com/projects/label-studio-storybook/deploys/6a582c0729881600077af5f3
😎 Deploy Preview https://deploy-preview-9820--label-studio-storybook.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c4e480cbff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread web/libs/editor/src/components/ImageView/ImageView.jsx
@ozymandiashh

Copy link
Copy Markdown
Author

Quick demo from real production use on dental panoramic radiographs. Apologies for my shaky hand while drawing the lines, but it shows how easily and quickly a contour can be repaired, and the precision you can get with press-and-drag compared to placing points one by one.

freehand contour repair demo

The compatibility guard that suppresses the synthetic mouse event following a
touch/pen freehand interaction was only time-checked, never cleared. If its
synthetic event never arrived (e.g. the browser cancels the click after a
sub-threshold cursor move between mousedown and mouseup), the stale guard could
suppress one later, unrelated mouse event with nearby coordinates.

Prune the guard as soon as it is found expired, extend the TTL from 500ms to
1500ms to comfortably cover the synthetic-event window, and route the timestamp
through an injectable clock so the behaviour is unit-testable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant