Epic: Deprecate Confidential Data Infra and Private Chat

[think-in-universe]

Goal

Deprecate NEAR AI Cloud's confidential-data infrastructure for Private Chat/stateful APIs, so Cloud can focus on private inference without continuing to accept or serve new Conversation/File state that depends on CVM-only storage guarantees.

The end state is:

• No new confidential Conversation/File data is written through the stateful API surfaces.
• Existing raw Conversation data is protected before the database is moved out of CVM.
• Stateful Responses, Conversations, Files, and related Private Chat APIs are retired.
• Private Chat is shut down after users have had a read/export window.

Scope

Components involved:

• Database: keep tracking in nearai/cloud-api.
• cloud-api: restrict stateful APIs in Stage I, then remove them.
• chat-api: update proxy/API behavior to match cloud-api deprecation and removal.
• private-chat frontend: disable write UX, add export/deprecation UX, then shut down the product surface.

Requirements

• During migration, do not expose raw confidential content to admins, operators, logs, or normal database access.
• Before moving the database out of CVM, encrypt existing raw Conversation content using a CVM-derived key from S3_ENCRYPTION_KEY or S3_ENCRYPTION_KEY_FILE.
• After Stage I, no supported user flow should create or update Conversations, Files, shares, pins, archives, copies, or other Private Chat state. The existing Cloud resource DELETE endpoints are the explicit exception: they remain available so Chat's established account-deletion worker can complete cleanup.
• Read access is temporary and exists only to support export/migration until Stage III.
• For Responses API compatibility, retain only stateless/no-store behavior.

Stage I: Restrict confidential-data writes

Target date: Sep 14

Database

Current state: the database runs inside CVM so raw confidential data is not exposed through normal administrative access. The migration path is to move the database out of CVM only after raw Conversation content has been encrypted.

Action items:

• Add an admin API in cloud-api that encrypts Conversation data with the CVM-derived key from S3_ENCRYPTION_KEY or S3_ENCRYPTION_KEY_FILE.
• Run the operational backfill to replace raw conversations table content with encrypted content through that API.

cloud-api

Restrict stateful APIs to temporary retrieval views plus the existing resource DELETE compatibility exception.

Action items:

• Deprecate stateful Responses API behavior and keep only stateless Responses API behavior.
• Retain only temporary Conversation/File retrieval access needed by migration/export tooling.
• Retain DELETE /v1/conversations/{conversation_id} and DELETE /v1/files/{file_id} so the existing Chat account-deletion worker can complete its normal Cloud cleanup.
• These DELETE operations remain the existing API-key/workspace-scoped endpoints; Stage I does not introduce an account-deletion-only token or internal path. They are therefore an intentional, limited public mutation exception.
• Disallow all other Conversation/File writes and conversation-related mutations such as copy, pin, archive, and similar operations.

chat-api

Update proxy endpoints to match the cloud-api restricted/deprecated behavior while preserving the existing account-deletion workflow.

Action items:

• Deprecate stateful Responses proxy behavior and keep only stateless Responses behavior.
• Disallow normal Conversations/File writes in proxy endpoints; keep only temporary read access.
• Disallow other conversation-related mutations such as share, copy, pin, archive, and similar operations.
• Preserve DELETE /v1/users/me and its established asynchronous worker, which uses Cloud's retained resource DELETE endpoints.

private-chat frontend

Deprecate the Private Chat product surface while preserving a read/export path.

Action items:

• Disable creating new conversations, submitting new prompts, copying conversations, uploading files, and any other non-deletion write action against Conversations or Files.
• Add conversation export in the Private Chat settings dialog.
• Show a deprecation banner that Private Chat will be shut down before Stage III.

Stage II: Migrate database from CVM

Target date: Sep 9

Stage II intentionally remains lightweight for now. The initial tracker should cover moving the database out of CVM after Stage I encryption and write-disablement are complete. More detailed component/action issues can be added once the migration plan is finalized.

Stage III: Remove stateful APIs and shut down Private Chat

Target date: Nov 1

cloud-api

Remove stateful API surfaces after the migration/export window and account-deletion lifecycle are complete.

Action items:

• Remove Conversation APIs.
• Remove Files APIs.
• Remove other conversation-related APIs, including the retained resource DELETE endpoints once account deletion is retired or replaced by the final cleanup process.

chat-api

Remove proxy endpoints for retired stateful API surfaces.

Action items:

• Remove Conversation API proxy endpoints.
• Remove Files API proxy endpoints.
• Remove other conversation-related proxy endpoints.

private-chat frontend

Shut down the Private Chat frontend.

Action item:

• Remove Private Chat frontend serving in chat-api.

Tracking

This epic is organized with GitHub sub-issues:

• Stage-level issues are linked directly under this epic.
• Component issues are linked under each stage.
• Component action items are linked under their component issue.
• Existing issues should be reused and linked rather than duplicated when they already cover the same work.

[think-in-universe]

Set up the sub-issue hierarchy for the three-stage deprecation plan.

Stage issues

• [Epic 953][Stage I] Restrict writes to confidential data #954 - Stage I: Disallow writes to confidential data
• [Epic 953][Stage II] Migrate database from CVM #955 - Stage II: Migrate database from CVM
• [Epic 953][Stage III] Remove stateful APIs and shut down Private Chat #956 - Stage III: Remove stateful APIs and shut down Private Chat

Stage I component trackers

• Database: [Epic 953][Stage I][Database] Encrypt conversation data before DB migration #957
  • [Epic 953][Stage I][Database] Add admin API to encrypt confidential database data #959 - Add admin API to encrypt conversation data
  • [Epic 953][Stage I][Database][Operation] Backfill encrypted conversation content #960 - Backfill encrypted conversation content
• cloud-api: [Epic 953][Stage I][cloud-api] Restrict stateful APIs and retain stateless Responses #934 (existing)
  • Existing action sub-issues already linked there include [Epic 953][Stage I][cloud-api] Make Responses API stateless (no-store only) #936, [Epic 953][Stage I][cloud-api] Retain Files views and deletion for account cleanup #937, [Epic 953][Stage I][cloud-api] Retain Conversation views and deletion for account cleanup #939, [Epic 953][Stage I][cloud-api] Remove dormant Responses history wiring #940, plus related cleanup/test issues.
• chat-api: [Epic 953][Stage I][chat-api] Retain established read views and selected DELETE routes; block writes chat-api#385
  • [Epic 953][Stage I][chat-api] Make the Responses proxy stateless chat-api#386 - Deprecate stateful Responses proxy
  • [Epic 953][Stage I][chat-api] Retain Conversation views and DELETE; block other writes chat-api#387 - Disallow Conversation API writes in proxy
  • [Epic 953][Stage I][chat-api] Retain File views and DELETE; block other writes chat-api#388 - Disallow Files API writes in proxy
  • [Epic 953][Stage I][chat-api] Retain sharing reads and selected DELETE routes; block writes chat-api#389 - Disallow share/copy/pin/archive conversation writes
• private-chat: [Epic 953][Stage I][private-chat] Disable writes and add export/deprecation UX private-chat#354
  • [Epic 953][Stage I][private-chat] Disable conversation and file write actions private-chat#355 - Disable conversation and file write actions
  • [Epic 953][Stage I][private-chat] Add conversation export in settings private-chat#356 - Add conversation export in settings
  • [Epic 953][Stage I][private-chat] Show Private Chat deprecation notice banner private-chat#357 - Show Private Chat deprecation notice banner

Stage II component tracker

• Database: [Epic 953][Stage II][Database] Migrate database from CVM #961

Stage III component trackers

• cloud-api: [Epic 953][Stage III][cloud-api] Remove stateful APIs #962
  • [Epic 953][Stage III][cloud-api] Remove Conversation APIs #963 - Remove Conversation APIs
  • [Epic 953][Stage III][cloud-api] Remove Files APIs and S3 startup configuration #964 - Remove Files APIs
  • [Epic 953][Stage III][cloud-api] Remove other conversation-related APIs #965 - Remove other conversation-related APIs
• chat-api: [Epic 953][Stage III][chat-api] Remove stateful proxy surfaces and runtime wiring chat-api#377 (existing)
  • Existing child issues already linked: [Superseded][Epic 953][Stage III][chat-api] Legacy stateful-surface retirement umbrella chat-api#379, [Epic 953][Stage I][chat-api] Preserve account deletion across Cloud migration chat-api#380, [Epic 953][Stage III][chat-api] Remove stateful runtime wiring after the read window chat-api#381
  • Stage-specific action issues added: [Epic 953][Stage III][chat-api] Remove Conversation API proxy endpoints chat-api#390, [Epic 953][Stage III][chat-api] Remove Files API proxy endpoints chat-api#391, [Epic 953][Stage III][chat-api] Remove other conversation-related proxy endpoints chat-api#392
• private-chat: [Epic 953][Stage III][private-chat] Shut down Private Chat frontend private-chat#358
  • [Epic 953][Stage III][chat-api] Remove Private Chat frontend serving chat-api#393 - Remove Private Chat frontend serving

Notes:

• I reused the existing cloud-api tracker [Epic 953][Stage I][cloud-api] Restrict stateful APIs and retain stateless Responses #934 instead of the duplicate [Epic 953][Stage I][cloud-api] Make stateful APIs read-only #958, which is already closed as duplicate.
• I reused the existing chat-api tracker [Epic 953][Stage III][chat-api] Remove stateful proxy surfaces and runtime wiring chat-api#377 for Stage III instead of creating another component tracker.

[hanakannzashi]

Hierarchy clarification: #380 is a Stage I chat-api action under #385, not a Stage III child of #377. The canonical Stage I chain is #953 → #954 → #385 → #380/#386–#389; #382 implements #386–#389. The canonical Stage III chain is #953 → #956 → #377 → #381/#390–#392. The existing account-deletion workflow remains active in Stage I, with Cloud #943 retaining the two resource DELETE routes.