Feature | CMG-703 | stack-to-stack migration with audit report#1078
Feature | CMG-703 | stack-to-stack migration with audit report#1078chetan-contentstack wants to merge 9 commits into
Conversation
Adds Contentstack-to-Contentstack migration across regions/orgs: CLI-driven source export, export-structure validation, audit report (unused assets, unpublished entries, empty content types, unused global fields), and import via the official CLI with link-field normalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
… and environment variable usage
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
chetan-contentstack
requested review from
shradha-nahar,
umesh-more-cstk,
umeshmore45 and
vikrantraut-cstk
and removed request for
umeshmore45
| import AuthenticationModel from '../models/authentication.js'; | ||
| import { setBasicAuthConfig } from '../utils/config-handler.util.js'; | ||
|
|
||
| const runCommand = (command: string, args: string[] = []): Promise<void> => |
There was a problem hiding this comment.
No timeout or kill mechanism on long-running child process.
api/src/services/exportCli.service.ts, lines 8–19
spawn() runs npx @contentstack/cli cm:stacks:export with no timeout. For large stacks this could hang indefinitely, blocking the event loop response and holding a server resource with no recovery path.
Suggestion: Add a configurable timeout (e.g., AbortController with AbortSignal) or a setTimeout that calls cmd.kill() and rejects the promise with a clear timeout error.
| const entryFiles = await collectEntryFiles(entriesDir); | ||
| const entryAudit: any[] = []; | ||
| const contentTypeEntryCount: Record<string, number> = {}; | ||
| const globalFieldUsage = new Set<string>(); |
There was a problem hiding this comment.
audit.service.ts — Asset reference detection via payload.includes(asset.uid) is substring-only
File: api/src/services/audit.service.ts, approximately lines 100–108
Detecting asset references via JSON.stringify(entryData) + payload.includes(asset.uid) can produce false positives if an asset UID appears as a substring within another UID or string value. For example, if asset_123 is a substring of entry_uid_asset_1234, it would be marked as "referred."
Suggestion: Use a more precise regex search: new RegExp("${asset.uid}") or check for a boundary like word characters.
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
There was a problem hiding this comment.
Pull request overview
Implements Contentstack stack-to-stack migration support end-to-end (export → validate → audit → map → test → final), adding new backend services/routes, upload-api validators/mappers, and UI flow updates (including cross-region source login/session handling and improved long-running migration UX).
Changes:
- Adds Contentstack export validation + locale extraction + upload-api validator/mapper support.
- Adds API support for source export/validation/audit (with sharded lowdb audit storage) and CLI import safeguards (link normalization + per-module fallback).
- Updates UI migration flow (audit step, long timeouts, better log parsing, and regional source-login token caching).
Reviewed changes
Copilot reviewed 67 out of 69 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| upload-api/tests/unit/validators/contentstack.validator.test.ts | Unit tests for Contentstack zip/json validator. |
| upload-api/tests/unit/services/contentstack.service.test.ts | Unit tests for Contentstack mapper service POSTing locales. |
| upload-api/tests/unit/services/contentstack.locales.test.ts | Unit tests for resolving export root + extracting locales. |
| upload-api/tests/unit/config/index.config.test.ts | Updates config env var precedence + mysql default tests. |
| upload-api/src/validators/index.ts | Registers Contentstack validator for zip/json formats. |
| upload-api/src/validators/contentstack.ts | Adds Contentstack zip/json validation logic. |
| upload-api/src/services/createMapper.ts | Adds Contentstack mapper selection. |
| upload-api/src/services/contentstack/locales.ts | Implements export-root resolution + locale extraction for upload-api. |
| upload-api/src/services/contentstack/index.ts | Creates Contentstack mapper (posts extracted locales). |
| upload-api/src/routes/index.ts | Tightens file path handling + response typing; fixes comment typo. |
| upload-api/src/config/index.ts | Aligns local path env precedence + formatting. |
| upload-api/migration-aem/package-lock.json | Removes @types/uuid entry from lockfile. |
| ui/src/utilities/migrationSourceSession.ts | Session storage helpers for source-region token caching. |
| ui/src/utilities/functions.ts | Ensures failure notifications always have non-empty text. |
| ui/src/utilities/constants.ts | Adds constant for popup postMessage source. |
| ui/src/services/api/user.service.ts | Adds profile fetch for an override token (regional login). |
| ui/src/services/api/upload.service.ts | Hardens getCall error fallback when err.response is missing. |
| ui/src/services/api/stacks.service.ts | Adds optional source_region + app token override when listing stacks. |
| ui/src/services/api/migration.service.ts | Adds source-config update, audit/export endpoints, and long timeouts. |
| ui/src/pages/Migration/index.tsx | Adds audit step + source_details persistence + multiple step flow changes. |
| ui/src/pages/Login/index.tsx | Adds popup regional login flow + return-to handling + safer errors. |
| ui/src/context/app/app.interface.ts | Extends migration state shape with source_details + audit + stepValue. |
| ui/src/components/TestMigration/index.tsx | Improves disable logic + state persistence + completion handling. |
| ui/src/components/TestMigration/index.scss | Tweaks layout spacing for test migration UI. |
| ui/src/components/MigrationFlowHeader/index.tsx | Makes CTA label dependent on step/CMS; fixes step gating logic. |
| ui/src/components/LogScreen/MigrationLogViewer.tsx | Buffers socket log chunks + improves parsing + completion handling. |
| ui/src/components/LogScreen/index.tsx | Same log chunk buffering + parsing improvements for test migration logs. |
| ui/src/components/LegacyCms/legacyCms.scss | Styles new Contentstack source panel UI. |
| ui/src/components/LegacyCms/Actions/LoadSelectCms.tsx | Initializes source_details and sets source_mode for Contentstack. |
| ui/src/components/DestinationStack/Actions/LoadLanguageMapper.tsx | Supports locale arrays as strings or objects (Contentstack format). |
| ui/src/components/Common/DeleteProjectModal/index.tsx | Fixes React prop casing: tabIndex. |
| ui/src/components/AuditReport/index.tsx | Adds UI for running/reviewing audit and persisting exclusions/summary. |
| ui/src/components/AuditReport/index.scss | Styling for audit report UI/table. |
| ui/src/cmsData/migrationSteps.json | Adds Audit step and renumbers steps to 6-step flow. |
| api/tests/unit/utils/normalize-entry-links.utils.test.ts | Tests link normalization across schema shapes (group/blocks/multiple). |
| api/tests/unit/utils/contentstack-user-orgs.utils.test.ts | Tests org mapping across region-specific role payload shapes. |
| api/tests/unit/services/validation.service.test.ts | Tests export root resolution + structure validation. |
| api/tests/unit/services/user.service.test.ts | Ensures profile response includes region and mapped orgs. |
| api/tests/unit/services/projects.service.test.ts | Updates step-progression tests + adds source/audit update tests. |
| api/tests/unit/services/migration.service.test.ts | Adds tests for export/validate/audit endpoints and step/status updates. |
| api/tests/unit/services/exportCli.service.test.ts | Tests CLI export execution + region handling + error cases. |
| api/tests/unit/services/contentMapper.service.test.ts | Updates expected step gating for mapper operations. |
| api/tests/unit/services/auth.service.test.ts | Updates org membership requirements + aligns error message constant. |
| api/tests/unit/services/audit.service.test.ts | Tests audit generation paths and edge cases. |
| api/tests/unit/routes/projects.routes.test.ts | Adds route wiring tests for new project endpoints. |
| api/tests/unit/routes/migration.routes.test.ts | Adds route wiring tests for new migration endpoints. |
| api/tests/unit/models/audit-lowdb.test.ts | Tests sharded audit persistence + retrieval behavior. |
| api/tests/unit/controllers/migration.controller.test.ts | Updates controller behavior to await long-running service calls. |
| api/src/utils/sanitize-path.utils.ts | Adds allowlisted export-root path assertion helper. |
| api/src/utils/normalize-entry-links.utils.ts | Adds export-wide legacy link normalization utility. |
| api/src/utils/contentstack-user-orgs.utils.ts | Adds robust org mapping util across role shapes/regions. |
| api/src/services/validation.service.ts | Adds Contentstack export structure validation + root resolution. |
| api/src/services/user.service.ts | Uses org mapping util + includes region in profile payload. |
| api/src/services/runCli.service.ts | Adds link normalization + module fallback + export-root resolution. |
| api/src/services/projects.service.ts | Adds source config + audit selection updates; adds audit step gating. |
| api/src/services/exportCli.service.ts | Implements Contentstack stack export via CLI with structured logging. |
| api/src/services/auth.service.ts | Uses org mapping util for admin/owner/member determination. |
| api/src/services/audit.service.ts | Implements audit generation + persistence to sharded lowdb store. |
| api/src/server.ts | Fixes log tailing to avoid awaitWriteFinish delays; adds offset reset. |
| api/src/routes/projects.routes.ts | Adds project routes for source-config and audit-selections. |
| api/src/routes/migration.routes.ts | Adds migration routes for export/validate/audit. |
| api/src/models/project-lowdb.ts | Extends project schema to store source_details/audit/validation/source_locales. |
| api/src/models/audit-lowdb.ts | Adds sharded audit storage model. |
| api/src/controllers/projects.controller.ts | Adds controllers for source-config + audit-selections. |
| api/src/controllers/migration.controller.ts | Adds controllers for export/validate/audit; awaits long operations. |
| api/src/constants/index.ts | Adds CMS.CONTENTSTACK, audit DB config, and renumbers step constants. |
| .gitignore | Ignores export-stack/ output directory. |
Files not reviewed (1)
- upload-api/migration-aem/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const url = `/projects/${projectId}/migration/steps/5`; | ||
| navigate(url, { replace: true }); | ||
|
|
||
| await updateCurrentStepData(selectedOrganisation.value, projectId); | ||
| handleStepChange(3); | ||
| handleStepChange(4); | ||
| }} |
| handleStepChange(4); | ||
| const url = `/projects/${projectId}/migration/steps/5`; | ||
| navigate(url, { replace: true }); |
| handleStepChange(5); | ||
| const url = `/projects/${projectId}/migration/steps/6`; | ||
| navigate(url, { replace: true }); | ||
| //} |
| if ( | ||
| !project?.legacy_cms?.audit?.summary || | ||
| project.status === NEW_PROJECT_STATUS[0] || | ||
| !isStepCompleted || | ||
| !project?.destination_stack_id | ||
| ) { | ||
| const reason = !project?.legacy_cms?.audit?.summary | ||
| ? 'Audit summary is missing on the project (generate the audit on this step so it can be saved).' | ||
| : project.status === NEW_PROJECT_STATUS[0] | ||
| ? 'Project is still in draft status.' | ||
| : !isStepCompleted | ||
| ? 'Legacy CMS or file format is incomplete.' | ||
| : 'Destination stack is not set.'; | ||
| throw new BadRequestError( | ||
| `You cannot proceed from the audit step. ${reason}` | ||
| ); |
| if (projectIndex > -1 && !isTest) { | ||
| ProjectModelLowdb.data.projects[projectIndex].isMigrationCompleted = | ||
| true; | ||
| ProjectModelLowdb.data.projects[projectIndex].isMigrationStarted = | ||
| false; | ||
| ProjectModelLowdb.data.projects[projectIndex].current_step = 6; | ||
| ProjectModelLowdb.data.projects[projectIndex].status = 5; |
| } | ||
| } | ||
|
|
||
| const assetUidSet = new Set(assets.map((a) => a.uid).filter(Boolean)); |
| { | ||
| "flow_id": "auditReport", | ||
| "_metadata": { "uid": "csauditstep0001" }, | ||
| "name": 3, | ||
| "title": "Audit Report", | ||
| "description": "Review source stack audit before mapping and migration", | ||
| "group_name": "Audit" | ||
| }, | ||
| { | ||
| "flow_id": "contentMapping", | ||
| "_metadata": { "uid": "cs82c582021cc46a87" }, | ||
| "name": 3, | ||
| "name": 4, | ||
| "title": "Content Mapping", | ||
| "description": "Review of content mapping on the target stack", | ||
| "group_name": "Layout" | ||
| }, | ||
| { | ||
| "flow_id": "testMigration", | ||
| "name": 4, | ||
| "name": 5, | ||
| "title": "Test Migration", | ||
| "description": "Preview the migration process", | ||
| "group_name": "Layout" | ||
| }, | ||
| { | ||
| "flow_id": "migrationExecution", | ||
| "_metadata": { "uid": "csb100b7a598d9c333" }, | ||
| "name": 5, | ||
| "name": 6, | ||
| "title": "Migration Execution", | ||
| "description": "Wait for the execution for migration complete.", |
Summary
Implements CMG-703 — content migration from a source Contentstack stack to a destination stack across regions, organizations, or within the same org/region.
The flow: source-stack auth → CLI-driven export → export validation → audit report → destination config → content mapper → test migration → final migration.
Scope delivered
Source stack selection
LoadUploadFile.tsx).npx @contentstack/cli cm:stacks:exportvia the newexportCli.service.ts; exported data lands underapi/export-stack/<stackId>/.Export validation
validation.service.tsresolves the new branch-nested export layout (main/,master/, …) and reports missing modules (content_types,entries,assets,global_fields,locales) with a clean message.resolvedRooton the project so downstream steps use one canonical path.Audit report (new Step 3)
audit.service.ts+audit-lowdb.tsgenerate a per-project sharded audit covering:AuditReport/index.tsxlets users review and exclude items from the upcoming migration; exclusions are persisted on the project.GET /audit-data/:projectId.Destination configuration
getUserProfileWithToken) so users authenticate once per region and the resulting token is cached for reuse.contentstack-user-orgs.utils.ts.Test & final migration
runCli.service→cm:stacks:importpath with two new safeguards:normalize-entry-links.utils.tsrewrites legacy{ url, title }link values to{ href, title }(recursive throughgroup/blocks).MODULE_NOT_FOUND) trigger a per-module fallback so the rest of the import still succeeds.ERROR:lines from the CLI are treated as non-fatal warnings; only known fatal strings abort the run.Acceptance criteria mapping
audit-lowdb,/audit-data/:projectId)Known limitations (worth follow-up tickets)
source_branchonly; stacks with multiple branches need one project per branch. UI doesn't expose multi-branch selection yet.resolveContentstackExportRootis duplicated betweenapi/andupload-api/; consolidating requires a shared workspace package.