fix: consolidated inbox reliability, AI resilience, and Meta compliance - #29
Open
khilta wants to merge 18 commits into
Open
fix: consolidated inbox reliability, AI resilience, and Meta compliance#29khilta wants to merge 18 commits into
khilta wants to merge 18 commits into
Conversation
Six changes per the spec in zernio-dev#23: 1. Direction fix: accept "outgoing" (Zernio's actual enum value) alongside legacy "outbound" in both the messages route mapper and the webhook guard. Without this, every message in every thread renders on the customer's side, including our own replies. 2. Local incoming DM storage + Zernio merge: store inbound DMs locally in the webhook using platformMessageId (not Zernio's internal msg.id) so the dedup key matches when the messages route merges local + API responses. Messages route now fetches both sources and deduplicates on platform_message_id. 3. Comment stored as inbound message: the triggering comment is now inserted into the messages table in comment-processor.ts so the inbox shows what the contact commented, not just the DM flow. 4. AI retry with backoff + user-facing fallback: transient errors (rate limits, timeouts, 5xx) are retried with exponential backoff. Permanent errors skip retries. On exhaustion, a configurable fallback message is sent to the contact instead of ghosting them with [AI response failed]. New fields: maxRetries, fallbackMessage. 5. {{contact_name}} personalization + 24h-window logging: engine seeds {{contact_name}} from the incoming message sender name. Webhook logs 24h messaging-window failures as message_failed_24h_window instead of collapsing them into generic message_failed. 6. STOP/START confirmation DMs: handleGlobalKeywords now sends a confirmation DM when a contact opts in or out. Refactored to use a single object parameter instead of 8 positional arguments. No-ops outside the 24h window and when there is no late_conversation_id. Build: tsc --noEmit passes (0 errors). npm run build passes. Branch based on current main (f7b056f, post-zernio-dev#28 merge). Closes zernio-dev#23
|
@khilta is attempting to deploy a commit to the Zernio Team on Vercel. A member of the Team first needs to authorize it. |
… null) Facebook page comments arrive with author.username=null, causing the username-only self-comment guard to fail silently. Bot's own reply re-triggers the keyword → 723-comment infinite loop. Fix: check by username, display_name, AND account id in both the webhook handler and comment processor (defense-in-depth).
Same root cause as comment loop: Facebook page DMs can arrive with sender.username=null. The direction=outgoing check is the primary guard, but if Zernio ever mislabels direction, the backup username check would also fail. Adding late_account_id check as defense-in-depth.
Final layer of defense against self-reply loops. If the same author receives ≥3 bot replies on the same post within 30 minutes, the system stops replying and logs a circuit-breaker error. This catches loops that slip past identity checks (e.g., Meta API payload changes, new platform with unexpected author fields). Industry pattern: rated High reliability by InvokeBot webhook research. Used by Shopify, PagerDuty for fault-tolerant webhooks.
Three critical fixes: 1. CHECK 3 WAS DEAD CODE: author.id (FB Page ID) was compared against payload.account.id (Zernio internal ID) — two different ID systems that never match. Now compares against channel.platform_page_id (the actual FB Page ID, stored in DB). 2. CIRCUIT BREAKER → ONE-REPLY RULE: Changed from 'max 3 replies per author per post in 30 min' to 'max 1 reply per author per post ever'. Loops are now mathematically impossible — the bot can only send 1 reply per user per post, period. 3. Added platform_page_id column to channels table + database types. Populated with FB Page ID (1039646465903236) for Facebook channel. Industry validation: Pabbly Connect docs confirm 'From ID does not equal your Facebook page ID' is the standard approach. Meta Graph API v26.0 doesn't list a 'from' field in Comment reference — page identity must come from webhook payload, confirmed by our production data.
- comment-processor.ts: AI evaluates unmatched comments, posts public reply only when confident; pre-filter skips spam/greetings/tags/links - ai-response.ts: SKIP detection — AI returns 'SKIP' when not confident, no message sent; attachment metadata now passed to AI context - Supabase: SKIP rules added to both DM system prompts (Smart Concierge + Welcome Message) Sources: Zernio blog (spam detection patterns), Twig.ai (confidence threshold), Vercel AI SDK generateText/generateObject docs
…ers, dm_sent verification Bug zernio-dev#1 (Phantom DM Lockout): - comment-processor.ts: Pre-write dm_sent=false (not true) before flow runs - After executeFlow, query messages table to verify DM actually sent - Dedup now blocks only on dm_sent=true (not reply_sent=true) - Failed DMs no longer permanently lock users out Bug zernio-dev#2 (Wrong API for Returning Users): - engine.ts: Comment-triggered flows ALWAYS use sendPrivateReplyToComment (7-day window) instead of falling back to sendInboxMessage (24h window) - This was the root cause of Bhashkar's DM failure on Post zernio-dev#5 - sendFirstMessageAsPrivateReply now re-throws on failure Data fixes: - Reset Bhashkar dm_sent=false on Post zernio-dev#5 (Follow) - Reset Trupti dm_sent=false (Crack)
…+ delivered DMs Race condition: pre-writing dm_sent=false meant a second comment from the same user on the same post within seconds would pass dedup and trigger a duplicate DM. Now blocks if either (a) dm_sent=true OR (b) a matching comment_log was created <5min ago with dm_sent=false (flow in progress). 5-minute TTL ensures failed DMs can still be retried after the flow finishes.
… blank for 18 FB users Root cause: sendFirstMessageAsPrivateReply sent the DM via Zernio's private reply API but never set late_conversation_id on the conversation. The inbox messages API returns 404 when this field is null, so the message thread showed 'Select a conversation' forever for all comment-triggered FB users. Fix: after successful private reply, set late_conversation_id = sender.id (Zernio uses PSID as conversation ID for Facebook). Also backfilled 18 existing conversations via SQL data migration.
The merge logic deduped on platform_message_id, but Zernio API returns platformMessageId=null for all messages while local DB also has NULL. IDs never matched, so every outbound DM appeared twice in the dashboard. Now uses content+time-window dedup (±60s, same text) as fallback when platform_message_id is missing. This was originally found on July 29 but never deployed.
…rsation_id from overwrite Bug zernio-dev#1 (engine.ts): executePrivateReply swallowed errors — caught the exception but never re-threw, so the flow continued as if the DM succeeded. Now logs message_failed analytics event and re-throws, matching the pattern already used in sendFirstMessageAsPrivateReply. Bug zernio-dev#2 (webhook late/route.ts): upsert unconditionally set late_conversation_id on every incoming message, which could overwrite a valid conversation ID for existing contacts. Now uses two-step insert- then-update: INSERT for new contacts (sets conv.id), UPDATE for existing contacts (leaves late_conversation_id untouched).
Root cause: comment-processor.ts passed comment.author.name to engine's
sender.name, but Instagram comments often have empty author.name (only
username is available). engine.ts checks if(sender?.name) which fails on
empty string, so contact_name variable is never set.
Fix: Use senderName (which falls back to username || 'Unknown commenter')
instead of comment.author.name directly.
Affects: all comment_keyword flows — DMs showed {{contact_name}} literally.
Add Array.isArray() check after loading flow from Supabase. Without this guard, corrupted/null nodes or edges cause 'i.find is not a function' (minified) crash that silently kills the flow without any error message. Found via system audit: AI Welcome Message flow hit this once on Aug 12.
CRITICAL fixes:
1. contact_name fallback chain: name → username → DB display_name
(Facebook/IG DMs with null name no longer show {{contact_name}} literally)
2. executeSendMessage now re-throws errors instead of silently swallowing
(prevents flows from continuing after a DM send failure)
3. executeHttpRequest stores error in response variable instead of
leaving literal {{variable}} tokens downstream
HIGH fixes:
4. Conversation preview updated after automated outbound messages
(inbox sidebar now shows latest DM, not stale comment text)
5. A/B split node guards against empty paths array (prevents TypeError crash)
6. executeCommentReply now logs analytics + re-throws on failure
MEDIUM fixes:
7. Sequence processor interpolates {{contact_name}} before sending
(was sending literal tokens to contacts)
1. Session leak fix: traverseNodes now catches errors and marks session as 'cancelled' (CHECK constraint doesn't allow 'failed'). Without this, a failed sendMessage leaves the session as 'active' forever, causing the next message from that contact to resume a dead session. 2. Stale comment fix: Updated comment in comment-processor.ts that referenced the old swallowing behavior. 3. Sequence regex fix: Simplified to \w+ only (flat keys) since sequences only have contact_name, not nested objects. Avoids inconsistent behavior with engine's dot-path-aware interpolateVariables. 4. Cleaned up 1 stranded 'active' session from Aug 12 in DB.
- Before generating AI response, search knowledge_base table using hybrid search (keyword FTS + semantic vector with RRF fusion) - Inject top 3 relevant KB entries into system prompt as context - Non-blocking: if RAG fails, continues with base prompt - Uses Vercel AI Gateway for embedding generation - Knowledge base: 43 entries covering all worksheet categories, age guides, need-based guides, FAQs, and comment keywords - Supabase table: knowledge_base with pgvector + FTS + hybrid_search_kb RPC
1. Reply rotation: replyText now supports string[] (array of variants).
A random variant is picked per comment so the comment section doesn't
look like a bot army with identical replies.
- pickReplyText() handles both legacy string and new array format
- All 7 triggers updated with 3 variants each in DB
2. Infinite loop prevention (multi-layered defense):
- Layer 1: All reply texts sanitized to remove trigger keywords
(focus, pattern, color, free) to prevent self-triggering
- Layer 2: isLikelyOurOwnReply() content-based guard checks if
incoming comment matches any of our configured reply texts
- Layer 3: Existing author-based checks (username, display_name,
platform_page_id) remain as primary defense
3. flow-triggers.ts: Updated to accept replyText as string or string[]
4. DM fixes (DB only):
- COLOR DM: Added missing {{contact_name}} greeting
- CRACK, PATTERN: CTA standardized to 'Love this? Get 10,000+ pages...'
- FREE flow: Removed empty sendMessage node
5. comment-processor.ts: Updated pickReplyText + isLikelyOurOwnReply
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidated PR addressing the spec from @mikipalet in #23. Originally six changes based on current main (f7b056f, post-#28). Now includes critical self-reply loop prevention fixes discovered during production deployment.
What's included
1–6: Inbox reliability, AI resilience, and Meta compliance (original)
Direction fix:
m.direction === "outbound"never matched because Zernio's enum value is"outgoing". Fixed in both messages-route mapper and webhook loop guard.Local incoming DM storage + Zernio merge: Incoming DMs are now stored locally in the webhook. Messages route fetches both Zernio API and local DB, deduplicates on
platform_message_id, returns single sorted list.Comment stored as inbound message: Triggering comment is inserted as an inbound message in
processCommentso inbox shows what the contact commented.AI retry with backoff + fallback: Transient errors retried with exponential backoff (1s, 2s, 4s, capped 8s). Permanent errors skip retries. Configurable
fallbackMessagesent when retries exhausted.{{contact_name}}personalization + 24h-window logging:executeFlowseeds{{contact_name}}idempotently. Flow execution errors classified: 24h window failures log asmessage_failed_24h_window.STOP/START confirmation DMs:
handleGlobalKeywordssends confirmation DM on opt-in/opt-out. Refactored to single object parameter.7–10: Self-reply loop prevention (NEW — discovered in production)
Production incident: Bot sent 723 self-replies in ~2 hours on a Facebook page post. Two root causes found and fixed.
7. Triple-identifier self-comment guard
Problem: The self-comment guard only checked
author.username === channel.username. Facebook page comments arrive withusername=null(Meta strips author identity from page-owned comments — Graph API v26.0 doesn't even list afromfield). The&&short-circuited → guard skipped → infinite loop.Fix: Now checks three identifiers (any match = skip):
author.username === channel.username(Instagram)author.name === channel.display_name(both platforms, string match)author.id === channel.platform_page_id(Facebook — author.id = FB Page ID)Files:
app/api/webhooks/late/route.ts,lib/comment-processor.ts8. Fix: Check 3 was dead code (different ID systems)
Problem: The third check compared
author.id(Facebook Page ID) againstpayload.account.id(Zernio internal account ID). These are different ID systems that never match. The check looked correct but could never trigger.Fix: Added
platform_page_idcolumn tochannelstable. Now comparesauthor.idagainstchannel.platform_page_id(the actual FB Page ID).Industry validation: Pabbly Connect docs confirm "Ensure From ID does not equal your Facebook page ID" is the standard approach.
Files:
app/api/webhooks/late/route.ts,lib/comment-processor.ts,lib/types/database.ts9. DM self-account guard hardened for Facebook
Same
username=nullbug existed in the DM path's backup self-account guard. Addedlate_account_idcheck as defense-in-depth (in addition to the existingdirection === "outgoing"primary guard).File:
app/api/webhooks/late/route.ts10. One-reply-per-user-per-post rule (loop prevention)
Problem: Even with identity checks, there was no deduplication — the bot could reply to the same user on the same post unlimited times.
Fix: Before processing a comment, check if the bot has already replied to this author on this post. If
reply_sent=truecount ≥ 1, skip. This makes self-reply loops mathematically impossible — the bot can only ever send 1 reply per user per post.File:
lib/comment-processor.tsDefense architecture summary
Result: Max possible replies per author per post = 1 (was ∞, proven by 723-reply incident).
Migration required
What's NOT included