Skip to content

fix: consolidated inbox reliability, AI resilience, and Meta compliance - #29

Open
khilta wants to merge 18 commits into
zernio-dev:mainfrom
khilta:contribute/consolidated-fixes
Open

fix: consolidated inbox reliability, AI resilience, and Meta compliance#29
khilta wants to merge 18 commits into
zernio-dev:mainfrom
khilta:contribute/consolidated-fixes

Conversation

@khilta

@khilta khilta commented Aug 10, 2026

Copy link
Copy Markdown

Summary

Consolidated PR addressing the spec from @mikipalet in #23. Originally six changes based on current main (f7b056f, post-#28). Now includes critical self-reply loop prevention fixes discovered during production deployment.

What's included

1–6: Inbox reliability, AI resilience, and Meta compliance (original)

Direction fix: m.direction === "outbound" never matched because Zernio's enum value is "outgoing". Fixed in both messages-route mapper and webhook loop guard.

Local incoming DM storage + Zernio merge: Incoming DMs are now stored locally in the webhook. Messages route fetches both Zernio API and local DB, deduplicates on platform_message_id, returns single sorted list.

Comment stored as inbound message: Triggering comment is inserted as an inbound message in processComment so inbox shows what the contact commented.

AI retry with backoff + fallback: Transient errors retried with exponential backoff (1s, 2s, 4s, capped 8s). Permanent errors skip retries. Configurable fallbackMessage sent when retries exhausted.

{{contact_name}} personalization + 24h-window logging: executeFlow seeds {{contact_name}} idempotently. Flow execution errors classified: 24h window failures log as message_failed_24h_window.

STOP/START confirmation DMs: handleGlobalKeywords sends confirmation DM on opt-in/opt-out. Refactored to single object parameter.


7–10: Self-reply loop prevention (NEW — discovered in production)

Production incident: Bot sent 723 self-replies in ~2 hours on a Facebook page post. Two root causes found and fixed.

7. Triple-identifier self-comment guard

Problem: The self-comment guard only checked author.username === channel.username. Facebook page comments arrive with username=null (Meta strips author identity from page-owned comments — Graph API v26.0 doesn't even list a from field). The && short-circuited → guard skipped → infinite loop.

Fix: Now checks three identifiers (any match = skip):

  • author.username === channel.username (Instagram)
  • author.name === channel.display_name (both platforms, string match)
  • author.id === channel.platform_page_id (Facebook — author.id = FB Page ID)

Files: app/api/webhooks/late/route.ts, lib/comment-processor.ts

8. Fix: Check 3 was dead code (different ID systems)

Problem: The third check compared author.id (Facebook Page ID) against payload.account.id (Zernio internal account ID). These are different ID systems that never match. The check looked correct but could never trigger.

Fix: Added platform_page_id column to channels table. Now compares author.id against channel.platform_page_id (the actual FB Page ID).

Industry validation: Pabbly Connect docs confirm "Ensure From ID does not equal your Facebook page ID" is the standard approach.

Files: app/api/webhooks/late/route.ts, lib/comment-processor.ts, lib/types/database.ts

9. DM self-account guard hardened for Facebook

Same username=null bug existed in the DM path's backup self-account guard. Added late_account_id check as defense-in-depth (in addition to the existing direction === "outgoing" primary guard).

File: app/api/webhooks/late/route.ts

10. One-reply-per-user-per-post rule (loop prevention)

Problem: Even with identity checks, there was no deduplication — the bot could reply to the same user on the same post unlimited times.

Fix: Before processing a comment, check if the bot has already replied to this author on this post. If reply_sent=true count ≥ 1, skip. This makes self-reply loops mathematically impossible — the bot can only ever send 1 reply per user per post.

if (comment.author.id) {
  const { count } = await supabase
    .from("comment_logs")
    .select("*", { count: "exact", head: true })
    .eq("channel_id", channel.id)
    .eq("post_id", comment.postId)
    .eq("author_id", comment.author.id)
    .eq("reply_sent", true);
  if ((count ?? 0) >= 1) return { matched: false, skipped: "rate_limited" };
}

File: lib/comment-processor.ts

Defense architecture summary

Layer What Catches
1. Webhook guard Triple identity check (username + name + page_id) Bot's own comments at ingestion
2. Processor guard Same triple check, repeated Defense-in-depth if webhook is bypassed
3. Reply text rule Keyword not in reply text Text-level loop trigger
4. One-reply rule Already replied to this author on this post? Everything else — mathematical guarantee

Result: Max possible replies per author per post = 1 (was ∞, proven by 723-reply incident).

Migration required

ALTER TABLE channels ADD COLUMN IF NOT EXISTS platform_page_id text;

What's NOT included

  • 5-second concurrent-flow guard (silently drops legitimate follow-ups, doesn't close the race)
  • Debounce / natural-delay (holds serverless function open, needs scheduled-jobs approach)

Six changes per the spec in zernio-dev#23:

1. Direction fix: accept "outgoing" (Zernio's actual enum value)
   alongside legacy "outbound" in both the messages route mapper and
   the webhook guard. Without this, every message in every thread
   renders on the customer's side, including our own replies.

2. Local incoming DM storage + Zernio merge: store inbound DMs locally
   in the webhook using platformMessageId (not Zernio's internal msg.id)
   so the dedup key matches when the messages route merges local + API
   responses. Messages route now fetches both sources and deduplicates
   on platform_message_id.

3. Comment stored as inbound message: the triggering comment is now
   inserted into the messages table in comment-processor.ts so the
   inbox shows what the contact commented, not just the DM flow.

4. AI retry with backoff + user-facing fallback: transient errors
   (rate limits, timeouts, 5xx) are retried with exponential backoff.
   Permanent errors skip retries. On exhaustion, a configurable
   fallback message is sent to the contact instead of ghosting them
   with [AI response failed]. New fields: maxRetries, fallbackMessage.

5. {{contact_name}} personalization + 24h-window logging: engine seeds
   {{contact_name}} from the incoming message sender name. Webhook logs
   24h messaging-window failures as message_failed_24h_window instead
   of collapsing them into generic message_failed.

6. STOP/START confirmation DMs: handleGlobalKeywords now sends a
   confirmation DM when a contact opts in or out. Refactored to use a
   single object parameter instead of 8 positional arguments. No-ops
   outside the 24h window and when there is no late_conversation_id.

Build: tsc --noEmit passes (0 errors). npm run build passes.

Branch based on current main (f7b056f, post-zernio-dev#28 merge).

Closes zernio-dev#23
@vercel

vercel Bot commented Aug 10, 2026

Copy link
Copy Markdown

@khilta is attempting to deploy a commit to the Zernio Team on Vercel.

A member of the Team first needs to authorize it.

khilta added 17 commits August 11, 2026 14:11
… null)

Facebook page comments arrive with author.username=null, causing the
username-only self-comment guard to fail silently. Bot's own reply
re-triggers the keyword → 723-comment infinite loop.

Fix: check by username, display_name, AND account id in both the webhook
handler and comment processor (defense-in-depth).
Same root cause as comment loop: Facebook page DMs can arrive with
sender.username=null. The direction=outgoing check is the primary
guard, but if Zernio ever mislabels direction, the backup username
check would also fail. Adding late_account_id check as defense-in-depth.
Final layer of defense against self-reply loops. If the same author
receives ≥3 bot replies on the same post within 30 minutes, the system
stops replying and logs a circuit-breaker error.

This catches loops that slip past identity checks (e.g., Meta API
payload changes, new platform with unexpected author fields).

Industry pattern: rated High reliability by InvokeBot webhook research.
Used by Shopify, PagerDuty for fault-tolerant webhooks.
Three critical fixes:

1. CHECK 3 WAS DEAD CODE: author.id (FB Page ID) was compared against
   payload.account.id (Zernio internal ID) — two different ID systems
   that never match. Now compares against channel.platform_page_id
   (the actual FB Page ID, stored in DB).

2. CIRCUIT BREAKER → ONE-REPLY RULE: Changed from 'max 3 replies per
   author per post in 30 min' to 'max 1 reply per author per post ever'.
   Loops are now mathematically impossible — the bot can only send 1
   reply per user per post, period.

3. Added platform_page_id column to channels table + database types.
   Populated with FB Page ID (1039646465903236) for Facebook channel.

Industry validation: Pabbly Connect docs confirm 'From ID does not equal
your Facebook page ID' is the standard approach. Meta Graph API v26.0
doesn't list a 'from' field in Comment reference — page identity must
come from webhook payload, confirmed by our production data.
- comment-processor.ts: AI evaluates unmatched comments, posts public
  reply only when confident; pre-filter skips spam/greetings/tags/links
- ai-response.ts: SKIP detection — AI returns 'SKIP' when not confident,
  no message sent; attachment metadata now passed to AI context
- Supabase: SKIP rules added to both DM system prompts (Smart Concierge
  + Welcome Message)

Sources: Zernio blog (spam detection patterns), Twig.ai (confidence
threshold), Vercel AI SDK generateText/generateObject docs
…ers, dm_sent verification

Bug zernio-dev#1 (Phantom DM Lockout):
- comment-processor.ts: Pre-write dm_sent=false (not true) before flow runs
- After executeFlow, query messages table to verify DM actually sent
- Dedup now blocks only on dm_sent=true (not reply_sent=true)
- Failed DMs no longer permanently lock users out

Bug zernio-dev#2 (Wrong API for Returning Users):
- engine.ts: Comment-triggered flows ALWAYS use sendPrivateReplyToComment
  (7-day window) instead of falling back to sendInboxMessage (24h window)
- This was the root cause of Bhashkar's DM failure on Post zernio-dev#5
- sendFirstMessageAsPrivateReply now re-throws on failure

Data fixes:
- Reset Bhashkar dm_sent=false on Post zernio-dev#5 (Follow)
- Reset Trupti dm_sent=false (Crack)
…+ delivered DMs

Race condition: pre-writing dm_sent=false meant a second comment from the
same user on the same post within seconds would pass dedup and trigger a
duplicate DM. Now blocks if either (a) dm_sent=true OR (b) a matching
comment_log was created <5min ago with dm_sent=false (flow in progress).
5-minute TTL ensures failed DMs can still be retried after the flow finishes.
… blank for 18 FB users

Root cause: sendFirstMessageAsPrivateReply sent the DM via Zernio's private
reply API but never set late_conversation_id on the conversation. The inbox
messages API returns 404 when this field is null, so the message thread
showed 'Select a conversation' forever for all comment-triggered FB users.

Fix: after successful private reply, set late_conversation_id = sender.id
(Zernio uses PSID as conversation ID for Facebook).

Also backfilled 18 existing conversations via SQL data migration.
The merge logic deduped on platform_message_id, but Zernio API returns
platformMessageId=null for all messages while local DB also has NULL.
IDs never matched, so every outbound DM appeared twice in the dashboard.

Now uses content+time-window dedup (±60s, same text) as fallback when
platform_message_id is missing. This was originally found on July 29
but never deployed.
…rsation_id from overwrite

Bug zernio-dev#1 (engine.ts): executePrivateReply swallowed errors — caught the
exception but never re-threw, so the flow continued as if the DM succeeded.
Now logs message_failed analytics event and re-throws, matching the pattern
already used in sendFirstMessageAsPrivateReply.

Bug zernio-dev#2 (webhook late/route.ts): upsert unconditionally set
late_conversation_id on every incoming message, which could overwrite a
valid conversation ID for existing contacts. Now uses two-step insert-
then-update: INSERT for new contacts (sets conv.id), UPDATE for existing
contacts (leaves late_conversation_id untouched).
Root cause: comment-processor.ts passed comment.author.name to engine's
sender.name, but Instagram comments often have empty author.name (only
username is available). engine.ts checks if(sender?.name) which fails on
empty string, so contact_name variable is never set.

Fix: Use senderName (which falls back to username || 'Unknown commenter')
instead of comment.author.name directly.

Affects: all comment_keyword flows — DMs showed {{contact_name}} literally.
Add Array.isArray() check after loading flow from Supabase. Without this
guard, corrupted/null nodes or edges cause 'i.find is not a function'
(minified) crash that silently kills the flow without any error message.

Found via system audit: AI Welcome Message flow hit this once on Aug 12.
CRITICAL fixes:
1. contact_name fallback chain: name → username → DB display_name
   (Facebook/IG DMs with null name no longer show {{contact_name}} literally)
2. executeSendMessage now re-throws errors instead of silently swallowing
   (prevents flows from continuing after a DM send failure)
3. executeHttpRequest stores error in response variable instead of
   leaving literal {{variable}} tokens downstream

HIGH fixes:
4. Conversation preview updated after automated outbound messages
   (inbox sidebar now shows latest DM, not stale comment text)
5. A/B split node guards against empty paths array (prevents TypeError crash)
6. executeCommentReply now logs analytics + re-throws on failure

MEDIUM fixes:
7. Sequence processor interpolates {{contact_name}} before sending
   (was sending literal tokens to contacts)
1. Session leak fix: traverseNodes now catches errors and marks session
   as 'cancelled' (CHECK constraint doesn't allow 'failed'). Without this,
   a failed sendMessage leaves the session as 'active' forever, causing
   the next message from that contact to resume a dead session.

2. Stale comment fix: Updated comment in comment-processor.ts that
   referenced the old swallowing behavior.

3. Sequence regex fix: Simplified to \w+ only (flat keys) since sequences
   only have contact_name, not nested objects. Avoids inconsistent behavior
   with engine's dot-path-aware interpolateVariables.

4. Cleaned up 1 stranded 'active' session from Aug 12 in DB.
- Before generating AI response, search knowledge_base table using
  hybrid search (keyword FTS + semantic vector with RRF fusion)
- Inject top 3 relevant KB entries into system prompt as context
- Non-blocking: if RAG fails, continues with base prompt
- Uses Vercel AI Gateway for embedding generation
- Knowledge base: 43 entries covering all worksheet categories,
  age guides, need-based guides, FAQs, and comment keywords
- Supabase table: knowledge_base with pgvector + FTS + hybrid_search_kb RPC
1. Reply rotation: replyText now supports string[] (array of variants).
   A random variant is picked per comment so the comment section doesn't
   look like a bot army with identical replies.
   - pickReplyText() handles both legacy string and new array format
   - All 7 triggers updated with 3 variants each in DB

2. Infinite loop prevention (multi-layered defense):
   - Layer 1: All reply texts sanitized to remove trigger keywords
     (focus, pattern, color, free) to prevent self-triggering
   - Layer 2: isLikelyOurOwnReply() content-based guard checks if
     incoming comment matches any of our configured reply texts
   - Layer 3: Existing author-based checks (username, display_name,
     platform_page_id) remain as primary defense

3. flow-triggers.ts: Updated to accept replyText as string or string[]

4. DM fixes (DB only):
   - COLOR DM: Added missing {{contact_name}} greeting
   - CRACK, PATTERN: CTA standardized to 'Love this? Get 10,000+ pages...'
   - FREE flow: Removed empty sendMessage node

5. comment-processor.ts: Updated pickReplyText + isLikelyOurOwnReply
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant