-
Notifications
You must be signed in to change notification settings - Fork 849
Replaced Pylon with SupportHog #18160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
ba39e69
9ca46ca
72f1868
882d80c
274667c
4b4930d
31000d8
cfe5808
8884463
58be99c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -34,7 +34,7 @@ Reach out before customers ask, even if it’s just to say, “We’re aware and | |||||
|
|
||||||
| | Level | Description | Examples | Channels | Cadence | | ||||||
| | ----- | ----- | ----- | ----- | ----- | | ||||||
| | **SEV 0 – Emergency** | Existential service failure; all or most customers impacted with no workaround. CMOC sends immediately via broadcast. Account owners do **not** gate comms. | Full platform outage, data loss, security breach with active customer impact. | Pylon broadcast → Email → DM/SMS | Immediate broadcast, then every 15–30 min; postmortem within 24h | | ||||||
| | **SEV 0 – Emergency** | Existential service failure; all or most customers impacted with no workaround. CMOC sends immediately via broadcast. Account owners do **not** gate comms. | Full platform outage, data loss, security breach with active customer impact. | Slack message → Email → DM/SMS | Immediate broadcast, then every 15–30 min; postmortem within 24h | | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| | **SEV 1 – Critical** | Major outage or data loss; widespread impact. | API unavailable, ingestion halted, login failures. | Slack → Email → (DM or SMS if needed) | Every 30–60 min; postmortem within 48h | | ||||||
| | **SEV 2 – Major** | Partial degradation or downtime; workaround available. | Replay or query delays \>30 min, flag evaluation slow. | Slack or Email | Every 1–2 hrs | | ||||||
| | **SEV 3 – Minor** | Limited impact or slow recovery. | Billing sync delays, isolated org issues. | Slack | Start and close | | ||||||
|
|
@@ -44,9 +44,9 @@ Reach out before customers ask, even if it’s just to say, “We’re aware and | |||||
|
|
||||||
| ### **Emergency (SEV 0)** | ||||||
|
|
||||||
| > **This overrides the standard workflow.** CMOC sends directly to all affected customer channels via Pylon broadcast without waiting for account owners. Account owners follow up individually once online. | ||||||
| > **This overrides the standard workflow.** CMOC sends directly to all affected Slack customer channels without waiting for account owners. Account owners follow up individually once online. | ||||||
|
|
||||||
| **Initial broadcast (Pylon):** | ||||||
| **Initial message (Slack):** | ||||||
|
|
||||||
| We're investigating a major incident affecting [feature/service]. [Symptom — e.g., "Event ingestion is fully stopped" or "The PostHog app is unreachable."] | ||||||
|
|
||||||
|
|
@@ -143,7 +143,7 @@ Heads up — maintenance on \[system\] from \[time window\]. No downtime expecte | |||||
|
|
||||||
| ### **Workflow** | ||||||
|
|
||||||
| > **SEV 0 override:** For SEV 0 incidents, the CMOC skips steps 4–5 and sends the initial message directly via [Pylon broadcast](#using-pylon-for-broadcasts) to all affected customer channels. Account owners are notified in [#group-cs-sales-support](https://posthog.slack.com/archives/C090RCG671C) simultaneously, and take over individual follow-up threads once online. The CMOC continues to own broadcast updates until the incident is resolved or downgraded. | ||||||
| > **SEV 0 override:** For SEV 0 incidents, the CMOC skips steps 4–5 and sends the initial message directly via Slack to all affected customer channels. Account owners are notified in [#group-cs-sales-support](https://posthog.slack.com/archives/C090RCG671C) simultaneously, and take over individual follow-up threads once online. The CMOC continues to own broadcast updates until the incident is resolved or downgraded. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| 1. **Incident declared** (Engineering). | ||||||
| 2. **CMOC activated**, notified of impact. | ||||||
| 3. **Assess customer impact**, this <PrivateLink url="https://us.posthog.com/project/2/insights/EBiXOD91">insight</PrivateLink> (or this <PrivateLink url="https://docs.google.com/spreadsheets/d/1EyV55L0vWTfD4W02j5A3Vx3YZYPUODmQQC1toFPcKXU/edit?gid=1396499197#gid=1396499197">Google Sheet</PrivateLink> as a backup) will help you understand which customers are using which components in which cloud environment. | ||||||
|
|
@@ -163,20 +163,3 @@ Heads up — maintenance on \[system\] from \[time window\]. No downtime expecte | |||||
| 4. **Account Owner sends the message** to their customers. Example outbound: “We’re investigating an outage affecting event ingestion. Updates every 30 minutes.” | ||||||
| 5. **During:** “Root cause identified (Redis queue saturation). Fix in progress.” | ||||||
| 6. **Resolution:** “Resolved at 11:42 UTC. Write-up soon.” | ||||||
|
|
||||||
| ## Using Pylon for broadcasts | ||||||
|
|
||||||
| It's best that communications are shared directly from the account owner; however, if speed is of the essence, i.e. for a SEV 1 or security issue, and some folks are not yet working or on PTO, the CMOC can use Pylon to send a broadcast to all customer Slack channels en masse: | ||||||
|
|
||||||
| 1. Log into app.usepylon.com with your PostHog Slack account. | ||||||
| 2. Go to the _Broadcasts_ link on the left-hand side of the navigation. | ||||||
| 3. Click _Create Broadcast_ in the top right-hand corner of the UI. | ||||||
| 4. Enter the message you want to send, ensuring the formatting looks correct in the preview on the right-hand side. | ||||||
| 5. Ensure the Send as option is set to PostHog, not your own user (unless you want to handle 450+ potential separate threads) | ||||||
| 6. Click _Next_ in the top right-hand corner of the UI. | ||||||
| 7. Select your audience. You can use the filters to select all channels not owned by specific people, e.g., those who are currently online and communicating 1:1 with customers. | ||||||
| 8. Make sure you click _Add to Audience_ to add the selected channels to the broadcast. | ||||||
| 9. Click _Next_ in the top right-hand corner of the UI. | ||||||
| 10. Set the engagement notification channel to be [#support-customer-success](https://posthog.slack.com/archives/C05MUMZLC13) | ||||||
| 11. Check that you're happy with the message and audience and click _Send Now_ in the top right of the UI. | ||||||
| 12. Ask everyone online to monitor [#support-customer-success](https://posthog.slack.com/archives/C05MUMZLC13) for replies and respond where necessary. | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,7 +15,7 @@ As a general principle, we try to make sure as much customer communication as po | |
|
|
||
| For existing customers, you'll sometimes send emails directly from <PrivateLink url="https://posthog.vitally-eu.io/">Vitally</PrivateLink>. To ensure these also make it to Salesforce, first look up your _Email to Salesforce Address_ from the [personal settings page](https://posthog.lightning.force.com/lightning/settings/personal/EmailToSalesforceUserSetup/home) in Salesforce, and then add it to your <PrivateLink url="https://posthog.vitally-eu.io/settings/profile/gmail">Vitally gmail settings</PrivateLink>. | ||
|
|
||
| All Slack messages sync up with the corresponding account in Salesforce. We use [Pylon](https://app.usepylon.com) for this sync, so make sure Pylon is added to the customer Slack channel integrations and the channel is [linked to the Salesforce account](https://app.usepylon.com/integrations/salesforce?tab=account-mapping) properly for the sync to work smoothly. | ||
| All Slack messages sync up with the corresponding account in Salesforce. We use [SupportHog](/handbook/growth/sales/slack-channels) for this sync, so make sure SupportHog is [added to the customer Slack channel](/handbook/growth/sales/slack-channels) and the channel's name is recorded on the relevant Salesforce account record for the sync to work smoothly. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think this is how this works.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @tjcran can you help with how Slack <> Vitally/Salesforce/PostHog sync works at the moment?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @abhischekt can assist here too
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hello! I'm already working on this in: PostHog/posthog#66974 in this current PR I'm focused on the aggregate stats that pylon sent to SFDC per: https://github.com/PostHog/posthog/pull/66974/changes#diff-ea9e087a5e10f2ff92bf65e3842a9098565c38709f69346877978128319fe5c9R32 Once this is shipped and running I'll figure out how to get the actual |
||
|
|
||
| You are most likely to use the following regularly: | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't know how specific we want to be -
It's not that SupportHog links to Salesforce. I actually think that Abe was looking at syncing all conversations.
Maybe worth mentioning here that you can go to conversations and view tickets from your accounts by filtering on your own name tag (will be ae_emaillocalpart or csm_emailloalpart)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Going to leave this for now as I assume this will all change with the move to customer analytics anyway