A Framework for Diagnosing and Communicating Performance Declines

One of the challenges in running acquisition for a SaaS company is that declines in performance are inevitable.

You can't have a fire drill every time there is a slight drop in performance, but not addressing real issues soon enough can have a big impact on the business.

Knowing how to handle the declines when they happen is what will make your efforts sustainable in the long run.

How I handle declines

In my 8.5 years at Sprout Social, I must have endured at least 100 declines in top-of-funnel performance.

Sometimes they only lasted 1-2 days, other times they lasted weeks.

Over time, I developed a system to make a quick diagnosis and have a consistent way to provide updates to the exec team. I have since refined this process as well.

Step 1 - Understand the magnitude

Let's say you get 100 free trial signups per day at your company. If you have a day where you only get 95, is that normal or is that a problem?

First, let's look at the actual trial signup numbers for the last two weeks (ideally you look at a larger data set). From here, we calculate the standard deviation.

Now we can see that from this dataset, the standard deviation is 2.6.

Here is how to assess the magnitude:

• Less than one standard deviation is normal fluctuation and does not warrant any time to investigate
• One standard deviation is only an issue if sustained for multiple days
• Two standard deviations may happen for a single day as a fluke but 2+ days requires immediate attention

Step 2 - Diagnose the issue

Here is how I suggest you invest your time accordingly depending on the magnitude of the issue.

• Less than one standard deviation = none
• One standard deviation sustained for 3+ days = run through the quick diagnostics and see if anything stands out (about 15 min of your time)
• Two standard deviations or more for 2+ days = run through the diagnostics in greater detail (about 1 hour of your time) and communicate (more below)

Quick diagnostic framework

In your web analytics and CRM data, run several reports (ideally saved) to view performance across channels, regions/countries, browser types, devices, and customer segments.

The goal is to have a quick way to look across all of your major performance variables to identify one of three issues.

• Systemic - Across all channels, browser types, segments, etc. you are seeing a similar decline in performance. This typically points to a systemic issue with your site, signup process, or something similar because nothing else would make every one of those variables change the same way.
• Isolated - Probably the more obvious one but you identify a channel or specific device type that dropped significantly and pulled down the average.
• Seasonal - Major, global holidays are obvious but this quick process can also be a great way to identify that the entire drop was from the UK for a bank holiday or something and everything else is pretty much normal.

Step 3 - Communicate and act

Proactive communication and an appropriate amount of action are two of the key things to keep performance where it needs to be and maintain trust within the company.

Based on the stuff outlined above here is what I recommend for communication and action.

• Less than one standard deviation - Ideally your company doesn't get upset about this but if necessary, communicate that you are aware of it, it is within a normal range of fluctuation, and your team will keep an eye on it.
• One standard deviation for 3+ days - Communicate that this threshold was crossed and that you are diagnosing and will report back with findings. Then do so and determine the next steps from there.
• Two standard deviations for 2+ days - Communicate that this critical threshold was crossed and that you are diagnosing the issue. Then, schedule a meeting 3-4 days out with all necessary people to address the issue if it is still a problem at that point. If it is, dig in as a group and take action. If it isn't, cancel the meeting and report the findings.

Moving forward

I know this may feel a bit intense, but once a company is $20m+ in ARR this really does become necessary and most importantly, can prevent a lot of unnecessary fire drills, anxiousness, and uncertainty.

Even if this isn't a perfect fit for your company, consider taking some of the elements I laid out to create your own process that makes performance declines more tolerable and rational.