Posted by Dan Peskin
This post was originally in YouMoz, and was promoted to the main blog because it provides great value and interest to our community. The author's views are entirely his or her own and may not reflect the views of SEOmoz, Inc.
Great Scott! I am finally back again for another spectacularly lengthy post, rich with wonderful titles, and this time - statistical goodness. It just so happens, that in my past short-lived career, I was a Forecast Analyst (not this kind). So today class, we will be learning about the importance of forecasting organic traffic and how you can get started. Let's begin our journey.
Forecasting is Your Density. I Mean, Your Destiny
Why should I forecast? Besides the obvious answer - it’s f-ing cool to predict the future, there are a number of benefits for both you and your company.
Forecasting adds value in both an agency and in-house setting. It provides a more accurate way to set goals and plan for the future, which can be applied to client projects, internal projects, or overall team/dept. strategy.
Forecasting creates accountability for your team. It allows you to continually set goals based on projections and monitor performance through forecast accuracy (Keep in mind that exceeding goals is not necessarily a good thing, which is why forecast accuracy is important. We will discuss this more later).
Forecasting teaches you about inefficiencies in your team, process, and strategy. The more you segment your forecast, the deeper you can dive into finding the root of the inaccuracies in your projections. And the more granular you get, the more accurate your forecast, so you will see that segmentation is a function of accuracy (assuming you continually work to improve it).
Forecasting is money. This is the most important concept of forecasting, and probably the point in where you decided that you will read the rest of this article.
The fact that you can improve inefficiencies in your process and strategy through forecasting, means you can effectively increase ROI. Every hour and resource allocated to a strategy that doesn’t deliver results can be reallocated to something that proves to be a more stable source of increased organic traffic. So finding out what strategies consistently deliver the results you expect, means you’re investing money into resources that have a higher probability of delivering you a larger ROI.
Furthermore, providing accurate projections, whether it’s to a CFO, manager, or client, gives the reviewer a more compelling reason to invest in the work that backs the forecast. Basically, if you want a bigger budget to work with, forecast the potential outcome of that bigger budget and sell it. Sell it well.
Okay. Flux Capacitor, Fluxing. Forecast, Forecasting?
I am going to make the assumption that everyone’s DeLorean is in the shop, so how do we forecast our organic traffic?
There are four main factors to account for in an organic traffic forecast: historical trends, growth, seasonality, and events. Historical data is always the best place to start and create your forecast. You will want to have as many historical data points as possible, but the accuracy of the data should come first.
Determining the Accuracy of the Data
Once you have your historical data set, start analyzing it for outliers. An outlier to a forecast is what Biff is to George McFly, something you need to punch in the face and then make wash your car 20 years in the future. Well something like that.
The quick way to find outliers is to simply graph your data and look for spikes in the graph. Each spike is associated with a data point, which is your outlier, whether it spikes up or down. This way does leave room for error, as the determination of outliers is based on your judgement and not statistical significance.
The long way is much more fun and requires a bit of math. I'll provide some formula refreshers along the way.
Calculating the mean and the standard deviation of your historical data is the first step.
Mean
Standard Deviation
Looking at the standard deviation can immediately tell you whether you have outliers or not. The standard deviation tells you how close your data falls near the average or mean, so the lower the standard deviation, the closer the data points are to each other.
You can go a step further and set a rule by calculating the coefficient of variation (COV). As a general rule, if your COV is less than 1, the variance in your data is low and there is a good probability that you don’t need to adjust any data points.
Coefficient of Variation (COV)
If all the signs point to you having significant outliers, you will now need to determine which data points those are. A simple way to do this is calculate how many standard deviations away from the mean your data point is.
Unfortunately, there is no clear cut rule to qualify an outlier with deviations from the mean. This is due to the fact that every data set is distributed differently. However, I would suggest starting with any data point that is more than one deviation from the mean.
Making your decision about whether outliers exist takes time and practice. These general rules of thumb can help you figure it out, but it really relies on your ability to interpret the data and be able to understand how each data point affects your forecast. You have the inside knowledge about your website, your equations and graphs don’t. So put that to use and start making your adjustments to your data accordingly.
Adjusting Outliers
Ask yourself one question: Should we account for this spike? Having spikes or outliers is normal, whether you need to do anything about it is what you should be asking yourself now. You want to use that inside knowledge of yours to determine why the spike occurred, whether it will happen again, and ultimately whether it should accounted for in your future forecast.
In the case that you don’t want to account for an outlier, you will need to accurately adjust it down or up to the number it would have been without the event that caused the anomaly.
For example, let’s say you launched a super original infographic about the Olympics in July last year that brought your site an additional 2,000 visits that month. You may not want to account for this as it will not be a recurring event or maybe it fails to bring qualified organic traffic to the site (if the infographic traffic doesn’t convert, then your revenue forecast will be inaccurate). So the resulting action would be to adjust the July data point down 2,000 visits.
On the flipside, what if your retail electronics website has a huge positive spike in November due to Black Friday? You should expect that rise in traffic to continue this November and account for it in your forecast. The resulting action here is to simply leave the outlier alone and let the forecast do it’s business (This is also an example of seasonality which I will talk about more later).
Base Forecast
When creating your forecast, you want to create a base for it before you start incorporating additional factors into it. The base forecast is usually a flat forecast or a line straight down the middle of your charted data. In terms of numbers, this can be simply be using the mean for every data point. The line down the middle of the data follows the trend of the graph, so this would be the equivalent of the average but accounting for slope too. Excel provides a formula which actually does this for you:
=FORECAST(x, known_y's,known_x's)
Given the historical data, excel will output a forecast based on that data and the slope from the starting point to end point. Dependent on your data, your base forecast could be where you stop, or where you begin developing an accurate forecast.
Now how do you improve your forecast? It’s a simple idea - account for anything and everything the data might not be able to account for. Now you don’t need to go overboard here. I would draw the line well before you start forecasting the decrease in productivity on Fridays due to beer o clock. I suggest accounting for three key factors and accounting for them well; growth, seasonality, and events.
Growth
You have to have growth. If you aren’t planning to grow anytime soon, then this is going to be a really depressing forecast. Including growth can be as simple as adding 5% month over month, due to a higher level estimate from management, or as detailed as estimating incremental search traffic by keyword from significant ranking increases. Either way, the important part is being able to back your estimates with good data and know where to look for it. With organic traffic, growth can come from a number of sources but these are a couple key components to consider:
Are you launching new products?
New products means new pages, and dependent on your domain's authority and your internal linking structure, you can see an influx of organic traffic. If you have analyzed the performance of newly launched pages, you should be able to estimate on average what percentage of search traffic from relevant and target keywords they can bring over time.
Using Google Webmaster Tools CTR data and the Adwords Tool for search volume are your best bet to acquire the data you need to estimate this. You can then apply this estimate to search volumes for the keywords that are relevant to each new product page and determine the additional growth in organic traffic that new product lines will bring.
Tip: Make sure to consider your link building strategies when analyzing past product page data. If you built links to these pages over the analyzed time period, then you should plan on doing the same for the new product pages.
What ongoing SEO efforts are increasing?
Did you get a link building budget increase? Are you retargeting several key pages on your website? These things can easily be factored in, as long as you have consistent data to back it up. Consistency in strategy is truly an asset, especially in the SEO world. With the frequency of algorithm updates, people tend to shift strategies fairly quickly. However, if you are consistent, you can quantify the results of your strategy and use it improve your strategy and understand its effects on the applied domain.
The general idea here is that if you know historically the effect of certain actions on a domain, then you can predict how relative changes to the domain will affect the future (given there are no drastic algorithm updates).
Let's take a simple example. Let's say you build 10 links to a domain per month and the average Page Authority is 30 and Domain Authority is 50 for the targeted pages and domain when you started. Over time you see as a result, your organic traffic increase by 20% for the pages you targeted on this campaign. So if your budget increases and allows you to apply the same campaign to other pages on the website, you can estimate an increase in organic traffic of 20% to those pages.
This example assumes the new target pages have:
Target keywords with similar search volumes
Similar authority at prior to the campaign start
Similar existing traffic and ranking metrics
Similar competition
While this may be a lot to assume, this is for the purpose of the example. However, these are things that will need to be considered and these are the types of campaigns that should be invested in from a SEO standpoint. When you find a strategy that works, repeat it and control the factors as much as possible. This will provide for an outcome that is the least likely to diverge from expected results.
Seasonality
To incorporate seasonality into a organic traffic forecast, you will need to create seasonal indices for each month of the year. A seasonal index is an index of how that month's expected value relates to the average expected value. So in this case, it would be how each month's organic traffic compares with average or mean monthly organic traffic.
So let's say your average organic traffic is 100,000 visitors per month and your adjusted traffic for last November was 150,000 visitors, then your index for November is 1.5. In your forecast you simply multiply by this weight for the corresponding index month.
To calculate these seasonal indices, you need data of course. Using adjusted historical data is the best solution, if you know that it reflects the seasonality of the website's traffic well.
Remember all that seasonal search volume data the Adwords tool provides? That can actually be put to practical use! So if you haven't already, you should probably get with the times and download the Adwords API excel plugin from SEOgadget (if you have API access). This can make gathering seasonal data for a large set of keywords quick and easy.
What you can do here, is gather data for all the keywords that drive your organic traffic, aggregate it, and see if the trends in search align with the seasonality you are observing in your adjusted historical data. If there is a major discrepancy between the two, you may need to dig deeper into why or shy away from accounting for it in your forecast.
Events
This one should be straightforward. If you have big events coming up, find a way to estimate their impact on your organic traffic. Events can be anything from a yearly sale, to a big piece of content being pushed out, or a planned feature on a big media site.
All you have to do here is determine the expected increase in traffic from each event you have planned. This all goes back to digging into your historical data. What typically happens when you have a sale? What's the change in traffic when you launch a huge content piece? If you can get an estimate of this, just add it to the corresponding month when the event will take place.
Once you have this covered, you should have the last piece to a good looking forecast. Now it's time to put it to the test.
Forecast Accuracy
So you have looked into your crystal ball and finally made your predictions, but what do you do now? Well the process of forecasting is a cycle and you now need to measure the accuracy of your predictions. Once you have the actuals to compare to your forecast, you can measure your forecast accuracy and use this to determine whether your current forecasting model is working.
There is a basic formula you can use to compare your forecast to your actual results, which is the mean absolute percent error (MAPE):
This formula requires you to calculate the mean of the absolute percent error for each time period, giving you your forecast accuracy for the total given forecast period.
Additionally, you will want to analyze your forecast accuracy for just a single period if your forecast accuracy is low. Looking at the percent error month to month will allow you to pin point where the largest error in your forecast is and help you determine the root of the problem.
Keep in mind that accuracy is crucial if organic traffic is a powerful source of product revenue for your business. This is where exceeding expectations can be a bad thing. If you exceed forecast, this can result in stock outs on products and a loss in potential revenue.
Consider the typical online consumer, do you think they will wait to purchase your product on your site if they can find it somewhere else? Online shoppers want immediate results, so making sure you can fulfil their order makes for better customer service and less bounces on product pages (which can affect rank as we know).
Top result for this query is out of stock, which will not help maintain that position in the long term.
Now this doesn't mean you should over forecast. There is a price to pay on both ends of the spectrum. Inflating your forecast means you could be bringing in excess inventory as it ties to product expectations. This can bring in unnecessary inventory expenses such as increased storage costs and tie up cash flow until the excess product is shipped. And dependent on product life cycles, continuing this practice can lead to an abundance of obsolete product and huge financial problems.
So once you have measured your forecast to actuals and considered the above, you can repeat the process more accurately and refine your forecast! Well this concludes our crash course in forecasting and how to apply it to organic traffic. So what are you waiting for? Start forecasting!
Oh and here is a little treat to get you started.
Are you telling me you built a time machine...in Excel?
Well no, Excel can't help you time travel, but it can help you forecast. The way I see it, if you're gonna build a forecast in Excel, why not do it in style?
I decided that your brain has probably gone to mush by now, so I am going to help you on your way to forecasting until the end of days. I am providing a stylish little excel template that has several features, but I warn you it doesn't do all the work.
It's nothing to spectacular, but this template will put you on your way to analyzing your historical data and building your forecast. Forecasting isn't an exact science, so naturally you need to do some work and make the call on what needs to be added or subtracted to the data.
What this excel template provides:
The ability to plug in the last two years of monthly organic traffic data and see a number of statistical calculations that will allow you to quickly analyze your historical data.
Provides you with the frequency distribution of your data.
Highlights the data points that are more than a standard deviation from the mean.
Provides you with some metrics we discussed (mean, growth rate, standard deviation, etc).
Oh wait there's more?
Yes. Yes. Yes. This simple tool will graph your historical and forecast data, provide you with a base forecast, and a place to easily add anything you need to account for in the forecast. Lastly, for those who don't have revenue data tied to Analytics, it provides you with a place to add your AOV and Average Conversion Rate to estimate future organic revenue as well. Now go have some fun with it.
________________________________________________________________________________________
Obviously we can't cover everything you need to know about forecasting in a single blog post. That goes both from a strategic and mathematical standpoint. So let me know what you think, what I missed, or if there are any points or tools that you think are applicable for the typical marketer to add to their skillset and spend some time learning.
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!