How Much Data do You Need to Reach Statistical Significance?
One of the hardest things to do when running split tests is to understand when you have results that are significant enough to signal it’s time to stop testing and move on to the next step.
I was recently running a very thorough split test that examined search vs content data. I took some of the ads that were running throughout that time and put them into several split tester calculators to see if I agreed with any of them.
You will quickly notice a downfall in the tools; but don’t despair. It comes down to the data. Do you really have enough data to compute statistical significance?
My suggestions are at the bottom of this article.
SplitTester.com
Splittester.com uses a combination of clicks and click through rate to determine significance. While it can be useful to see which ad will get the higher click through rate (CTR); most of us are much more interested in profit. However, if you were looking for traffic – CTR is a useful metric.
Just a couple hours into the campaign, AdWords refreshed the data. I took the initial data and put it into splittester.com. Below are the results:
Only 13 clicks and it seems I have statistical significance. I hope everyone of you just sat up and yelled that wasn’t enough clicks. You’re right.
By day 2, the results were exactly the opposite. Ad copy 2 was now statistically more relevant.
I was actually testing several more ads than 2, but you can’t compare several different ad copies at this site. In addition, there’s no mention of conversions, just clicks on the first test site. If traffic is your goal, and you have the minimum amount of data necessary (see below), this could be useful.
SEO Book’s Calculator
Next, I went over to Aaron Wall’s calculator. SEObook has a very nice suite of tools which are free to use. One of them is a PPC G-test calculator. The very first thing you’ll notice is that SEObook’s calculator allows you to test several variations.
The very first thing I noticed about Aaron’s tool is that it includes a warning that my data points are too low. Thank you, Aaron. However, the ad which has a 97.64% confidence that it should be dropped is the eventual winning ad.
I do like that Aaron’s calculator is based upon successes. You can use this tool to calculate ad tests, landing page tests, or even combinations of the two. It’s quite useful when you have enough data.
Not going to bore you
I tested out many more tools, and almost all of them gave me similar results.
When do you have enough data to utilize the calculators?
It’s not that these are bad tools. When used correctly, they are both useful tools.
The issue is that that you need to understand when you have enough data to actually believe your results.
Segment search vs content data. Do not combine these two mediums into one set of test results.
Below are my ‘rules of thumb’ to determine if I have enough data to even move onto calculators:
Time:
- Minimum: 1 week. Each day has different characteristics. Allow those variances to run over a week’s time. (One week case study)
- Better: 1 month. Each week has variances (especially payday weeks). Allow that to play over a month’s time. (Common to see variances in luxury goods)
- Ideal: 3 buying cycles (with a minimum of one month)
Traffic:
- Minimum 300 clicks per ad (and I still think this is too low)
- Better: 500 clicks per ad
- Ideally: 1000 clicks per ad
Conversions:
- Minimum: 7 conversions per ad
- Ideally: 15+ conversions per ad
Temperance:
If you are running 10,000 clicks a day, you might want more data as results can change over the buying cycle.
If you are receiving 1000 clicks a month, you might need to weigh when you can make a decision vs the data you have.
You will have to weight how much data you receive vs making decisions.
Ideally, you’d want to reach every milestone (1000 clicks, 15 conversions, and 3 buying cycles with a minimum of one month) before making decisions.
Of course, use common sense. If you’ve been testing for a month and one ad has 30 conversions and the other has 4 (assuming they have a similar amount of clicks), you can make assumptions.
The golden rule of optimization?
It is more important to believe the data then to complete a test.
Do not make decisions based upon insufficient data – all you will do is hurt your business
Conclusion
Online tools and excel calculators are not inherently bad – they’re just doing some math.
As a marketer, your job is to not only run tests – but to ensure that you have the proper amount of data before using such tools to complete a test. As always, you want to get the actionable analysis stage. That is the goal – to run a test to determine which marketing message promote your business’s goals so everyone succeeds.
You can’t do that with insufficient data.
Sometimes you have to be patient.
The SEMMYS, Controversy, and Analytics Blogs You Should Read
Matt has taken the best posts from his feed reader over the past year and organized them into categories for judging the best posts of 2007. It’s usually easier to judge a blog then individual posts, so this is a pretty ambitious endeavor.
Of course, the SEMMYS are highly controversial and generating its fair share of buzz.
Personally, I’m judging the Analytics and Local Search categories.
I also have two blog posts nominated for SEMMYS:
which I find nice for someone who only blogs part-time (at best).
As the awards are based on Matt’s last year’s feed reader - you can see the problems:
- Matt can’t read every blog post
- Matt’s reader could be skewed away / towards certain types of blogs
- How can you nominate blog posts in the future
- How can a panel share and collaborate on next year’s posts
Regardless of the complications of the first year beta-SEMMYS; Matt has put in quite a bit of work, and he does read a significant number of blogs. Kudos for trying to pull off an ambitious project that involves the actual post over the entire blog.
I’ve already given Matt a few additional blogs (especially in analytics) to read for the upcoming year to expand the SEMMYS for 2009.
I find very few good analytics blogs. What I’ve seen is that more and more analytics blogs are focusing on Google Analytics. While there is a need for GA blogs, there is also a need for non-GA blogs as well.
If you’re looking for analytics blogs, here’s a list to start with:
- Occam’s Razor
- Google Analytics Blog
- Web Analytics World
- Web Metrics Guru
- Unofficial Google Analytics Blog
- Luna Metrics
- Web Analytics Association
- Web Analytics Association RSS feeds (I also like the articles)
- Optimize and Prophesize (some metrics plus other commentary)
I’m looking for more blogs that really talk about A/B, multivariate, focus, and other testing areas (as well as good non GA blogs) if anyone has suggestions. I’m sure there is also a good blog about Google’s website optimizer, I just haven’t found it yet.
Google Analytics Now Showing Site Overlay
Site Overlay is one of the most popular aspects of some high scale (and cost) analytic platforms. Site overlay allows one to view their site within the analytics program. The analytics program then overlays certain statistics on the website, so you can view exactly how certain pages and links are preforming.
The most common use of site overlay is to make quick decisions on how the home page is preforming. Do you have a product on the home page that isn’t converting well? Site overlay will let you know that, and then you can make some quick adjustments to hte image, or replace the valuable real estate with a higher preforming link or image.
To access Google Analytics Site Overlay, log into your analytics account. Naviage to analytics, and you should see the navigation screen below:
Click on ‘-Site Overlay’ and you’ll be presented with your website.
Some items to note about the Analytics Frame:
- If you are using a frame busting script, Site Overlay will not work as your site should reload the Google Analytics page onto your website.
- If you are using a frame busting script and it doesn’t break out of the Site Overlay frame - find another script.
Some things to note about my Conversion Tracking:
- G3 = Goal 3 in the user defined analytics.
- G3 = Someone who has filled out the form for my Free AdWords Keyword Book
- I’ve set the form at a dollar rate of $10 (For free items, round numbers are useful to examine dollars and percentages)
In the Site Overlay screen, there are blue and green bars.
- The blue bar represents total number of clicks on any particular link.
- The green bar represents the quality of that click (as it pertains to average score)
Average Score is a new metric for Google Analytics and is defined as:
Many Google Analytics reports include the Average Score metric. Average Score is a way of determining which pages on your site are most valuable. Pages with the highest scores are those which lead to a large proportion of high-value conversions.
Average score for a page is calculated as follows:
(Total Value of Conversions Subsequent to Page Visit) / (Total Number of Page Visits)
From: https://adwords.google.com/support/bin/index.py?fulldump=1#27260
One can click on multiple bars and leave them open for number comparisons:
If this were an ecommerce site, there are some interesting observations we could make based upon the numbers presented.
The first screen shot is the main link from the homepage that goes directly to the eBook page:
What we can learn from these statistics:
- Not many people form the homepage click on this link. Only 3.71%, or 21 people during this time frame, clicked on the link.
- Of those who did click on the link, 38.1% of them filled out the form and became a conversion.
If this were an eCommerce site, this is the assumptions that I’d be starting from:
- The conversion rate is excellent, meaning the destination page from that link does not have to be modified.
- The click percentage is relativly low. Since this link has a high conversion rate, it would be time to think about how to increase that link’s click through rate.
- This could take the form of a different from page copy.
- It could mean adding an image to the page (images capture the eye and often lead to higher CTRs).
- The other thing to note, is if the page has such a high conversion rate - is the page accessible from other pages on the website? Adding a navigational item to the download could aid in more total conversions.
The next screenshot is those who clicked on the home page link.
Please note, this is a screenshot of the homepage, so this is the number of people who clicked on the homepage link from the homepage.
It’s no surprise that the conversion rate and average score on this link is fairly low. If someone was already looking at the homepage and didn’t click the more direct product link, then odds are, reloading the same page is not going to do much for their user experience.
However, the fact that 15.37% of people clicked the homepage link from the homepage probably means they didn’t realize where they were in the first place. This is obviously an issue, and in an ecommerce site should be addressed. Possible ways to address this could include:
- Removing the homepage link completely if one is one the homepage.
- Adding a bread crumb navigation so visitors understand exactly where in the website they are browsing.
- Adding more attractive options on the homepage so that visitors don’t feel they need to view the homepage to find what they’re looking for.
The next link leads to the ‘about section’:
The actions on this link surprised me, therefore I included it in the screenshot:
The about page doesn’t have a direct link to the download. One has to navigate away from that page into the projects page, homepage, or blog entry about the eBook for one to arrive at the download page. While the total number of clicks is pretty low on this link, the fact that those who follow that path through the site convert at a stunning 36% could mean a few things:
- The about page does its job of convincing someone they are getting information from an expert in the field (This is not a bragging statement, its what I’d tell a 3rd party if I was doing usability and conversion consulting for them).
- It would be worth testing adding a link on the about page (within the context of the page, so a short bullet point about the eBook) to see if that further increased conversion rates.
- With a click percentage of less than 2%, not many people are looking that deep into the website’s author. So, while spending a little time on the page could be useful, since its not a major traffic stream, it would not be worth spending several hours working on that page compared to the benefits that would be reaped by other suggestions.
Conclusion
The one metric that is missing is the average page views after a particular link is clicked (There are some technical issues with that however, as average page view by navigation should have a filter of all, from initial page view, from current page view, etc - and puts additional strain on the database).
Overall, the use of Site Overlay can give one an excellent insight into how visitors are engaged with various pages. With the web, and the amount of statistics we can bury ourselves in - its important to know when something should be used.
If you have a 100k page site, a tool like this is very useful for the top level pages and special promotion pages. Using a tool like this is wonderful for taking a sampling of interior pages and how visitors are interacting with your website. However, on a site of that size, one should be looking at overall averages as well for various product pages in determining designs and wording that help with conversions.
In a small site, this tool is wonderful as you can start to customize every page by conversion rates and visitor activity. However, overall averages are still useful in seeing your site from a holistic standpoint and not just page by page.
Please note: When I talk about overall averages - they are only so useful. Very good averages start to occur when you filter results. An example of filtering would be looking only at averages from people who came from AdWords, or YSM, or a particular banner ad. Each form of traffic has its own characteristics, and you should embrace those differences, not average them into your overall statistics.
Google Analytics Site Overlay is a great tool. It also has a difficult price to beat: Free. Google continues to make its mark throughout the web by offering very powerful tools at this price point. Offering an analytics program for free that has the ability for one to understand their AdWords conversions and ROI is inherently dangerous. If AdWords doesn’t produce the ROI, conversions, etc that advertisers want, then they could start spending less.
On the flip side, if analytics proves to advertisers how well their advertising dollars are succeeding on Google, it helps them increase their budget and offer more profit for all.
I have a feeling (based on many conversations with Google individuals) that the real answer is they completely believe in their product. Its this belief in the quality of both their traffic and their product that they want to see advertisers succeed, and they are offering the tools to assist them in making these beliefs a reality.
Traffic does Not Convert the Same: SEO vs PPC vs Press Releases vs Social Bookmarking
Each type of traffic has its own unique characteristics. Some distribution brings in more traffic, other types of traffic provides more backlinks, yet another traffic source will bring in more brand awareness. This test is an examination of the conversion rates by traffic source.
Traffic does Not Convert the Same: SEO vs PPC vs Press Releases vs Social Bookmarking
I have long suspected that not all traffic converts the same. While I had rough statistics to back it up (and very detailed statistics in some cases), I had never run a formal test to include ‘Social Bookmarking’ traffic to determine conversion rates and percentage of traffic.
Over the past 3 months, I’ve been running traffic statistics on myFree Keyword Research eBook offer to determine the difference in conversion rates by traffic source.
My original assumptions were:
- Press Release traffic would be the lowest conversion among all sources. My reasoning was that many people who read press releases are journalists looking for stories (and bloggers count as journalists for this purpose). They would be looking at news, a few would download the eBook, but more would just link or reference the press release in their writings without ever having looked at the book.
- Social Bookmarking traffic (i.e. delicious, technorati, digg, etc) will be the second lowest conversion rate among all traffic. My reasoning for this was that many of these visitors are just looking at the latest news and aren’t in the mindset to be converted. These individuals are often browsers, and clicking on social links are more akin to browsing than searching to find the answer to a progblem. They are also more sophisticated that the ‘average’ internet user - thus they will be immune to many offers.
- SEO traffic (free search engine traffic) would convert somewhere in the middle. Often, a page ranks for something other than the most converting keywords. Thus, ranking for broad keywords would provide traffic that is looking for something other than what a page offers, and thus these visitors will not convert. However, SEO traffic must be broken down into two groups to really test the effectiveness of the traffic:
- Search engine traffic for direct keywords
- Search engine traffic for broad keywords.
- PPC traffic would be the highest converting traffic. Being able to test keywords by conversion and make those adjustments would lead to the highest conversion rate among the entire group.
What is not taken into account is profits by total traffic. In other words, if Social Bookmarking provided 10x the traffic as SEO, but converted at 3x less, it would bring in more total conversions. I didn’t want to test the quantity of the traffic and conversions, just the quality.
I also wasn’t tracking RSS feeds differently than other traffic sources. It would be interesting to be able to add RSS feed conversions into this test - maybe next time.
The Methodology Was:
On the day the eBook was released:
- Write a blog post about it which pings the major blog sites.
- ‘Seed’ the tagging engines with enough information to bring in some visitors (this worked better than anticipated).
- Send a Press Release through all the major PR services, including a few donations to make sure the press release was picked up by Yahoo news and a few other large news aggregators.
- Send PPC traffic to the page from Yahoo Search Marking, Google AdWords, and MSN adCenter (adCenter traffic was included at a later date as the ’self serve’ environment was not operational when I launched the test)..
- Drop a few links from other sites to the page so that it would gain some rankings for very specific keywords (I also did a few 301s from pages which were ranking for specific keywords already).
Important Note: The keywords involved are things like ‘AdWords keyword research’, ‘PPC Keywords’, etc. They are mostly ‘jargon’ terms that would suggest a more sophisticated audience that is doing some internet marketing.
The first 2 months of results:
| Month 1: October | ||
| Percentage of All Traffic | Conversion Percentage | |
| Press Releases | 22 | 0.89% |
| Social Bookmarking Sites | 26 | 0.91% |
| SEO - Direct Keywords | 11 | 2.93% |
| SEO - Broad Keywords | 8 | 1.47% |
| PPC | 29 | 6.42% |
| Unknown (or Direct) | 4 | 2.41% |
| Month 2: November | ||
| Percentage of All Traffic | Conversion Percentage | |
| Press Releases | 9 | 1.12% |
| Social Bookmarking Sites | 11 | 1.14% |
| SEO - Direct Keywords | 13 | 2.29% |
| SEO - Broad Keywords | 12 | 1.51% |
| PPC | 47 | 6.53% |
| Unknown (or Direct) | 8 | 2.32% |
In December, I decided to do a bit of split testing on the PPC conversions, which also meant I upped my PPC budget (hence why PPC is such a large percentage of traffic). It had been a while since I’d written a ‘One Page Wonder’ site, and didn’t want to take the time to put one together. So, I wrote a simple informational message and removed the right hand navigation from the page. I then tested the original landing page against the new landing page.
The results are NOT a typo. I was pretty amazed by this conversion difference with such a small change. Usually a change like this can be quite significant. I had assumed that because the audience was a more internet sophisticated audience that it would be a bit more immune to these changes.
| Month 2: December | ||
| Percentage of All Traffic | Conversion Percentage | |
| Press Releases | 4 | 1.85% |
| Social Bookmarking Sites | 5 | 1.95% |
| SEO - Direct Keywords | 13 | 1.92% |
| SEO - Broad Keywords | 9 | 0.98% |
| PPC - Original Landing Page | 33 | 6.35% |
| PPC - New Test Page | 32 | 19.1% |
| Unknown (or Direct) | 4 | 1.37% |
What the results tell us (please remember, this is based on a single test, it is not across multiple platforms - so these inferences do lack the data integrity of running the same test across 20+ sites or 10+ offers should be noted).
Traffic Sources:
Press Release and Social Bookmarking traffic declined pretty steadily after the initial surge. However, their conversion percentage continues to climb - presumably, this is because those finding the information at such a later date are searching for something more specific. The amount of traffic that did continue to come through these sources amazed me. I was anticipating a larger drop. Yahoo News is by far the leading referral source among the Press Release traffic (I counted Yahoo News in the PR traffic as that’s the only reason I received that referral traffic at all).
I’m a huge fan of PPC traffic, both in quality and testing. I always find it useful when launching something more ‘official’ than just a free eBook to first test pages and offers with PPC traffic and then start to SEO, use press releases, etc (increased traffic) once you know what offer actually works.
I’m not sure why there was such a jump in ‘Unknown or Direct’ traffic in month 2. I could probably investigate it further, but its not worth the invested time for such a small test. What is interesting to note however, is that my highest page views per visit (for the broad sweeping categories above, not by individual referrer) came from direct traffic (as an FYI - my IP is blocked from my various stats program, so it’s not my own page views bringing this number higher) in all 3 months of testing.
Traffic Conversions:
Most of the initial assumptions proved to be fairly accurate. Social Bookmarking traffic did overtake Press Release traffic in terms of conversions in December.
I was surprised that the ‘SEO Direct Keyword’ traffic conversion percentage wasn’t more inline with the PPC traffic. Even for the exact same keywords, the PPC traffic had a higher conversion rate. I’m sure there’s some inferences that can be made about those who click on ads as opposed to natural listings for these results.
As a note, in the PPC traffic the highest conversion percentage came from MSN (which did have a small sample set due to their not being included in the initial test and the volume of traffic they can deliver), the second highest from Yahoo Search Marketing, and the lowest from Google AdWords. However; Google AdWords by far made up the most conversions among PPC traffic.
It is also good to note that why I didn’t break it out above, the conversion percentage from AdWords content match was higher than from AdWords Search (and actually higher than Yahoo Search Marketing). The cost per conversion on AdWords content match was three times lower than AdWords search. I definitely will not make any broad sweeping statements about AdWords content match here, as I’ve seen cases where it can be great and other times where it just doesn’t convert. This is just a note for this particular test.
Testing Analysis:
My mantra at every single conference I speak at is this: Anything is possible - test the results - the numbers will ultimately tell the story. Again, this proved true. The PPC test between the original landing page and the new landing page was significant. While I expected the new landing page to have a few points higher in terms of conversions, I did not expect it to be 300% higher.
PPC is an environment where you have more control than any other traffic environment on the web. Use that control to test ads, landing pages, keywords, etc - everything can be tested. Only by testing can you really know what happens.
Conclusion:
I love testing and putting together traffic numbers. Speculation is great - but in the end, there are only a few metrics that actually matter.
Social Bookmarking and Press Release traffic does not convert as well as other types of traffic. However, they do have an unspoken impact on SEO (gaining links, coverage, and overall exposure) which can be beneficial in the long run. While the traffic did not convert as well as other traffic, I wouldn’t say that it’s worthless traffic. It did convert somewhat, and bestowed branding exposure which is very worthwhile in the long term.
If you are running a purely direct response site, then the effort may not be worthwhile. If you are running a website that relies on repeat visitors, branding, organic traffic, etc - then the benefits of going through the effort of building social awareness is time well spent.
This is the story of just one website and one single test.
Always test your traffic. Test by keywords, traffic sources, country, time of day, etc.
Only testing, measuring, and interpreting results can tell the story of any website and its traffic.
Related Links:
Google Offers Urchin For Free
Google has announced today the launch of Google Analytics. Essentially, it’s Urchin 5.0 for free.
Once must use a Google account to access the tools. This seems just one more step in the consolidating of user information by Google.
Urchin 5.0 is a 3rd party cookie serving system. So, if you’re currently having issues with 3rd party cookies, this software won’t solve your problems like a 1st party system will. However, if you don’t know where to start for stats and integrated information, then this might be a great place to start.
The other aspect that hasn’t been talked about yet is it’s ability to integrate with Google’s cross channel tracking system. Google launched a program quite some time ago which enabled one to track any PPC or Advertising action on their website with the Google code. This might be a way for Google to facilitate for tracking all your advertising.
Before you think that Google is trying to store all your information and could use it against you, remember:
1. The use of Urchin is voluntary.
2. YSM (yahoo search marketing) has been allowing advertisers to utilize this type of tracking for well over a year.
Along with the free product, Google has a new set of information on their site entitled, Conversion University. Currently there are two sections, Driving Traffic, and Converting Visitors.
If you’ve been using PPC and online advertising for a while, the articles will seem quite basic. If you’re still new to the PPC game, they’re worth reading (as is just about anything that deals with converting visitors).
As the site is currently down (seems the website wasn’t ready for the traffic), there is no way to tell quite what will be involved, such as:
1. What is Google’s Privacy Policy on this tracking?
2. How often will your analytics be down?
3. Will Google place a visible Urchin logo on your site (like the Google conversion tracker does on any converting pages at present)
4. How will this affect your Google logins. Do you want your Froogle, GMail, Local Business, AdWords, AdSense, Search Query, RSS Feeds, Bookmarks, and Tracking information in a single database?
Only time will tell the success of the program; however, there is no doubt that it will be widely used. Analytics are often the last thing small advertisers approach, and the no cost barrier will help small advertisers get more out of their visitors and statistics than they normally would. For that, Google has done a good service for small businesses. For the rest - we’ll see.
Conversion Tracking Problems
There are a few times when strict tracking of pay per click accounts will not result in accurate numbers.
The first is Value of a Lifetime Customer. If a customer buys from your site multiple times, yet only the first visit came from a pay per click search engine, then the ppc account will only be credited with the first purchase, and not the repeat purchases which are directly attributed to this method of advertising.
Read more
Pay Per Click Tracking - Java Script
Java script is a common method of tracking. Most users have java script enabled on their computers (it is often enabled by default on many browsers). However, some do not.
Read more
Pay Per Click Tracking - Cookies
The advantage of cookies is that they stay on a users computer, and can track people for longer periods of time over the initial visit. This allows repeat customers, and customers who do not return to your website on the second visit through a pay per click search engine, to be properly credited to your pay per click advertising.
Read more
Pay Per Click Tracking - Images
Image tracking occurs by placing a small invisible image throughout your website. When the image is loaded, the visitor is tracked, and can be tracked on your landing page and your conversion page.
Read more
ROI (Return on Investment) Tracking
There are several items that you need to track to properly understand and maximize your ROI.
Read more
Google AdWords
Seminars for Success
Learn about Google AdWords from experts hand selected by Google.
These seminars will educate advertisers on the creation and management of successful AdWords campaigns.
Upcoming Seminars:
Boston AdWords Seminars June 24th & June 25th
Minneapolis AdWords Seminars July 16th & July 17th
Denver AdWords Seminars August 4th & 5th
Learn More about the Seminars:
Seminar Information
Official Google Seminar Page
Suggest a new city
Brad Geddes
Leslie Clark
Other Memberships
