How Much Data do You Need to Reach Statistical Significance?
One of the hardest things to do when running split tests is to understand when you have results that are significant enough to signal it’s time to stop testing and move on to the next step.
I was recently running a very thorough split test that examined search vs content data. I took some of the ads that were running throughout that time and put them into several split tester calculators to see if I agreed with any of them.
You will quickly notice a downfall in the tools; but don’t despair. It comes down to the data. Do you really have enough data to compute statistical significance?
My suggestions are at the bottom of this article.
SplitTester.com
Splittester.com uses a combination of clicks and click through rate to determine significance. While it can be useful to see which ad will get the higher click through rate (CTR); most of us are much more interested in profit. However, if you were looking for traffic – CTR is a useful metric.
Just a couple hours into the campaign, AdWords refreshed the data. I took the initial data and put it into splittester.com. Below are the results:
Only 13 clicks and it seems I have statistical significance. I hope everyone of you just sat up and yelled that wasn’t enough clicks. You’re right.
By day 2, the results were exactly the opposite. Ad copy 2 was now statistically more relevant.
I was actually testing several more ads than 2, but you can’t compare several different ad copies at this site. In addition, there’s no mention of conversions, just clicks on the first test site. If traffic is your goal, and you have the minimum amount of data necessary (see below), this could be useful.
SEO Book’s Calculator
Next, I went over to Aaron Wall’s calculator. SEObook has a very nice suite of tools which are free to use. One of them is a PPC G-test calculator. The very first thing you’ll notice is that SEObook’s calculator allows you to test several variations.
The very first thing I noticed about Aaron’s tool is that it includes a warning that my data points are too low. Thank you, Aaron. However, the ad which has a 97.64% confidence that it should be dropped is the eventual winning ad.
I do like that Aaron’s calculator is based upon successes. You can use this tool to calculate ad tests, landing page tests, or even combinations of the two. It’s quite useful when you have enough data.
Not going to bore you
I tested out many more tools, and almost all of them gave me similar results.
When do you have enough data to utilize the calculators?
It’s not that these are bad tools. When used correctly, they are both useful tools.
The issue is that that you need to understand when you have enough data to actually believe your results.
Segment search vs content data. Do not combine these two mediums into one set of test results.
Below are my ‘rules of thumb’ to determine if I have enough data to even move onto calculators:
Time:
- Minimum: 1 week. Each day has different characteristics. Allow those variances to run over a week’s time. (One week case study)
- Better: 1 month. Each week has variances (especially payday weeks). Allow that to play over a month’s time. (Common to see variances in luxury goods)
- Ideal: 3 buying cycles (with a minimum of one month)
Traffic:
- Minimum 300 clicks per ad (and I still think this is too low)
- Better: 500 clicks per ad
- Ideally: 1000 clicks per ad
Conversions:
- Minimum: 7 conversions per ad
- Ideally: 15+ conversions per ad
Temperance:
If you are running 10,000 clicks a day, you might want more data as results can change over the buying cycle.
If you are receiving 1000 clicks a month, you might need to weigh when you can make a decision vs the data you have.
You will have to weight how much data you receive vs making decisions.
Ideally, you’d want to reach every milestone (1000 clicks, 15 conversions, and 3 buying cycles with a minimum of one month) before making decisions.
Of course, use common sense. If you’ve been testing for a month and one ad has 30 conversions and the other has 4 (assuming they have a similar amount of clicks), you can make assumptions.
The golden rule of optimization?
It is more important to believe the data then to complete a test.
Do not make decisions based upon insufficient data – all you will do is hurt your business
Conclusion
Online tools and excel calculators are not inherently bad – they’re just doing some math.
As a marketer, your job is to not only run tests – but to ensure that you have the proper amount of data before using such tools to complete a test. As always, you want to get the actionable analysis stage. That is the goal – to run a test to determine which marketing message promote your business’s goals so everyone succeeds.
You can’t do that with insufficient data.
Sometimes you have to be patient.
Q&A - Should I use both Google Analytics and AdWords Conversion Tracker?
Yes! Please! I often survey attendees of an AdWords Seminar, and a common theme often emerges.
- Most of the audience uses Google Analytics.
- Most of the audience does not use the AdWords conversion tracker.
They are different tools that should be used differently.
AdWords Conversion Tracking
The AdWords conversion tracker just tracks conversions. You can define a conversion in many different ways (sale, lead form, page view) etc. You can define different conversions within the same account. Once you’ve enabled conversion tracking, you can run reports to see performance statistics by keyword, ad group, campaign, ad copy, landing page, content sites, etc by different conversion types (what keywords lead to contacts vs. shopping cart checkouts).
AdWords conversion tracking passes that stats back to your AdWords account so you can easily access all you conversion information cross referenced with your AdWords data within the single AdWords reporting interface. This is a huge timesaver.
Google Analytics
Google Analytics is an analytics system. It will give you more information than you need to know about your site and visitors. You can easily define goals (conversions) within your analytics account. Google Analytics will tell you about visitors to your site that came from other sources than AdWords. You can view browser, screen resolution, referring site info, and your AdWords account within Analytics (as well as hundreds of other data points).
Like AdWords conversion tracking, analytics should be used to make decisions. Whenever you decide to implement analytics, the first question you should ask is ‘What information do I need to know to get my job done well?’. You can ask that of the design, marketing, and other departments within your organization. Then look for an analytics solution that fits your needs.
For most small businesses, Google Analytics will fit that need. If you need to make real time decisions, then it will not meet your needs.
However, it is much more difficult to extract data such as what content sites are sending you converting traffic from your Analytics account. From within the AdWords conversion tracker – it’s easy. From Analytics – incredibly difficult.
Use Both Google Analytics and the AdWords Conversion Tracker
Assuming you’re willing to use Google Analytics, and it gives you the data you need to perform your job – then you should use them both. Since these two tools perform different functions, you need a tool for each function. Neither are difficult to install, and the data can be invaluable for increase your website’s effectiveness.
AdWords Conversion Tracking Resources:
- Video: Setting up conversion tracking (watch this first)
- Setting up AdWords conversion tracker FAQs
- PDF guide to setting up AdWords conversion tracking
- Troubleshooting conversion tracking
What Yahoo’s New Minimum Bid Means To You & When conversions outdo Sphinns
My newest article at Search Engine Land came out last Monday. It’s about what the new Yahoo minimum bid means your PPC campaigns.
I spent more time writing this article that almost any I’ve done in the past year. I had access to Yahoo decks, Yahoo PDFs, and Yahoo employees when I wrote this, and really tried to check the accuracy of everything as this is such a new change.
What I found fascinating was that while this column only produced 3 Sphinns; I received 34 contacts via my blog about the story. I think that falls somewhere in the top 20 ever received by this blog from a single post within a week (and it wasn’t even linked to from my blog - the only way to find this blog from the article is in the author section at the bottom of the article).
Sometimes social media looks like one thing (3 Sphinns is very little), while the results can be far different (I consider 34 contacts exceptionally high). Normally when my stories end up on social media sites, I receive very little comments, so I found this quite fascinating as it’s far from the norm that I see.
Enjoy the article, it’s worth reading as this is a change to the philosophy of minimum bids.
Short Descriptions Finally go Live on Yahoo
The YSM blog has been putting out warnings for some time that Yahoo would go to all short descriptions.
It seems today is that day.
Starting today, ads that appear in U.S. Yahoo! Search results with descriptions longer than 70 characters will be cut off (â€truncatedâ€) at the nearest complete word, followed by an ellipsis. The optional 190-character long descriptions may still be displayed on some of our distribution partners’ sites.
If you haven’t gone through your Yahoo Panama account and created short descriptions, you definitely should. Yahoo will automatically truncate your ads, and when the ads are truncated, you are losing control of your marketing message.
As an advertiser, one of your top priorities should be controlling your message.
More information from Yahoo on how to change your ads.
Some info about ad description writing.
Yahoo Rolls out Quality Based Pricing
Yahoo recently rolled out Quality Based Pricing.
Essentially, this says that not all clicks from the content network are created equal, and therefore, they advertisers should pay a different price based upon the quality of that click.
What’s interesting is Yahoo is saying they are using conversion data as one of the data points (there are others) in determining this pricing.
For those familiar with AdWords, it’s very similar to the AdWords Discounter.
Yahoo Offering Free Consultation for Panama Upgrades
Yahoo is sending out emails offering custom assistance for Panama upgrades. What’s interesting is that I received these emails even for a small account spending under $20 per month and another account which hasn’t been active in over two years.
There seem to be two subject lines that they are using:
- Need Help Restructuring Your Account?
- Call us Today to Help Restructure Your Account
If you need custom help understanding the upgrade or restructuring the account, Yahoo seems quite eager to help at present.
Here’s the email I’m receiving:
Dear Advertiser,
Our records show that you recently upgraded to the new Sponsored Search system. During the upgrade, your account was reorganized to match our new account structure, which we believe provides you with more flexibility to help manage your campaigns.
We know that not every customer is satisfied with the results of their upgrade and would like some help. If that’s you, as a valued customer, we’d like to offer a complimentary, one-time assistance with the restructuring of your upgraded account.*
This offer includes:
- Phone consultation of how you would like to reorganize your account
- Helpful advice on how to best structure your account
- Upload of your restructured account
Once you agree on how you’d like your account restructured, we’ll do the work and then we will upload the information to your account. Please note that your account will be taken offline after the upload, until you review the changes and put your account back online. You will be notified via email to log into your account, review the new structure and set your account status back to “On.” This entire process should be completed within approximately 5 business days after the consultation. However, the process time may vary depending on the complexity of your account.
If you’d like to take advantage of this offer, please contact us at [phone number removed].
Also, make sure to review our Quick Start Guide to get helpful information on the new system. Thank you for your business and we’re committed to helping you be successful.
Sincerely,
Your Partners at Yahoo! Search Marketing
Random Notes on Yahoo Panama
I sat through a Yahoo Panama seminar today and dug into some of the quality score algo and other information about need to know items on Panama.
Here are some rough notes about the new system:
There is an import process where you can upload your Google AdWords account and make it into a Panama account.
Accounts and campaigns have budgets. An account budget by default is off. If you ever enable an account budget, you can never disable it again (however, you could set an account budget to a million dollars a day if you wanted an unlimited spend).
An account budget can be lower than all of the campaigns combined (Why can’t Google have this feature?).
If you are using Yahoo’s conversion tracking system, you will want to update the code. Upon updating the code (or getting a new code) if you’re not sure if you want total stats or regular stats - choose regular stats. You can update to total stats, you can’t downgrade to regular stats.
If you never updated your ad curatives from 190 to 70 characters; its time. With the new ad ranking algos, you’ll want a bit more control over your ads.
The keyword and ad intersection is where quality score lives. Yahoo displays the ad’s quality score. They do not display the keyword quality score. This will be the subject of a much longer and detailed post.
Yahoo says more general ads do better (from a click through standpoint) than more specific ads on the content network.
Local search accounts will be migrated last. Local search accounts are going away. If it’s easy to map the location to a DMA, the account will be changed to the appropriate DMA (geographic targeting). If the account cannot be easily mapped - Yahoo will contact the advertiser.
An AdGroup can have up to 20 ads.
Content match is bid at the AdGroup level.
You can have a campaign which is only content match (or only search).
Reports are generally 1-3 hours behind real time.
Check your accounts time zone, that’s where you are seeing your stats.
You can choose not to show your ads to other continents (more on this later - it’s a cool feature) regardless of the search.
One can set permission levels on a login (administrator, campaign manager, etc); however, one cannot yet see changes by login.
You can now change your password without having to contact Yahoo.
If a keyword didn’t get a click in 13 months or an impression in the last 17 months - the keyword will not be migrated from your old account to your Panama account.
You can set a campaign’s start date to be in the future. You can set a campaign end date. This is useful for making sure all of your ads are approved before your campaigns start date. Very useful for time-sensitive campaigns (FYI - this could be done with Google bulk uploads, but now that those are going away - Google needs to follow Yahoo on this one).
Hope that was useful. More details to come.
Yahoo Panama Keyword Tool Allows For Site Spidering
One of my favorite keyword tools is the Google AdWords keyword tool that lets an advertiser spider one’s site. This functionality is useful in finding high quality score keywords, understanding what Google thinks a page is about, etc.
Yahoo Panama keyword tool now has similar functionality.
In the Yahoo Search Marketing keyword tool, one has several options:
- The typical ‘find related keywords’Â
- Look for words from a particular URL
- When searching for keywords, one can specify which words those phrases must contain
- When searching for keywords, specify which words must not be included
One must input three keywords that Yahoo will use as it’s ’seed’ keywords. In other words, these are keywords that you can find related keywords to, or if you are using site spidering (find keywords from a particular URL) it helps Yahoo to narrow down what it finds on your site.
However; what if you want Yahoo to spider your webpage and bring back results regardless of what you wish to filter by?
It’s easy - use blank spaces as a keyword.
Essentially, Yahoo’s only verification is that a character has been entered. It doesn’t care about the spelling, that it’s a space, etc - they just care something was entered.
Below is a screenshot of the input functions.
I inputted my Free AdWords Keyword research ebook (which I think will finally be update in the next couple of weeks) into the tool, and was presented with this information:
Now, because this is a blog and has recent articles, there are a lot of poor keyword choices such as ‘hotel Denver’ and ‘Chicago’. Therefore, I inputted the phrases ‘marketing’; ‘AdWords’; and ‘advertising’ to see what would be presented.
These seem a bit too general. So, what to do next?
Use the ‘refine feature’. This I do like quite a bit. You can tell Yahoo what words are relevant and what aren’t relevant and then the keyword tool will refine your results based upon your input.
Overall, Yahoo’s keyword tool is decent. It’s defiantly better research than their old interface tools. I still think Google’s is better at present; however, this is large strides for Yahoo and I can only hope to expect more of this in the future.
As Yahoo goes towards more quality score, this tool will be more useful in finding out what Yahoo thinks about your page. If you want to expand your keyword research, its also a good place for some expansion.
It is still a tool. The major effort first comes from brainpower.
Note: I’m currently sitting in a Yahoo seminar about Panama right now which is pretty good. I’ll post random useful notes later.
Yahoo Misses ‘Panama’ PPC Launch
The new features of Yahoo Search Marketing’s Panama project have been widely discussed and speculated upon. Terry Semel made it public yesterday that the launch of the new system will be delayed until Q4.
It seems that many inside Yahoo have been as uncertain of the launch dates as those outside of Yahoo. No wonder they’re are not many answers. We’ve already had several conference calls with Yahoo about the new API, new interface, new features, etc - and had always been led to believe that August 15th was the launch date. In fact, I had heard that launch date as far back as Q1 of this year.
It seems Yahoo set an aggressive date, tried to keep that date both publicly and internally, and then in the end - just couldn’t hit it.
Yahoo shares fell yesterday after they failed to meet ad sales expectations (and for your internet PPC geeks, many of their sales are CPM based, they have one of the best CPM behavioral buys around). The new Panama system will increase Yahoo’s inventory and profitability - something they sorely need as shares fell after the news was made public.
More Information:
- Yahoo’s Analyst Day Information
- Yahoo’s Rumored New Features
- Stock Ticker: YHOO Information
- What is all this PPC Stuff?
Yahoo Search Marketing’s New Features
Here’s a quick summary of what to expect from the new Yahoo Search Marketing capabilities. The US launch will occur in q3 of 06 with the first accounts being transitioned around August 15th. The international transition will occur in q1 of 07.
A full API with sandbox is available for Yahoo API users.
Account Organization
- Master Accounts
- Individual Accounts (unlimited)
- Campaigns (max 100)
- Ad Groups (max 1000 per campaign)
- Keywords (max 1000 per AdGroup)
- 50k-250k keywords per account
- Dynamic Keyword Insertion into titles or descriptions
Bidding
- Will utilize a quality index
- Placement will be based on CTR and Quality Index scoring
- Quality index will be made up of many factors including, ad copy and keyword history
- Quality index will be shown in the account similar to the current 1-5 rating (green bar) one currently sees.
- Competitor bids will be shown (similar to the current system)
- Composite quality index and keyword bid will not be displayed
Tools
- Forecasting
- Will have forecasting tools which include clicks, impressions, and position
- Keyword research tools
Additional Capabilities
- Geo Targeting - US & Uk only to start
- Keyword Research Tools
- Keyword research estimates
- Related keyword tool
Google AdWords
Seminars for Success
Learn about Google AdWords from experts hand selected by Google.
These seminars will educate advertisers on the creation and management of successful AdWords campaigns.
Upcoming Seminars:
Boston AdWords Seminars June 24th & June 25th
Minneapolis AdWords Seminars July 16th & July 17th
Denver AdWords Seminars August 4th & 5th
Learn More about the Seminars:
Seminar Information
Official Google Seminar Page
Suggest a new city
Brad Geddes
Leslie Clark
Other Memberships
