How Much Data do You Need to Reach Statistical Significance?
One of the hardest things to do when running split tests is to understand when you have results that are significant enough to signal it’s time to stop testing and move on to the next step.
I was recently running a very thorough split test that examined search vs content data. I took some of the ads that were running throughout that time and put them into several split tester calculators to see if I agreed with any of them.
You will quickly notice a downfall in the tools; but don’t despair. It comes down to the data. Do you really have enough data to compute statistical significance?
My suggestions are at the bottom of this article.
SplitTester.com
Splittester.com uses a combination of clicks and click through rate to determine significance. While it can be useful to see which ad will get the higher click through rate (CTR); most of us are much more interested in profit. However, if you were looking for traffic – CTR is a useful metric.
Just a couple hours into the campaign, AdWords refreshed the data. I took the initial data and put it into splittester.com. Below are the results:
Only 13 clicks and it seems I have statistical significance. I hope everyone of you just sat up and yelled that wasn’t enough clicks. You’re right.
By day 2, the results were exactly the opposite. Ad copy 2 was now statistically more relevant.
I was actually testing several more ads than 2, but you can’t compare several different ad copies at this site. In addition, there’s no mention of conversions, just clicks on the first test site. If traffic is your goal, and you have the minimum amount of data necessary (see below), this could be useful.
SEO Book’s Calculator
Next, I went over to Aaron Wall’s calculator. SEObook has a very nice suite of tools which are free to use. One of them is a PPC G-test calculator. The very first thing you’ll notice is that SEObook’s calculator allows you to test several variations.
The very first thing I noticed about Aaron’s tool is that it includes a warning that my data points are too low. Thank you, Aaron. However, the ad which has a 97.64% confidence that it should be dropped is the eventual winning ad.
I do like that Aaron’s calculator is based upon successes. You can use this tool to calculate ad tests, landing page tests, or even combinations of the two. It’s quite useful when you have enough data.
Not going to bore you
I tested out many more tools, and almost all of them gave me similar results.
When do you have enough data to utilize the calculators?
It’s not that these are bad tools. When used correctly, they are both useful tools.
The issue is that that you need to understand when you have enough data to actually believe your results.
Segment search vs content data. Do not combine these two mediums into one set of test results.
Below are my ‘rules of thumb’ to determine if I have enough data to even move onto calculators:
Time:
- Minimum: 1 week. Each day has different characteristics. Allow those variances to run over a week’s time. (One week case study)
- Better: 1 month. Each week has variances (especially payday weeks). Allow that to play over a month’s time. (Common to see variances in luxury goods)
- Ideal: 3 buying cycles (with a minimum of one month)
Traffic:
- Minimum 300 clicks per ad (and I still think this is too low)
- Better: 500 clicks per ad
- Ideally: 1000 clicks per ad
Conversions:
- Minimum: 7 conversions per ad
- Ideally: 15+ conversions per ad
Temperance:
If you are running 10,000 clicks a day, you might want more data as results can change over the buying cycle.
If you are receiving 1000 clicks a month, you might need to weigh when you can make a decision vs the data you have.
You will have to weight how much data you receive vs making decisions.
Ideally, you’d want to reach every milestone (1000 clicks, 15 conversions, and 3 buying cycles with a minimum of one month) before making decisions.
Of course, use common sense. If you’ve been testing for a month and one ad has 30 conversions and the other has 4 (assuming they have a similar amount of clicks), you can make assumptions.
The golden rule of optimization?
It is more important to believe the data then to complete a test.
Do not make decisions based upon insufficient data – all you will do is hurt your business
Conclusion
Online tools and excel calculators are not inherently bad – they’re just doing some math.
As a marketer, your job is to not only run tests – but to ensure that you have the proper amount of data before using such tools to complete a test. As always, you want to get the actionable analysis stage. That is the goal – to run a test to determine which marketing message promote your business’s goals so everyone succeeds.
You can’t do that with insufficient data.
Sometimes you have to be patient.
AdWords Local Business Ads Receive Upgrade – Learn how to Take Advantage of Google Maps Marketing
AdWords recently announced an upgrade to Local Business Ads:
Beginning today, your ad’s information window in Google Maps will feature new interactive links that are designed to connect users to your business quickly. The info window is the window that opens when a user clicks on an organic search listing or a sponsored result on Google Maps.
Interaction Report to be Updated
Soon, AdWords will also add more information to the Interaction Report. Right now, the interaction report is mostly used for learning about how user are interacting with your rich-media ads. In fact, the only help files for this report are under gadget ads. Hopefully, Google will update the report FAQs with gadget, video, and local business ads information.
Use Your Business Name in the Ad Copy
Local business ads are they only AdWords ad type where your default title should be your business name. Rarely is your business name the best headline. It’s always worth testing your business name in ad copy, but in general, it’s not a good headline for search.
Where are Local Business Ads Displayed
- Google maps
- Google maps for mobile
- Content sites using maps (this would be the content network, you can turn it off)
- Google Earth
- Other sites showing text ads
The last bit is the tricky part of LBAs (Local Business ads). If you just create a local business ad, that ad could be shown all over the AdWords networks. Don’t forget that you can control your LBA display with your campaign settings. I’ve even seen some companies with custom campaigns just for LBAs for tracking and ad serving control.
Creating Local Business Ads
This is one of the few places where Google uses non-AdWords data to control AdWords. Your business must first be listed in Google Maps before you can create an LBA. You can do a search for your business in Google Maps. If you don’t see your business listed, then you can add your business to Google Maps. Even if your business is listed in Google Maps, it’s a good idea to create a Local Business Center account and take control of your listing.
Note: If you have many locations, use the Bulk Upload function to save yourself time. If you’re someone who markets for local businesses, it can be a great idea to write a program that does this automatically from your customer’s source data.
Measuring Local Business Ad Results
You can always run an Ad Performance Report and see how your LBA is performing. If you use Google Conversion Tracker, this report will show you the cost per conversion and conversion rate of your LBAs (please use both Conversion tracking and analytics).
However, LBAs show a phone number in the ad copy. It could be your main business number – but it doesn’t have to be. You could use a different phone number in the ad copy and track how many calls came from just the ad. You’re not charged when someone views your LBA on Google maps – you are only charged when someone clicks on the ad and visits your site.
Conclusion
If you have a local business – you should use AdWords Local Business ads. The amount of inventory (impressions/clicks) available on Google maps has surprised many a business. Consumers generally go to Google Maps for two reasons:
- Driving Directions
- Find a business
If someone is looking for your products or services – you want to be in front of them. Local Business ads will definitely get you there.
Businesses are confused by internet marketing – Can your business find the right product mix?
The Yellow Pages aren’t dead.
Small businesses are confused by search marketing.
Over 50% of small and midsize businesses fail to properly track successes (Source)
Who will empower reps to sell the full marketing package?
Will your company be the one that breaks the magic formula?
There are millions of dollars for both small agencies and large sales forces lying around just by helping small businesses.
Some Background
In 2005, I spoke at Pubcon on how search engines need to make changes to help local adoption. About half of my points have been implemented by the search engines. At that time, there wasn’t an ad display hierarchy system that examined the search intent (by keyword or property) to display ads. These suggestions were implemented:
- Stop showing national ads over local ads on local search intent queries
- Show local ads on local properties
- Better tools for local advertisers
In 2006, I spoke at SES Local in Denver about SME campaigns. I had two points to make:
- Fulfillment from aggregators had to be better: Most campaigns were being set up with 1 ad copy and 25 keywords due to technology and scale limitations (although, I’m proud to say LocalLaunch could do hundreds of ads and thousands of keywords for a single SME campaign).
- Search Engines needed to empower agencies. A small business doesn’t have to understand the inner workings of search marketing; they just have to believe in the power of it. While part of this responsibility falls on the sales force – the rest falls on the search engines. There’s a reason both manufacturers and dealers run ads – they’re complimenting each other.
Both of these points have improved – but are definitely not solved.
In 2007, I spoke at Google on the network effect of searchers to publishers to advertisers. Along the way I made several suggestions; of which 90% have now come to pass (Thank you AdWords Team).
While the network effect works well for non-local queries; the process is still breaking down at the local level.
Yellow Pages + Internet Yellow Pages + Search Engine Marketing
Throughout those years, I worked with some very smart sales trainers on selling the “Triple Play”, which is a combination of Internet Yellow Pages, Yellow Pages, and Print advertising. It was RHD’s future strategy; and was so believed in that RHD acquired Business.com.
Why sell all 3?
There’s not enough search for all local businesses. The Yellow Pages are still highly used. One should never lean on just one source of leads for their business. IYPs have fantastic conversion rates. The list goes on.
However, just like one should not just buy AdWords or just rely on SEO for all of their traffic; advertisers should diversify their traffic sources.
While we were selling the Triple Play; that doesn’t mean you need to sell print. You need to sell a combination of advertising mediums that make up most (if not all) of a company’s leads.
A company’s position should be: When our clients think of advertising – they think of us first.
You should be the gateway between your clients and their leads.
Do you want to be part of the marketing mix or do you want to be the marketing mix?
Where does that leave us today?
Print and Search Engines are highly used to find local data:
You can also look at this and not think just YP and Search. It’s where eyeballs currently lie. If you can put a business in front of enough (and the correct) eyeballs, then you can be the marketing mix.
According to the study, the first sources used are Search Engines (31%), Print Yellow Pages or White Pages (30%), Internet Yellow Pages Sites (19%) and Local Search Sites (11%). This represents a change from last year’s study, which ranked Print Yellow Pages first (33%), followed by Search Engines (30%).
- 47% of consumers have used a search engine in the past 30 days, compared with 64% for print Yellow Pages
- Teens are almost equally likely to have used a search engine (52%) or the print Yellow Pages (47%) in the past 30 days
- Search engines were deemed the “most useful” of the eight key platforms tested; they rated highest in satisfaction for being a free service, providing the right amount of information quickly, acting as a trusted resource, and providing relevant results
Businesses are confused by search engine marketing.
Source: Opus Research, AllBusiness.com (2008)
The study revealed that 59 percent of small businesses with Web sites don’t currently use paid search marketing, and of those, 90 percent have never even attempted it.
One quarter of respondents believe paid search marketing is too complex. Twenty-one percent thought it would be too time-consuming. Thirty-five percent felt they would need an agency to help set up a search marketing campaign.
If we examine the numbers:
- 15% State lack of time. Translation – they need a 3rd party
- 19% State Internet not relevant. Translation – they haven’t heard an internet marketing product pitch that resonates with them, waiting for the proper 3rd party to come along.
- 7% Skeptical of effeteness. Translation – still waiting on the correct product to come along.
- 25% State Confusion. Translation – they need a 3rd party.
That means 66% of businesses which are confused by search engine marketing just need the right product and sales rep to come along.
Will your company be the one that breaks the magic formula?
The formula isn’t hard:
- Create marketing products that:
- The business, executive team, and sales reps believe in.
- Bring ROI to the customer
- Will produce excellent renewal rates
- Can be understood by the business
- Can be understood by the sales reps
- Can be measured to show tangible value
- Train sales reps to:
- Use a need based analysis sales approach
- Understand what the business needs
- Determine which mix of products fits that need
- Be able to sell multiple products if necessary. If you’re not a newspaper or Yellow Page company, you can still sell display, IYP, websites, local data inclusion, etc. While search marketing is sexy; all of the above venues can work for a company.
- Understand the product
- Believe in the products
- Explain the products in simple terms
- Training the sales reps is often undervalued – please make sure your reps have the proper training.
- Use a need based analysis sales approach
- Train your analysts to:
- Understand search marketing
- Understand other marketing avenues of your product (IYP, display, print, etc)
- Understand the product (sales and analyst alignment are crucial)
- Properly fulfill the product
- Don’t use the ‘set and forget’ mentality
- Understand search marketing
- Implement technology to:
- Scale your business
- Improve your product’s value
- Showcase your product’s value to your clients
I could write another 20 pages about the formula, so this is the simplistic view.
Where does your business fit?
Small Businesses are confused about search marketing.
There is value in selling multiple products.
There is value in selling SEM (SEO + PPC).
There are millions of dollars on the table.
Is your company ready to help businesses?
If the answer is yes, good luck, there’s a lot of money sitting on the table waiting for the right product.
Q&A – Should I use both Google Analytics and AdWords Conversion Tracker?
Yes! Please! I often survey attendees of an AdWords Seminar, and a common theme often emerges.
- Most of the audience uses Google Analytics.
- Most of the audience does not use the AdWords conversion tracker.
They are different tools that should be used differently.
AdWords Conversion Tracking
The AdWords conversion tracker just tracks conversions. You can define a conversion in many different ways (sale, lead form, page view) etc. You can define different conversions within the same account. Once you’ve enabled conversion tracking, you can run reports to see performance statistics by keyword, ad group, campaign, ad copy, landing page, content sites, etc by different conversion types (what keywords lead to contacts vs. shopping cart checkouts).
AdWords conversion tracking passes that stats back to your AdWords account so you can easily access all you conversion information cross referenced with your AdWords data within the single AdWords reporting interface. This is a huge timesaver.
Google Analytics
Google Analytics is an analytics system. It will give you more information than you need to know about your site and visitors. You can easily define goals (conversions) within your analytics account. Google Analytics will tell you about visitors to your site that came from other sources than AdWords. You can view browser, screen resolution, referring site info, and your AdWords account within Analytics (as well as hundreds of other data points).
Like AdWords conversion tracking, analytics should be used to make decisions. Whenever you decide to implement analytics, the first question you should ask is ‘What information do I need to know to get my job done well?’. You can ask that of the design, marketing, and other departments within your organization. Then look for an analytics solution that fits your needs.
For most small businesses, Google Analytics will fit that need. If you need to make real time decisions, then it will not meet your needs.
However, it is much more difficult to extract data such as what content sites are sending you converting traffic from your Analytics account. From within the AdWords conversion tracker – it’s easy. From Analytics – incredibly difficult.
Use Both Google Analytics and the AdWords Conversion Tracker
Assuming you’re willing to use Google Analytics, and it gives you the data you need to perform your job – then you should use them both. Since these two tools perform different functions, you need a tool for each function. Neither are difficult to install, and the data can be invaluable for increase your website’s effectiveness.
AdWords Conversion Tracking Resources:
- Video: Setting up conversion tracking (watch this first)
- Setting up AdWords conversion tracker FAQs
- PDF guide to setting up AdWords conversion tracking
- Troubleshooting conversion tracking
Two Words about the Google Local Business Referral – Content & Awareness
The Google Local Business Referral program has been commented about heavily in the last 24 hours.
I’ve known about this program for a while, and being heavily involved in local search, I think it’s being severely misunderstood. It’s not about a sales force – it’s about local business content and customer awareness.
One of the challenges about local search is up-to-date accurate content.
I was asked at Search Engine Strategies San Jose last year: “What do you think the tipping point is for local search?”. Some of the answers were around having 1 million listings, others were about how many advertisers Google had, my answer was, and remains the same:
The tipping point of local search is all about the quantity and integrity of data. The day I can go to a local search engine and receive information that’s as accurate, as fast, and more useful than opening the Yellow Pages, is the day local search will have truly arrived.
Google is targeting college students with this initiative, not sales reps.
The second challenge in online advertising is customer awareness.
If you ask the companies why they are not advertising on the web, the most common answer is ‘We don’t trust it’. However, trust is usually a cover for some other issue. If you follow-up that question with ‘Why don’t you trust it?’. The most common answer is:
We don’t understand it.
It’s not about trust, it’s not about awareness and understanding.
If you look at the typical buying cycle – the very first stage is awareness.
In fact, I believe so much in the fact that awareness plays a critical role in online advertising, that I’m speaking at an RHD Seminar, ‘Introduction to the Web‘ seminar this Thursday (which Yahoo is attending and giving a fun Swag Quiz Presentation afterwards); and then following it with another Awareness Seminar in Vegas later this year.
Collect the Content – Raise the Awareness – Local Search will Arrive
Consumers still have challenges with local search. Often numbers are outdated, the listings are inaccurate, or the listings don’t exist for a business.
To layer on top of that the difficulty for a business to actually give their information to every local search engine. Business’s don’t know, and don’t have the time to submit a Google, Yahoo, Superpages, Yelp, Truelocal, Local.com, Switchboard, etc, etc listing.
There is a reason why programs such as Register Local exist. Register Local is a single point of data distribution for local businesses. Submit it once – view it everywhere.
Local search will arrive someday. However, it will take feet-on-the-street sales reps, content collection programs, and awareness raising for both businesses and consumers for that day to arrive.
Local Business Ads to be syndicated to Maps API Sites
These third-party websites use the Google Maps API, which allows them to embed customizable Google Maps within their site. Google technology will only display your clients’ ads when they’re related to the surrounding content of the webpage. As with all content targeted ads, your clients pay only when someone clicks through to the website.
From a Google newsletter.
It’s not clear if you have Local Business Ads if you will choose to syndicate these ads to Google mashups, or if they will be syndicated if that campaign has the ‘content network’ turned on.
It is clear that Google is pushing LBAs and trying to find more inventory for them (which I applaud), such as showing LBAs on Google Earth.
I hope that you can have the option to syndicate LBAs to maps mashup API sites without showing them on the general content network. That would give the advertisers more control, while receiving primarily local based inventory.
Google Upgrades Local Business Center
There are a few people in the industry who I’ve not really had a chance to talk to, and amazingly, Matt is one of them. We’ve been at the same conferences several times, yet never quite hooked up for some good conversation.
Matt has a fantastic article on the upgrades to the LBC. And since he beat me to writing about it, I’ll just let you follow the link to LBC Upgrade.
He’s also a Search Engine Land columnist (like myself) and wrote a nice piece today about the Top 20 Don’ts in SEO.
What the colors mean in Google Local Traffic View
Google recently caught up with other local search engines by showing traffic on local.google.com .
Here’s the key to what the colors mean:
- Green: more than 50 miles per hour
- Yellow: 25 – 50 miles per hour
- Red: less than 25 miles per hour
- Gray: no data available
Google Maps Now Showing Subway Icons
Google maps has recently started showing subway stops on maps. It seems much more prominent to see these icons on the directions page than on the general maps page.
However, if you were traveling to Chicago, this map would be quite confusing.
First off, several of those metros don’t connect. While the pins mark the stations, Chicago has several different trains that run downtown (named ‘the el’) and one often has to switch trains (and know where to transfer) to find a destination.
The second aspect is that mapping software doesn’t seem to take under ground vs above ground into determining directions. I will admit that the above is probably a trick query, it’s for Intelligentsia Coffee’s Pedway location (the pedway is an underground walkway that connects several parts of downtown Chicago. There’s no way to drive to this location, but you’d never know it from the map’s directions.
Overall, it’s probably a precursor of things to come. It would be very useful to be able to use a search engine’s mapping site to find public transportation directions. New York and Chicago can be difficult cities to navigate by car. It’s often faster and easier (once one understands the systems) to use public transportation. However, easy access to that information can sometimes be difficult.
Transitchicago.com is useful for finding door to door public transportation directions. However, if you’re trying to find a restaurant, read reviews, see what’s close, and then want directions to that restaurant, a search engine is the more logical choice to find that information.
Hopefully, we can look forward to finding public transportation directions on maps.google.com sometime in the near future. It would be an extremely useful tool.
Hijacking Google Local Listings
SEO Roundtable has a post on hijacking Google Local Listings.
Hijacking local business listings isn’t new news, I wrote about hijacking Yahoo Local listings in December 2005.
This is the first case where I’ve heard of Google Listings being hijacked; however.
The search engines need to do a better job of letting local businesses control their data.
Google AdWords
Seminars for Success
Learn about Google AdWords from experts hand selected by Google.
These seminars will educate advertisers on the creation and management of successful AdWords campaigns.
Upcoming Seminars:
Phoenix AdWords Seminars April 14th & 15th
Chicago AdWords Seminars April 29th & 30th
Vancouver AdWords Seminars May 5th & 6th
Denver AdWords Seminars May 18th & 19th
Minneapolis AdWords Seminars May 26th & 27th
Learn More about the Seminars:
Seminar Information
Official Google Seminar Page
Suggest a new city
Brad Geddes
Leslie Clark
Other Memberships
