Recent Posts

Q&A: How can I tell if Google’s Ads Bot can spider my landing page?

Posted: September 18, 2008

Google’s adsbot will spider your landing page to determine your landing page’s impact on your quality score. If Google’s ads bot cannot properly spider your page, you may see very high minimum bids and low exposure.

Google webmaster tools has a tool where you can see if Google’s adsbot spider will index specific pages on your site. Here’s a step-by-step guide to checking if specific pages can be spidered.

Create a Google webmaster tools account.
Verify your domain.
Once your domain is verified, login to your account and navigate to: Google webmaster Tools > Tools > Analyze robots.txt
Either view your robots.txt or enter a new robots.txt into the first text box
- Please note, if you use meta tags or nofollow tags to control what pages Google can spider, this tool will not show what pages are being disallowed by methods other than the robots.txt file.
At the bottom of the page is a section labeled ‘Choose user-agents’. Select Adsbot-Google
Hit the ‘check’ button and view the results.

Click picture to see full view.

Here you can see where to navigate to view the robots.txt tool. Next, the first box is your robots.txt file (and you can change what is being tested here), the URLs you wish to check, and finally the results. In the graphic below I entered Google.com to show why you need to verify your site. You cannot see information outside of your domain.

Additional note: If you have a global disallow, adsbot-Google will ignore your entire robots.txt file (but Google’s other bots will follow your robots.txt instructions).

Article resource links:

Share and Enjoy:

Related Information:

« Q&A: I use location targeting but I don’t live in the areas where I advertise. How can I see my ads?

The Best Entertainment Distractions that Easily fit into your Computer Bag »

Written by Brad Geddes aka eWhisper · Filed Under Blog, Google AdWords

Comments

One Response to “Q&A: How can I tell if Google’s Ads Bot can spider my landing page?”

James on September 19th, 2008 10:34 am

This is quite interesting point. For one of the projects, I had created separate landing pages for separate group of keywords. All landing pages were almost same in terms of design layout and content. One can say that almost 90% of landing pages were same except the headlines and title tags.

To avoid issues like duplicate content I had excluded those landing pages in robots.txt file. As of now I don’t see any high minimum bids for any of the keywords. But not sure what would happen in future. Should we allow robots to index landing pages so we don’t see spike in minimum bids? Or should we exclude them in robots.txt file to avoid duplicate content issues?

Got something to say?