Have you ever wondered how far down into your site the search
engine spiders are going? Are the pay inclusion spiders coming at regular
intervals like they’re supposed to?
Wouldn’t it be great to see which spiders are visiting your
site and which pages they’re crawling . . . at a glance?
Robot Manager
Pro is undoubtedly one of the most helpful software programs I’ve
seen in a long time. It actually serves two purposes:
- It will create robots.txt files for you, and
- It will monitor spider activity to your Web sites.
What is a Robots.txt File?
A robots.txt file is simply a text file that
“disallows” (or keeps spiders out) of certain pages or areas of your
Web site. For example, if you have a section of your site that’s devoted
to your personnel, you can keep the spiders from finding and crawling that
entire section by including it in a robots.txt file.
If you’ve ever tried to create a robots.txt file by hand,
you know that it’s nothing short of tedious, and it’s so easy to make
a mistake. So having a program create the file for you automatically is ideal.
However, what’s really exciting to me about Robot Manager
Pro is how it monitors spider traffic. But first, let’s look at the
software as a whole.
At Look at Robot Manager Pro
The program is simple to use. On the left-hand side of the
screen, you’ll see numbered areas. On the right-hand side of the screen,
you’ll see a Quick Help section that explains how to use that portion of
the program.
What you’re doing in steps #1-3 is creating your robots.txt
file. You’ll tell the program which areas or pages that you want to keep
the spiders from crawling, and the software will create the robots.txt file for
you and upload it to the root directory of your Web site.
Keep in mind that you don’t have to create a robots.txt
file if you don’t want or need to. You can simply use the spider
monitoring portion by itself.
Now Let’s Get to the “Fun” Stuff
As I stated previously, the most exciting part of the program to
me is the way it monitors spider traffic.
The first time you download your log files to analyze your
traffic, it won’t be quite as interesting as it gets each additional time
you use the program.
Every time you use the program, you’ll open the initial
file you created, hit “refresh” to download the latest log files, and
let the program update your spider visits.
This screen shot shows a listing of the spiders that have
visited the example site.
If we were to click on one of the spiders, such as Googlebot,
we’d see this screen:
Notice how many pages from this site that Google has spidered,
and this screen shot certainly doesn’t show them all. Google is a deep
crawling engine, and it really can find most of your Web pages if you provide
text links to them.
If we were to click on any of the actual Web pages, we’d
see a screen that tells us when the spider has visited the page.
In this example, look at the number of times that Googlebot
spidered this particular page over the last few months.
Can you see the value in having this type of information?
· If an engine hasn’t found one of your important
pages, maybe you need to add more links to that page from pages that are
getting spidered more frequently.
· When you create new pages, you’ll be able to see
at a glance when the spiders find and crawl those pages.
· If you use pay inclusion, you’ll be able to
monitor whether the spiders are visiting like they’re supposed to.
· You’ll know exactly which pages of your Web
site are being spidered by which search engines, and when.
In Conclusion
If you’re interested in tracking spider visits to your
site, Robot Manager Pro is a “must have” software program for you.
It’s fast and easy to use, and it provides key information about how the
spiders are crawling your Web site.
Visit the Robot Manager Pro Web site at
http://www.websitemanagementtools.com/robot.html
to learn more or to download a free trial version. They also offer a regular
version of Robot Manager, but it doesn’t monitor spider traffic. So, be
sure to click on the Professional Edition on the left tool bar.
Robin Nobles is the Co-Director of Training of Search Engine
Workshops with John Alexander. They teach 2-day beginner, 3-day advanced, and
5-day all-inclusive "hands on" search engine marketing
workshops in locations across the globe. She also teaches
online search engine marketing
courses through http://www.onlinewebtraining.com,
and she’s a member of Wordtracker’s official
question support team. With partner John Alexander, she's co-authored a series
of e-books called, "The Totally
Non-Technical Guides to Having a Successful Web Site." And, they opened a
networking community for search engine marketers called
The Workshop Resource Center for Search Engine
Marketers.
Copyright 2005 Robin Nobles. All rights reserved.
This work is licensed
under a Creative Commons License.
