Search Engine Workshops - Intensive hands-on search engine marketing training during a 2-, 3-, or 5-day workshop Search Engine Workshops Home PageRegister for onsite search engine marketing workshopsContact us for more information about our onsite search engine marketing workshopsLost? Find it in our site map.

Search Engine Workshops . . . the premiere onsite search engine marketing workshops in the world! Click here to go back to the index of SEO articles.

Topics covered at our search engine marketing workshopsDates for future search engine marketing workshopsRegister for onsite search engine marketing workshopsArticles on search engine marketingTestimonials from our search engine marketing workshop guestsSearch Engine Workshops PartnersCommonly asked questions about search engine marketing and our workshopsLinks to useful sites and search engine marketing informationAbout Search Engine Workshops



Spider Facts - 6 Tips About How Search Engine Spiders Work

By John Alexander

Over the years there has been a lot of confusion about SEO and how search engines work. One of the things that comes up all the time is the subject of search  engine spiders or as they are sometimes called search engine robots.

When a search engine robot visits your Web page, it is important to understand a few things if you are hoping to get your Web page indexed. Search Engine Robots are simply software programs that come out to your Web page, capture certain data and then return that data to the search engine from where it came.

Do Search Engines "see" your entire Web site as a whole collection of pages or do you think they only see the whole world as a collection of individual pages?

Actually, the truth is neither "if you are thinking" based on how a search engine robot sees the world.

Think about this now. 

A search engine robot does not see a  world of pages or a world of Web sites. 

A search engine sees its world made up only of URLs or (Uniform Resource Locators.) 

This is one reason why search engine spiders cannot some times find their way out of what is called a Spider Trap. For example in some dynamic sites, you will  often have variable strings at the end of a URL that are session identifiers and  while the URL may be unique, the search engine does not realize it may be on  exactly the same page over and over in a loop and may not be able to get out.

Do search engine spiders help determine what your rankings are? This is another wrong idea that some people have. The search engine spider has one sole job and  that is just to crawl content capture certain information and links and take it  back to the main search engine.

How smart are search engine spiders? 

The answer is they just are not very smart at all. They cannot think. 

In fact most search engine spiders are updated very rarely with some powerful new feature. The reason why is because as most technology advances, the demands on a SE robot remain the same basically. They still only do exactly the same job they have always done. There are some spiders that are a little better than others at being more thorough and crawling more complexly. 

But all things considered, why keep rebuilding something if what you have now works great?

Tips you should be aware of with spiders, crawlers and SEO and search engine  optimization:

1. A spider is designed to crawl the Web. The smartest way to keep spiders  consistently coming back to visit, is to focus on content freshness. Add new  articles, pages, FAQs, good useful information to your Web site on a consistent  basis - instead of doing major updates. Instead of adding 20 new articles all at  once, try adding only 1 article a day (everyday) and watch what happens.

2. You do not need to include a robots revisit tag to your HTML. It does not work and the robot will come back more consistently with you adding fresh content, you may see it stay longer and actually dig deeper into your Web site.

3. Do you want to know one reason why Blogs are so busy and actively crawled by  the spiders? It's because Blogs are often very fresh content sometimes being  updated several times per day.

4. Remember that spiders and crawlers do NOT need to be submitted to for them to  find you. There are actually good reasons not to use submission software but you  are wiser to link to a new site from a Blog or another site that you know is already being indexed. Remember that you will still need to submit to directories though because there, you are dealing with human editors and not robots.

5. Make sure your Web site has a nice Site Map and then have at least one link to your site map on every page. Matt Cutts has said for example that this is one of  the things that people most often forget. Remember, you have no idea whether a  search engine spider will come in on your home page, or on your about us page or  maybe even a policy page. But regardless of which way it comes in, it will be able to crawl ALL of your pages if you've created a good site map.

6. Do you think that it is best to get into Google immediately as quickly as you can?

Think about it for a minute. With Google, off page factors are VERY important.  

Suppose Google's search engine robot visits you right away. But unfortunately, none of the other search engines have visited your page so even if you have some  good links, nobody really knows it. Perhaps you need to think of a reverse  approach and first submit to a few free or paid directories. Then you could begin
getting your page indexed by some of the other spiders first, then finally when  Google comes in it sees some nice off page factors and link reputation for which  it will reward you for.

Grab hundreds and hundreds of SEO tips - FREE. 

One free tip a day is quick to read, easy to absorb and it's amazing how many folks have said they have improved their results just reading these Free SEO Tip of the Day mailings! 

They are free and you can unsubscribe or subscribe whenever you please.
Would you like a Free SEO Tip of the Day?


About John Alexander
John Alexander is Co-director of Training at Search Engine Workshops offering live, SEO Workshops with his partner SEO educator Robin Nobles, author of the very first comprehensive online search engine marketing courses. John is author of
an e-book called Wordtracker Magic and has taught SEO skills to people from 87 different countries world wide. John's articles can be read in publications like REALTOR Magazine, Search Engine Guide, WEBpro News and many others.  

Click here to go back to the index of search engine marketing articles

Creative Commons License
This work is licensed
under a Creative Commons License.

We'll keep you advised on all of our future training dates the moment things are firmed up.
Simply enter your name and e-mail in the box below.

See what other dates and locations are still available

Sign up to receive our NEWSLETTER, PLUS free SEO strategies 
and tips and notification of future SEO workshops by e-mail.


Your Name:

Your E-mail Address:

John Alexander



Click here to go back to the index of SEO articles

Search Engine Marketing Workshops | Site Map | Affiliate Program

Privacy Policy | Terms of Service