Spider
Facts - 6 Tips About How Search Engine Spiders Work
By John Alexander
Over the years there has been a lot of confusion about SEO
and how search engines work. One of the things that comes up
all the time is the subject of search engine spiders or
as they are sometimes called search engine robots.
When a search engine robot visits your Web page, it is
important to understand a few things if you are hoping to get
your Web page indexed. Search Engine Robots are simply
software programs that come out to your Web page, capture
certain data and then return that data to the search engine
from where it came.
Do Search Engines "see" your entire Web site as a
whole collection of pages or do you think they only see the
whole world as a collection of individual pages?
Actually, the truth is neither "if you are
thinking" based on how a search engine robot sees the
world.
Think about this now.
A search engine robot does not see a world of pages or a
world of Web sites.
A search engine sees its world made up only of URLs or
(Uniform Resource Locators.)
This is one reason why search engine spiders cannot some times
find their way out of what is called a Spider Trap. For
example in some dynamic sites, you will often have
variable strings at the end of a URL that are session
identifiers and while the URL may be unique, the search
engine does not realize it may be on exactly the same
page over and over in a loop and may not be able to get out.
Do search engine spiders help determine what your rankings
are? This is another wrong idea that some people have. The
search engine spider has one sole job and that is just
to crawl content capture certain information and links and
take it back to the main search engine.
How smart are search engine spiders?
The answer is they just are not very smart at all. They
cannot think.
In fact most search engine spiders are updated very rarely
with some powerful new feature. The reason why is because as
most technology advances, the demands on a SE robot remain the
same basically. They still only do exactly the same job they
have always done. There are some spiders that are a little
better than others at being more thorough and crawling more
complexly.
But all things considered, why keep rebuilding something if
what you have now works great?
Tips you should be aware of with spiders, crawlers and SEO
and search engine optimization:
1. A spider is designed to crawl the Web. The smartest way
to keep spiders consistently coming back to visit, is to
focus on content freshness. Add new articles, pages,
FAQs, good useful information to your Web site on a consistent
basis - instead of doing major updates. Instead of adding 20
new articles all at once, try adding only 1 article a
day (everyday) and watch what happens.
2. You do not need to include a robots revisit tag to your
HTML. It does not work and the robot will come back more
consistently with you adding fresh content, you may see it
stay longer and actually dig deeper into your Web site.
3. Do you want to know one reason why Blogs are so busy and
actively crawled by the spiders? It's because Blogs are
often very fresh content sometimes being updated several
times per day.
4. Remember that spiders and crawlers do NOT need to be
submitted to for them to find you. There are actually
good reasons not to use submission software but you are
wiser to link to a new site from a Blog or another site that
you know is already being indexed. Remember that you will
still need to submit to directories though because there, you
are dealing with human editors and not robots.
5. Make sure your Web site has a nice Site Map and then
have at least one link to your site map on every page. Matt
Cutts has said for example that this is one of the
things that people most often forget. Remember, you have no
idea whether a search engine spider will come in on your
home page, or on your about us page or maybe even a
policy page. But regardless of which way it comes in, it will
be able to crawl ALL of your pages if you've created a
good site map.
6. Do you think that it is best to get into Google
immediately as quickly as you can?
Think about it for a minute. With Google, off page factors
are VERY important.
Suppose Google's search engine robot visits you right away.
But unfortunately, none of the other search engines have
visited your page so even if you have some good links,
nobody really knows it. Perhaps you need to think of a reverse
approach and first submit to a few free or paid directories.
Then you could begin
getting your page indexed by some of the other spiders first,
then finally when Google comes in it sees some nice off
page factors and link reputation for which it will
reward you for.
Grab hundreds and hundreds of SEO tips -
FREE.
One free tip a day is quick to read, easy to absorb and it's
amazing how many folks have said they have improved their
results just reading these Free SEO Tip of the Day mailings!
They are free and you can unsubscribe or subscribe whenever
you please.
Would
you like a Free SEO Tip of the Day?
About John Alexander
John Alexander is Co-director of Training at Search
Engine Workshops offering live, SEO Workshops with his
partner SEO educator Robin Nobles, author of the very first
comprehensive online search engine marketing courses. John is author of an
e-book called Wordtracker
Magic and has
taught SEO skills to people from 87 different countries world
wide.
John's articles can be read in publications like REALTOR
Magazine, Search
Engine Guide, WEBpro
News and many others.
