The reading level for this article is All Levels
With so many different Search Engines online today you need to take time to understand how their spiders work so that you can truly optimize your domain in order to get the best out of your web presence. In this article you will learn how some of the more popular Search Engine spiders actually work.
One of the smaller, popular Search Engines is InfoSeek. This Search Engine only indexes the first 200 words on your web page. This is why it is very important to make sure that you have good meta tags on your site. These meta tags should list the most important information first because this is the information that will be used as a description for your site. You should realize that most meta tags can only contain about 200 characters of text. However, you can place 1,000 characters within your keywords meta tag.
You really should keep these simple rules about meta tags in mind for all of the Search Engines that you are working with. At the same time though, you should also realize that a lot of Search Engines do not even look at meta tags. This is why it is so important to have the same information in your web site’s body as is in your meta tags. Of course you cannot enter a list of keywords without this being detrimental to your site’s content.
The next Search Engine worth looking at is Alta Vista. Its spider is named Scooter and it may take three months before it fully indexes your site. This is different from the average spider which only takes between 6 and 8 weeks to index your entire site. However, normally Scooter will only spider between 2 and 10 pages of your web site each week. This is why it takes it longer but at the same time the longer you have your web site up the better it will do on Alta Vista.
While Excite was once a search powerhouse it is no longer the provider of AOL and Netscape search, and thus it is a lot less important than it once was. Excite uses a very complex algorithm to determine your site’s keyword relevance. The way it works is that once it indexes your pages it attempts to summarize them by selecting what it thinks are the most relevant sentences. In order to keep this up to date, Excite will then continue to review your page approximately every two weeks.
Excite also owns WebCrawler but WebCrawled is operated independently of Excite. If you are able to get listed with WebCrawler, then you should do your best to keep your listing because this is the easiest Search Engine to get listed with. However, you should know that its submission process is tough, but not impossible. This is because it uses hit-and-miss standards combined with sporadic indexing methods.
Lycos has fully integrated DMOZ (the Open Directory Project aka ODP) into its mainstream results pages. They also use search results from AllTheWeb. It is important to do well on Lycos though because this is the site that runs click-throughs to HotBot, which is their sister site. All of this makes it very hard to understand Lycos. Generally though your site is going to eventually be indexed by Lycos if you are indexed by DMOZ and AllTheWeb.
Of course, the biggest Search Engine out there today is Google. This Search Engine uses a page ranking system that is often referred to as PR. It is their central basis for their index. At one time it was almost impossible to manipulate PR but webmasters soon discovered that the more links they had the higher their PR and the better their Google ranking. This Search Engine’s spider is regularly sent out whenever Google re-indexes its own database.
Another important Search Engine to get listed in is MSN. If you can get a good ranking in Google, MSN and Yahoo! your site will do well because these 3 Search Engines provide the majority of Search Engine traffic. Usually MSN will be the first Search Engines to index your site. It will also list the most pages the fastest.
While nobody can tell you precisely when a specific Search Engine will index you, it is a good idea to check weekly. One thing that you should never do is re-submit your site more often than every 2 months. Re-submitting your site more often may keep you from getting indexed at all.
Now that you have a basic understanding of how some of the numerous Search Engines online today actually work, you need to begin working to optimize your domain so that you can make the best of your web presence. Hopefully this article has started you on your way to learning how to use these Search Engines to your benefit.
