Part 1: Understanding Search Engines

Lesson (1): Classification of Search Engines
Lesson (2): Crawler-Based Search Engines
Lesson (3): Human-Powered Search Engines
Lesson (4): Pay-for-Performance Search Engines
Lesson (5): META-Engines
Lesson (6): Who Feeds Who - Search Engine Relationships
Lesson (7): How Search Engines Rank Pages
Selected Reading List
Quiz (1): Understanding Search Engines


Lesson (1): Classification of Search Engines
The term "search engine" (SE) is often misused to describe both directories and pure search engines. In fact, they are not the same; the difference lies in how result listings are generated.
There are four major search engine types you should know about. They are:
crawler-based (traditional, common) search engines;
directories (mostly human-edited catalogs);
hybrid engines (META engines and those using other engines' results);
pay-per-performance and paid inclusion engines.
Crawler-based SEs, also referred to as spiders or Web crawlers, use special software to automatically and regularly visit websites to create and supplement their giant Web page repositories.
This software is referred to as a "bot", "robot", "spider", or "crawler". All these terms denote the same concept. These programs run on the search engines. They browse pages that already exist in their repositories, and find your site by following links from those pages. Alternatively, after you have submitted pages to a search engine, these pages are queued for scanning by a spider; it finds your page by looking through the lists of pages pending review in this queue.
After a spider has found a page to scan, it retrieves this page via HTTP (like any ordinary Web surfer who types an URL into a browser's address field and presses "enter"). Just like any human visitor, the crawling software leaves a record on your server about its visit. Therefore, it's possible to know from your server log when a search engine has dropped in on your online estate.
Your Web server returns the HTML source code of your page to the spider. The spider then reads it (this process is referred to as "crawling" or "spidering") and this is where the difference begins between a human visitor and crawling software.
While a human visitor can appreciate the quality graphics and impressive Flash animation you've loaded onto your page, a spider won't. A human visitor does not normally read the META tags, a spider can. Only seasoned users might be curious enough to read the code of the page when seeking additional information about the Web page. A human visitor will first notice the largest and most attractive text on the page. A spider, on the other hand, will give more value to text that's closest to the beginning and end of the page, and the text wrapped in links.
Perhaps you've spent a fortune creating a killer website designed to immediately captivate your visitors and gain their admiration. You've even embedded lots of quality Flash animation and JavaScript tricks. Yet, a search engine spider is a robot which only sees that there are some images on the page and some code embedded into the "