Search Engines

It’s hard to imagine now, but in the very early days of the web, it was technically possible to view every website. But the web exploded in popularity and thousands, then millions of new websites were created. People quickly realized that they needed a way to search for information. With the demand growing, it was only natural that solutions would soon follow, and they soon arrived in the form of the first search engines.

“All text” Search Engines

The first “all text” based search engines were WebCrawler and Lycos, both released in 1994. These search engines “crawled” the web, taking snapshots of all of the text on each website to build an index that could be used for search queries.

Yahoo

At around the same time, the founders of Yahoo were building a directory that contained websites they liked, split out into individual categories. This “hand-picked” selection proved to be very popular with early web users, and when Yahoo added search functionality to the directory, it quickly became one of the most highly trafficked sites on the entire web.

But these early search engines had problems.

The “all text” search engines often provided poor results. They were returning queries based on the text on a web page, so it was quite easy for people to “keyword stuff” the pages, adding irrelevant words to the page to try and trick the search engine and gain more traffic. While a curated search directory like Yahoo may have worked in the earlier days of the web, it simply couldn’t scale up with the growing demands of users as the web expanded massively in size and scope.

Google

This is where Google entered the picture.

Founded in late 1998 by Sergey Brin and Larry Page, Google’s search engine used a new algorithm called Page Rank (PR) to provide users with significantly better search results.

While most search engines were simply returning results based on the text on a web page, Google was also taking into account how many pages were linking to each web page, and giving each page a “rank” based on the perceived quality of these links. A page with a few links from highly ranked pages would be considered higher quality than a site with thousands of links from low-quality sources so that it would display higher up on the search results page.

The improved algorithm removed many spam problems and gave users much more accurate results, and users soon flocked to Google. Google quickly came to dominate the search engine market, a position they continue to hold on to today.