What is a search engine?

A search engine is a system or software that generates hyperlinks to websites in response to a query inputed by a user. 

A user looking for a particular web page types in a word or phrase into the search bar of the engine. The search engine then lists out relevant web pages related to the users intent. 

A search engine is basically an index of websites or webpages.

It is a database of information that has been collected and stored, and can be searched. 

Several search engines exist today, but Google is the dominant one. 

A Brief History of Search Engines

At first, Search engines had a process of manual indexing of web pages by hand. When a new site was created, it had to be submitted to the index to be listed, but as more websites and webservers came online, this method of indexing became inefficient. 

The switch to an automatic system for indexing new webpages, websites, and webservers was necessary. 

Modern search engines now use web crawlers (also known as spiders) to find and index webpages faster and more efficient, following the ranking rules of the Engine to find and deliver pages that would be relevant to the users. 

The first search engine was Archie, created in 1990 by a McGill University student, Alan Emtage. Archie used the FTP (File Transfer Protocol).

The first web search engines that used web crawlers emerged in 1993, with Wandex, ALIWEB and Jumpstation as pioneers, with Jumpstation being the first to actually perform the necessary functions of a search engine, that is, crawling, indexing and searching, though it was limited to only the titles and headings of webpages. 

By 1994, WebCrawler was created. It allowed users to search for any word or phrase instead of just headings. This created a new standard followed by search engines since. 

Now if you want to search for something on the web, you don't need to remember the exact webpage title, but you can look for a word or phrase and a list of relevant webpages would be shown. 

Google was launched in 1998, and rose to prominence in the 2000s especially with its method of ranking pages, PageRank, which defined the relevance of pages based on how many other webpages linked to them, with the believe that a webpage that is very useful or of high quality would have a large number of webpages linking to it. Google's ranking algorithm has evolved since then, with various rules to determine how relevant a webpage is to a user's query. 

This makes Google one of the most popular search engines as it delivers quality hyperlinks that are very relevant to a users search intent.

Process of a Search Engine 

Modern search engines now use a process of 

• Crawling
• Indexing
• Searching

Let us explain each of these three terms

Crawling

A web robot, called a web crawler scans the web by automatically following links between pages. 

The links create a path for the crawler to move from one page to another, and from one website to another, and this new webpages are discovered. The discovered webpages are then sent over to the engine to be indexed. 

Indexing 

After crawling, the webpages are then associated with keywords found on the page and are listed based on relevance. The exact method of indexing varies by search engines, but majorly, the discovered pages are displayed according to the search engines algorithm and made available for a user's search. The webpages are cached in between visits by the crawlers, to quickly render them to users.This means that if a webpage has been updated before the crawler visits the webpage again,  a user who visits would be served the cached version of that page. 

Searching

 Now, a user comes to the search engine and types in a word, or phrase and clicks search. The search engine compares the keywords typed in by the user with a list of webpages that have those words and associated words in their headings, titles or content. It then orders and arranged them based on relevance, using it's algorithm to order which webpage gets listed first. Note that there would be thousands if not millions of different webpages that contain a particular keyword, so the search engine ranks these pages based on quality and relation to user's search intent. 

As mentioned above, the exact method of ranking varies from search engine to search engine. The user also has the option of filtering the results as needed. 

 For example a user types the words "car deals" in a search engine. There are millions of websites that have the words "car", "deal", "car deals" in their content. The search engine then selects top relevant webpages and places them on the first page of the index, the second page, third page and so on. The user can filter these search results based on say, date or location, and the search engine adjust the weighting of the index accordingly, showing results for a specific date range (for example pages created or updated within the last 3 months) or/and a specific location (example car deals near me, or car deals in Nigeria).

Search Bias 

Because of economic, political or social factors, search engines may not follow their algorithms to the core. A search engine may favour websites that advertise with them, putting them in the front pages of organic search. The search engine may also exclude certain webpage from its index due to censorship. For example,  China censors it's internet and would exclude certain pages from search results. Webpages that  go against certain social norms may lose ranking to favour pages that conform to certain societal norms.

 Some specialized search engines, for example religious search engines may filter out results that go against their faith and doctrines. 

Other search engines like Kiddle, modify results for children, filtering out violent or pornographic materials. 

Classifications of Search Engines 

We can group search engines into three:

 

1. Engines powered by robots or web crawlers that crawl the web and creates a database of webpage to be served to users
2. Human powered search engines or directories in which webpages are submitted by humans to the search engine, and/or are reviewed by humans before being listed on the index. 
3. Hybrid search engines that use a combination of the two listed above.

In Summary 

• Search engines are applications that search for, store and retrieve data based on an algorithm or criteria.
• The basic processes of Search engines are Crawling or Discovering the webpage, Indexing it, and rendering it Searchable for users. 
• Different search engines may produce different search results for the same queries because each search engine has its own method of indexing and ranking pages.

Things to note

• When you use a search engine, you are not actually searching the entire web, rather you are searching the index or database of that search engine. 
• Many search engines cache webpage, so when you visit a page via a search engine, depending on when last that webpage was crawled, you may be accessing content that is outdated.

 If you enjoyed the article,  leave a comment below, and if you have any questions, ask and I'll be more than eager to answer them. 

Till my next post!