Search Engines
For most of us when we think “search engine” we think Google.
After all, Google gets over 63,000 searches per SECOND.
And for most people, “the internet” is what comes up when you search things in Google.
However, these results that you see are only what Google has found & wants to include.
Most of the internet is NOT in Google. Why? Because:
- Google doesn’t condone it (ex: websites doing illegal things)
- Google doesn’t like it (ex: a website that Google sees as providing no value)
- The website doesn’t want to be there (so it tells the crawler “hey don’t include me”)
This means that these websites are “un-searchable”.
They cannot be found using a search engine. You have to know the exact website address to access them.
Fun fact: It is estimated that 96-99% of the entire internet is “Deep Web”
In this article we are going to talk about what a search engine actually is, how it works, and we will even cover the most popular search engine types & search engine companies.
What is a Search Engine?
Search Engine Definition: An internet search engine, or web search engine, is a web-based software designed to conduct & carry-out an internet search (performed by a user) on the “World Wide Web” based on a set of rules (called an algorithm) that is intended to bring back a page of search results (called a SERP, for Search Engine Results Page) that the algorithm “believes” will help answer the users search query (aka question).
Wow that was a mouthful.
Basically, it’s a program that brings back a bunch of websites it thinks will be the absolute best fit to satisfy the reason you typed in your search query in the first place.
So, when you Google “How to make a grilled cheese sandwich” the search engine is going to look through it’s entire index (like the digital version of a library) and give you a bunch of recipes, videos, pictures, etc.
It doesn’t know EXACTLY what you want to know about making a grilled cheese sandwich, so it will show you what it “believes” (they don’t actually have beliefs) will be a good variety of resources that will help you solve your problem.
How do Search Engines Work?
Search engines are competitors.
Yahoo is its own company just like Google is its own company and just like Bing is its own company – they don’t come together to decide how a search engine should work.
They compete.
(although they do come together to make rules about what they want us to do for them so it’s easier on their crawlers to profit off our info…. just speed read the homepage of schema.org if you don’t believe that)
So, each search engine has a “secret formula” (extremely complex mathematical calculations called an algorithm) that the company has created that it thinks will deliver the best search results.
Why? Because their customers are searchers.
If they provide good results every time, you will keep coming back.
I stopped using Yahoo a long time ago because it sucks. When I search something I never get what I really need.
To make things even more complex – these companies are always innovating. They update their algorithms constantly to continue to improve the results, and so we (the general public) never really know exactly how it works.
An Example of How A Search Engine Works
Let me give you an example to illustrate how a search engine works, but also give you a little diatribe about why the thing that powers my entire agency (it is named SERP Co, after all) is actually one giant extortion machine.
Disclaimer: This is an educational rant, but I will try to keep it short.
Search engines crawl around the World Wide Web by following hyperlinks (aka “links”).
A link is that clickable thing that takes you somewhere else. Here is an example of a link to SERP Co.
As they follow links they read everything they can from the page.
They read the HTML, the CSS, Javascript, etc.
So, when you see this:
They see this:
As they are reading the page they begin to form an “understanding” about what the website as a whole (and the specific pages on it) are about.
When they crawl around & read our website, they can understand they we are a digital marketing company (it says it prominently in the HTML of our home page title tag, and also as the main heading (called an
After the search engine forms an understanding about your website it will “file it away” in a large database, called an Index, that acts like a digital library of all the websites on the internet (that it has crawled).
So, when you perform a search on a search engine, the search engine does not go out and quickly crawl the entire web for answers, it simply looks up in its own database to find the closest match to your search query.
Search Engine Algorithms
So, you want to know how search engines work, huh?
- Recommended Reading: How Does Google Search Work
How A Traditional Search Engine Algorithm Works
Let’s say you search for a “digital marketing company in los angeles”, the search engine will look through it’s already formed index and find all the websites it has previously crawled, categorized and included in its index that are relevant to that search you performed.
Then, it will return ALL the results.
However, each page of search results only has room for 10 organic listings.
So, the search engine must decide “who should I show 1st, 2nd, 3rd, etc.”
It does this based on a large variety of search engine ranking factors that help the search engine determine who is the “best” or “most relevant” result.
This is really where the algorithm comes into play – the set of rules that determine how a search engine understands & prioritizes your website.
Non Traditional Search Engines
Colloquially when we say “search engine” most people think of text based search, like Google.com (a crawler based search engine).
That would be the example we just gave above.
However, with the continual progression of technology we have seen many more types of search engines (and thus search engine algorithms) arise.
Basically, anything you “search on” is a search engine – but it does not necessarily have the same algorithm as the text based search engine.
Let’s get into some examples.
Examples of Search Engine Algorithms
Crawler Based Search Engines
Crawler-based search engines use software algorithm / programs to find & categorize websites & webpages.
They “crawl” (kind of like the digital version of clicking on a link) these links from page to page, and website to website, and begin to make a giant “web” of relationships and categorizations.
Hence why these crawlers are called “crawlers”, or “spiders”, or “bots”.
Web crawlers a websites, download them to their “index” & analyze the information so they can appropriately categorize them (and then display / serve them to a searcher looking for that kind of information).
When a user searches for something, the search engine checks its “index” for relevant websites that it can provide the user.
The end result of all these “spiders crawling around” is a gigantic, interconnected “web” which we call – the World Wide Web, aka internet.
Examples of Crawler Based Search Engines
- Yahoo
- Bing
- Vivisimo
- Dogpile
- Altavista
- Overture
- HotBot
- AlltheWeb
- AOL
- Ask.com
- Blekko
- Duck Duck Go
- Excite
- ExactSeek
- ePilot
- ICQ
- iWon
- Jayde
- Kanoodle
- Kosmix
- LookSmart
- Lycos
- Netscape
- Open Directory
- 7Search
- Teoma
- WiseNut
- Yebol
Directory Search Engines
A directory type of search engine has humans decide which categories the websites will belong to.
People manually add & organize the websites into categories, check the sites for quality, determine the ranking, etc. based on a pre-defined set of evaluations / rules.
Meta Search Engines
Meta search engines are like a search engine of search engine results – they take the search results from the other search engines and then combine these results into one aggregated results page.
Sound a little ridiculous? I think so too, for the most part.
Examples of Meta Search Engines
- Answers.com
- Clusty
- DeeperWeb
- Dogpile
- Excite
- Harvester42
- HotBot
- Info.com
- Ixquick
- Kayak
- LeapFish
- Metacrawler
- Mobissimo
- Turbo10
- WebCrawler
- yolink
Hybrid Search Engines
Hybrid search engines are a mix of directory based search engines & crawler based search engines.
Map Search Engines
Map search engines help users find business locations on a map.
They usually combine user submitted data, algorithmic prioritization, street views from car video cameras, satellites, user reviews, etc.
Examples of Map Search Engines
- Google Maps
- Apple Maps
- Bing Maps
- Waze
- MapQuest
- Maps.me
- Citymapper
- Here.com
Multimedia / Image Search Engines
Image / Multimedia search allows multimedia to be searchable by using search queries in multiple data types.
Because computers are still learning how to understand what an image is, we can add other layers of data to our multimedia to help it understand, categorize, etc.
This additional data, called Metadata, can be things like text descriptions, links to other related content, or even additional “code markup”, from a set of descriptive standards, that tell the machines what the image/video/etc. actually is.
Read more (and bore yourself to death) at: schema.org
Examples of Multimedia Search Engines
- Bing Videos
- blinkx
- FindSounds
- Google Images
- Google Video
- Picsearch
- Podscope
- ScienceStage
- Songza
- TV Genius
- Veveo
- TinEye
- Yahoo! Video
- YouTube
Voice Search Engines
Voice search allows us to speak to devices and get feedback in return.
You are probably familiar with this already.
Examples of Voice Search Engines
- Amazon Alexa
- Google Home
- Apple’s Siri
Although, these still need a bit of work… 😂
Shopping Search Engines
Shopping search engines help consumers find products.
The most obvious example is Amazon.
They are based off of their own algorithms, product uploads, user reviews, etc.
Examples of Shopping Search Engines
- Amazon
- Ali Express
- Bing Shopping
- Google Shopping
- Shopping.com
- ShopWiki
- Shopzilla
A List of Popular Search Engines
Google is a multinational tech company that specializes in Internet-related products & services, including: online advertising, search engine technologies, cloud computing, software & hardware.
Google is the largest search engine on the internet, with over 90% market share.
YouTube
YouTube is an video-sharing platform, originally founded by 3 former PayPal employees – Steve Chen, Jawed Karim & Chad Hurley in February 2005.
YouTube has over 1.5 billion logged in users every month, streaming 1,000,000,000 (billion) hours of video every day by users.
YouTube is now owned by Google, and is the 2nd largest search on the internet.
Bing
In 2009, Bing replaced “MSN Search” as Microsoft’s primary search engine, and their attempt to gain market share back from Google.
Bing ships on many PCs and is the default search engine for all machines running on the Windows OS
Yahoo
Founded in 1994, by Jerry Yang, and currently owned by Verizon Yahoo was one of the earliest search engines on the internet.
It’s popularity has continued to decline.
Amazon
Amazon is an online shopping platform, founded by Jeff Bezos in 1995.
Amazon is a leading company in data & is considered one of the powerhouse companies fighting for control of search (having great success through their Amazon Alexa products).
More than half of ALL online shopping searches start on Amazon.
Baidu
Baidu is a Chinese search engine / internet technology company, headquartered in Shanghai.
It’s basically the Chinese equivalent of Google.
Ask
Ask.com, originally called “Ask Jeeves”, was created as a website search-engine that could get answered to everyday questions, and return those answers back in natural language – as if you were actually asking someone an answer to your question.
Facebook is an online social media & social networking company.
As of June 2019, there are more than 2,410,000,000 world-wide active monthly users of Facebook.
AOL
Also acquired by Verizon, AOL is one of the original web based services.
Who still uses AOL as a search engine? I don’t personally know anyone – but my parents still have @aol.com email addresses!
Duck Duck Go
DuckDuckGo is an internet search engine that emphasizes protecting searchers’ privacy and distinguishes itself by showing all users the same search results for a given search term – emphasizing returning the best search results, rather than the most results, from over 400 individual sources, including crowdsourced sites such as Wikipedia, and other search engines like Bing, Yahoo!, and Yandex. As of August 2019, it had 44,196,092 daily direct searches on average. (source)
How Do Search Engines Make Money?
Search engines make money by serving advertisements in line with the “organic” results.
Companies can pay the search engine to “show up” on Page 1 by paying money to be there.
They also make money by putting affiliate links inside of search results.
Final Thoughts
Search engines are where (almost) everywhere goes now to get answers to their questions, recommendations of businesses, shop for products, learn new skills, etc.
They run our society & our markets.
If you are a business owner you need to have first page real estate for the products & services you sell, if you don’t you will be missing out on an astronomical amount of new business.