Search

When you type a search query into Google, do you ever stop to think about how the results you see are delivered to you? It's tempting to think that that those results are mediated by some magical algorithm that is agnostically capable of providing "perfect" search results. The truth is that the algorithms that search engines use are created by people to achieve very specific goals. Moreover, they are constantly in a state of flux, as those individuals try to guarantee that the results are as accurate and on-topic as possible. And, most importantly, all of that tweaking and adjusting isn't being done for altruistic reasons: Google isn't trying to deliver perfect search results for you because it CARES about you. No, Google wants your search results to be as targeted and accurate as possible so that it can display advertising (that is equally targeted) side-by-side next to the search results you seek.

The Basics: How Search Engines Work
Underneath the hood of Google (and every search engine) is a MASSIVE farm of servers and databases that store information about Web pages. The information itself is being constantly gathered by special pieces of software called searchbots (or spiders). Searchbots are single-minded programs that do nothing but scour the Web for data. They crawl from Web site to Web site (following links) and gather up information that is they stored in huge index databases. It's important to now that when we say "information," we really mean that these programs literally download and store the actual text content on Web pages. When then encounter images, they may download and store those actual images to index databases as well.

When you enter a search query into a search engine, it searches these index databases for content that matches your query. Based upon an elaborate (and super-secret) algorithm, the results that are found are analyzed and sorted. The results you see reflect the output of that algorithm.

Caching & Refreshing
Because Google stores the ACTUAL content Web pages in it's database, it's able to show you snippets of content in that search result; in some cases, it even provides a link to a cached version of the site from the database. This can be especially useful if you're looking for a page that has changed or is no longer available.

The content that is stored in the index databases is ephemeral. Since those searchbots are constantly scouring the Web, they are also constantly updating the index data. When they encounter a link that is broken, they begin to remove the data about that site from the index, but this can take some time. Google, for example, maintains redundant servers to store all of that index data. It can take time for all of those servers to be updated when a site changes or disappears.

If you've ever searched for something on Google, followed a link, and then discovered that the page you were sent to doesn't contain your query, you may be a victim of an update lag on the index servers.

Staying out of the Index
Sometimes, there are might be times when you put up content on your site that you don't want Google (or other search engines) to index. The only foolproof way of guaranteeing this is to make your site private and require an account/password to access your content. Even this, however, isn't really foolproof; passwords can be cracked.

Alternatively, you can put up a content and just not link to it (or publicize the link). Note this is generally impossible when you're using software like WordPress. Blogging systems are designed to automatically create links to new content, so it can be very tricky to ensure that a page or post in WordPress is truly unfindable.

It is possible to include a special file on your site called robots.txt that sends information to searchbots about what content can and can't be indexed on a site. Generally, search engines try to obey these rules, but they're not 100% effective.

What Does Advertising Have to Do With It?
Google's (and other search engine's) bread and butter is in online advertising. They serve up ads next to and above search results that are contextually related to the search query you've entered, hoping that you'll click on those ads and make them some money. It is Google's interest to keep your search results as accurate as possible so that you keep coming back and, hopefully, click on ads. In addition, Google is constantly tweaking the layout of the search results page so that you're more likely to click on ads. You may notice that ads seem to be more-and-more embedded in and interlaced with your search results. This isn't accidental; it's Google's attempts to refine and improve search results page so that they're more likely to generate ad revenue.

Search Engine Optimization
Once you've built a site and started to share stuff on it, you may start to become interested in your Google rankings. Assuming you want people to find what you've put on your site, it's in your interests to understand a bit about how these rankings work.

It's impossible to describe exactly the way in which a search engine's algorithm works, thus allowing you to potentially reverse engineer the process and generate high rankings for yourself. Google and other search engines keep a tight wrap on these formulas. It's HUGELY problematic for them if the details leak out as it allows individuals and companies to game the system, thereby corrupting the accuracy of search results (and eating into advertising revenue).

That said, there are basic principles you can follow that can generally help you to improve your rankings. Moreover, learning some of these practices help you to understand more clearly how results are generated:


 * In the olden days of the Web, search engines made use of meta tags hidden in the invisible parts of a Web page to collect information about what a page contained. There are meta tags for title, description, and keywords, each of which can be customized and modified to more accurately reflect a page's content. However, Web developers quickly began abusing these tags, filling them with terms that weren't related to the content but were likely to get searched frequently. Consequently, search results became less accurate, and search engines began ignoring these tags. Filling in title, description, and keywords tags WON'T help your rankings, That said, there are some good reasons to still use the title and description tags. Search engines still use the content of these in the search results page, so having these accurately reflect your page content could increase the chances that someone will click on a link to your site when conducting a search.


 * It's well known that Google (and other search engines) not only gather information about the content of a site but also information about where that site links to -- and what links to it. Part of the search algorithm involves analyzing these linking patters to ascertain that a site is legitimate. As a result, it's in your interests to get links back to your site from other legitimate sites. Unfortunately, this is hard! Generally, you have no control over the content (and links) on other sites. For a while, when blogging was becoming very popular, it was common to leave comments on blogs posts that contained links to sites in order to generate backlinks. However, as with meta tags, comment backlinks became so abused that most blogging software began including code that signaled to search engines they shouldn't follow (and pay attention) to links on comments. As a result, it's very unlikley that you can increase your Google juice by just leaving comments on blogs. HOWEVER, building relationships with other bloggers is a great way to expand your online network. Through these relationships, you might end up with genuine backlinks. Imagine, for example that you stumble across another blogger who is working in a similar area to you. You begin to follow her blog and regularly leave comments, including links back to things you've written that you think she might be interested in. As your relationship with this blogger develops, she decides to write a post that is a direct response to something you've blogged about or commented upon, and, in doing so, she links back to your own blog. This kind of link is a genuine backlink, and it can help your search results. In the end, generating links to yourself these days is as much about regularly sharing your work, developing a network of like-minded online individuals, and cultivating that network.


 * Following basic usability and accessibility rules can help improve your rankings, too.
 * Always provide alt tags for images you put on your site. Google can't "read" an image (yet), but it can read an alt tag that you've written to describe the content of the image.
 * When you create links, pay attention to the actual text that you are creating a link out of. The text should be descriptive of what you're linking to. In other words, don't just say:  click here to learn about nameservers. Instead, construct a meaningful link: You can  learn more about nameservers here.
 * Use semantically accurate tags to markup your post. When you're writing a post in WordPress, for example, you have the option to use the built-in header tags to set-off titles and subtitles. Use these! Google does pay more attention to content that occurs within header tags.

Personalized Search Results
Have you ever done a search of a pizzeria in Google and noticed that all of the results you see are to restaurants near where you're located? That's not because your local pizza joint has the best google rankings in the entire world. Rather, it's because is able to discern (based on your computer's network address) generally where you are geographically located; it tunes your search results to this location so that you find a pizza place down the road instead of, say, on the other side of the country.

Google (and other search engines) are doing more and more to try and adjust search results so that they are more likely to be relevant to you. It starts with geographic adjustments based on your network address. If you're logged into Google (for example, because you use Gmail), your results will be even more customized. Google pays attention to what you've searched before and what you've clicked on before; the results you say are tweaked to reflect these patterns.

If you're interested in getting non-customized results, it is possible. This article contains information about how to set up non-customized results in Chrome. It also contains general information about how to search Google with your personalized filters off.

THE VIDEO
If you just want a really quick, good overview of how search results work, here's a great video:

BNHR6IQJGZs