Posts Tagged ‘robot.txt’

Search Engine Spiders – The Basics

March 24, 2008

The average user of the internet is familiar with which search engines they use for searches.  However most people often wonder how Google, Yahoo or any of the search engines go about determining which websites get presented on the results page.  Here is a brief overview of how it works including a bit of information on Search Engine Optimization

Spidering websites

Every search engine, such as Google, has spiders that literally crawl all the websites in the internet.  As they crawl a website they look at content only.  So for example, if your website is all flash, the spiders may find your site but leave immediately.  Without content they have nothing to crawl.   The spiders have actually over time gotten quite smart.  They pick up on specific keywords and by crawling all of the content will hopefully gain an insight into what the main theme of your site is.

Indexing websites

After they crawl a website, the information in then put in an index or database.  The information resides in this index and is ready to be utilized if certain keywords pertaining to the subject matter are used in a search.   How they get ranked will be discussed in a moment.

Does every page of a site get spidered and indexed?

How effectively the site is crawled and ultimately indexed depends on many factors.  Let’s say the architectural design of your site is poor.  Google’s spider may only crawl a few pages because it gets stalled or blocked from continuing.  Maybe the site is missing the robot.txt information in your HTML code.   Maybe there have been black hat techniques used on your site that look like spam.  And the list goes on and on….

How are sites ranked?

Just because the spiders have crawled a website doesn’t mean it will rank well.  Google alone has an algorithm with over 200 variables that are utilized when determining how to rank which sites and in which order.   In addition, each of these 200 variables is weighted differently based on importance.  Over time the variables may change and/or the weighting of the variables may change.

How to achieve good rankings

Ok here’s my plug, but with the little information shared above, optimizing your own site isn’t for the faint of heart.  Actually we’re just talking about rankings; we haven’t even begun to talk about traffic or conversions.  So we truly are at the beginning of the story when it comes to search engine optimization.  If you’re interested in gaining new customers or new business, contact Top SEO Marketing today.  Our business is to drive business to your site at the Local, Regional and National levels.  Let us take the headache out of optimizing your website.

Rebecca Ratliff
Top SEO Marketing