Posts Tagged ‘search engine spiders’

Search Engine Spiders – The Basics

March 24, 2008

The average user of the internet is familiar with which search engines they use for searches.  However most people often wonder how Google, Yahoo or any of the search engines go about determining which websites get presented on the results page.  Here is a brief overview of how it works including a bit of information on Search Engine Optimization

Spidering websites

Every search engine, such as Google, has spiders that literally crawl all the websites in the internet.  As they crawl a website they look at content only.  So for example, if your website is all flash, the spiders may find your site but leave immediately.  Without content they have nothing to crawl.   The spiders have actually over time gotten quite smart.  They pick up on specific keywords and by crawling all of the content will hopefully gain an insight into what the main theme of your site is.

Indexing websites

After they crawl a website, the information in then put in an index or database.  The information resides in this index and is ready to be utilized if certain keywords pertaining to the subject matter are used in a search.   How they get ranked will be discussed in a moment.

Does every page of a site get spidered and indexed?

How effectively the site is crawled and ultimately indexed depends on many factors.  Let’s say the architectural design of your site is poor.  Google’s spider may only crawl a few pages because it gets stalled or blocked from continuing.  Maybe the site is missing the robot.txt information in your HTML code.   Maybe there have been black hat techniques used on your site that look like spam.  And the list goes on and on….

How are sites ranked?

Just because the spiders have crawled a website doesn’t mean it will rank well.  Google alone has an algorithm with over 200 variables that are utilized when determining how to rank which sites and in which order.   In addition, each of these 200 variables is weighted differently based on importance.  Over time the variables may change and/or the weighting of the variables may change.

How to achieve good rankings

Ok here’s my plug, but with the little information shared above, optimizing your own site isn’t for the faint of heart.  Actually we’re just talking about rankings; we haven’t even begun to talk about traffic or conversions.  So we truly are at the beginning of the story when it comes to search engine optimization.  If you’re interested in gaining new customers or new business, contact Top SEO Marketing today.  Our business is to drive business to your site at the Local, Regional and National levels.  Let us take the headache out of optimizing your website.

Rebecca Ratliff
Top SEO Marketing


Search Engine Spiders, Let em BITE!

February 5, 2008

 For the do-it yourself types, there is one aspect of search engine optimization that is not for the faint of heart.  Real spiders have a tendency to be feared by many.  However if your site is not search engine spider friendly, they may come to visit but not stay.  IN this instance spiders aren’t to be feared. The only fear is if they come and don’t stay and take a bite.  Search engine spiders that encounter impediments to accessing the pages of your site, results in the pages of your site not being crawled or indexed.    Analysis of your website to ensure the search engine spiders are able to access all the pages on your site, takes an understanding of the structure and mechanics of your website.   Here are just a few of the considerations:

· Is the code compliant with WC3 standards?
· Is your site predominantly flash?
· Does the site have bloated code that could slow down or stall the spiders
· Has the robot.txt code been added to every page and is it setup to guide the    spiders properly
· Is the navigational structure of the site set up so the spiders can easily locate all the pages on the site no matter how deep?
· Are your navigational buttons images?
· Which pages have been crawled and how often?
· Which search engines have crawled your site?

So if you’ve spent a lot of time designing a website with great content and you think it’s going to be really appealing to your audience, your site must be found by the search engine spiders.  Let Top SEO Marketing handle the technical aspects of your website to ensure the spiders find your site friendly and inviting.  As part of a complete Search Engine Optimization campaign we’ll review your entire site and ensure that all search engine spider impediments have been removed.  Don’t let technical issues keep you from being bit by the spiders, in this case not getting bit can hurt you.  At Top SEO Marketing our business is getting you business by driving traffic to your website. 

Rebecca Ratliff
VP of Local Search Marketing
Top SEO Marketing