The content on this page is based on SearchEngineNews.com’s e-book Winning The Search Engine Wars.
To learn more about Search Engine News, click here.

How to design a search engine friendly website

The first step to properly optimizing your site for the search engines is learning how to design a search engine friendly website. Simply put, a search engine friendly web site means that you make it as easy as possible for the search engines to download and process your web pages. After all, your optimization efforts mean little if the search engines can’t access and/or process your website.  But creating a search engine friendly site is more than a preventative measure, it also can help improve your rankings in the search engine, as will become clear in this tutorial.  There are, in essence, six different steps you can take to creating a search engine friendly website.  We will outline each of those steps here, with later tutorials going into greater detail where need be.

NOTE: some of the material covered in this section is rather technical.  You may need to consult your programmer to properly implement some of the ideas mentioned.  Nonetheless, we will make every effort possible to keep this tutorial as non-technical as possible. 

Six steps to designing a search engine friendly web site

• Use Well Formed, W3C Validated HTML Code
• Avoid (or at least limit) the use of flash technology, JavaScript & fames
• Proper use of URLs
• Proper website and directory structure
• Limiting the size of your web pages
• Proper use of the robots.txt file

Let’s understand each of these concerns a bit better.

Using Well Formed, W3C Validated HTML Code

The first step towards creating your search engine friendly web site is to make sure that your HTML code is correct and well-formed.  Make sure that you have closed all tags which need to be closed and that you haven’t used source code which could “confuse” the search engine spider (such as old and outdated or which is proprietary to a
single browser or code which is so new that it has not yet been recognized by most search engine spiders). 

To learn more about using proper code see our SEO Friendly HTML Code Tutorial –click here to access the tutorial.

Use of Flash Technology, JavaScript & Frames
While it is possible to optimize Flash, JavaScript and/or Frames pages, it is significantly more difficult to do so.  As such, if you want to design a search engine friendly website it is best to avoid using these technologies if possible.  With that said, if you feel that you have to use them, or you are optimizing a site which is already built around these technologies, then it is important to know what to do. 

If you find yourself in this situation then you should check out our Flash, JavaScript, & Frames Tutorial. — click here to access the tutorial.

Proper use of URLs
When designing a search engine friendly site, you want to make sure that you use URLs that the search engine spiders can easily follow. Search engine spiders find and process web pages by following links from one site to another. This means that if your website’s URLs make it difficult for the search engine spider to download and/or process the pages found on your website it is possible that your web pages may never get processed.  And, of course, a page which is not processed is also a page which will not show up in the search engine results.  In other words, if you don’t use a search engine friendly URL you may find that all of your optimization efforts are for naught given that the search engines won’t even know about your web pages.  On the other hand, the right URL may actually help your rankings in the search engine. 

To learn more about the proper use of URLs see our SEO Friendly URLs Tutorial — click here to access the tutorial.

Note:  you may want to first read our tutorial Introduction to URLs tutorial, which explains all of the various "sections" of a URL.  You can read this tutorial by clicking here.

Website and Directory Structure
In general, the deeper a page is on your website (i.e,. the more links that a visitor has to click to get to a particular page), the harder it is for it to get indexed by the search engines.  Unless your site happens to be exceptionally popular with a lot of incoming links you can assume that the search engines are not going to dig that deeply down into your site. Because of this problem many sites employ a sitemap which lists all of the pages on one’s web site. When done properly, this allows you to logically organize the material on your website while simultaneously ensuring that the search engines are able to reach (and thus process) all of the pages on your site.

To learn more about creating a site map for the search engines see our Site Map Tutorial.  You can read this tutorial by clicking here.

Limiting the size of your web pages
Each of the major search engines has a general rule of how large a page they will index.  This amount varies from search engine to search engine, and even from site to site (more popular sites, not surprisingly, have a larger limit than less popular sites).  In general, it is best to limit the size of your pages to 100k or less (not including picture size, which does not effect how much text gets indexed).  This limit (i.e., 100k) is based upon the fact that this may still be Google’s limit.  Even though there have been some indications that Google may have raised that limit, it is best to be safe rather than sorry.  Also, it is easier to optimize
smaller pages and when you split up your larger pages into smaller pages you provide the search engines with more opportunities to find and fully index your web pages.

Proper use of Robots.txt file
Robots.txt files are used to tell search engine spiders which pages not to index. The advantage of using a Robots.txt is that it focuses the search engines on indexing your “important” customer-development, product, and/or sales pages. Remember that the search engine spiders function with limited time and resources when indexing sites, it would be best to use that time wisely. For example, there is no need for the search engines to index your shopping cart (can you think of any reason why you would want your shopping cart checkout pages to show up in the search engines?). Other pages that you don’t want indexed by the search engines are directories that contain images or otherwise sensitive company data (for instance, do you want the search engine spiders to index your password files?). You will also want to use a Robots.txt file for anything that is in your cgi-bin folder.

Warning: don’t accidentally set up your robots.txt file in such a way as to prevent the search engine spiders from crawling your site. The following two lines of code are sufficient to prevent all of the major search engines from ever crawling your site:

User-agent: *
Disallow: /

There you have it, six important steps to help you create a search engine friendly website.  If you find this information too technical, just concentrate on understanding the basics and make sure that your programmer understands what it is that he or she is suppose to do.  While it is not crucial that you understand all of the details on this page, it is crucial that your programmer (or whoever it is that is building your site) does understand the information described above. 

In the meantime, to learn more about creating a search engine friendly web site continue on to our next tutorial SEO Friendly HTML Code.

Next tutorial: SEO Friendly HTML Code.

• Return to the search engine optimization tutorial page
• Return to the main tutorials page

This tutorial written by:
Moshe Morris
President of SEMBasics
Chief Research Analyst at Internet Marketing Initiative (www.internetmi.com)

The content on this page is based on SearchEngineNews.com’s e-book
Winning The Search Engine Wars.
To learn more about Search Engine News, click here.