Getting Indexed By Google
After writing a new page or website it's nice to have someone else in the world read it apart from you. Getting indexed by Google is the first step to becoming the top of search results.
The 'Index' is created when Googlebot trawls through the net and reads a load of pages. Every page Googlebot finds gets added to the Index. When someone does a search, the query they enter doesn't search the whole internet, it searches the Index. From the Index the most relevant pages are displayed as search results.
This means it's important Googlebot finds as much of your site as possible. Usually the homepage and other top level pages will be indexed without any effort. The lower level pages may be harder to find by Googlebot, they usually require some extra work to get indexed. It is important to make sure everything is indexed, there isn't much point have a page on a server that can't be found through search.
Firstly it's nice to find out if any of your pages are already indexed. To do this type "site:www.yoursite.com" (without the quotes, replace yoursite.com for your domain ) into the Google search box. This lists all the indexed pages of www.yoursite.com. This also works on MSN search to find out how many pages are indexed in MSN. I think it should work in Yahoo and other search engines too but Yahoo is giving me an error when I try at the moment.
Using site: can also be useful once all the URLs have been indexed. Searching using this prefix while using SeoQuake gives a good overview of PageRank distribution within the domain. It can also be used to find out how many pages your competition have indexed. There are often surprising results with many pages which aren't obviously linked to from the main site.
Get Indexed Fast
As with most things, the faster the better. To get indexed fast you need to help Googlebot find what is on your site.
One of the best ways to do this is to create a sitemap. Sitemaps can be written in a few formats, some are designed for users who want a simple navigation page for the whole site. Other sitemaps are designed with the search bots in mind.
For purely indexing objectives an XML sitemap is a good choice. I have written a PHP script to generate a new XML sitemap each time a new page is added. They can be hand written too for more information check this page Sitemap Protocol. This site lets you automatically create an XML Sitemap for free by entering the URL of your site xml-sitemaps.com/.
Once a sitemap has been created you simply need to upload it to the root directory of your domain, and tell Google Webmaster Tools where it is.
Another way of getting indexed by Google is to get some backlinks from other sites. Social networking sites like Digg and Stumbleupon allow you to add links to pages. Doing this often gets a page indexed the next day in my experience.
The design of the site can have an effect on getting indexed. If any images are used as links there should always be a text link alternative. It is possible to use CSS to create buttons with rollover images that are still text links.
It is best not to create more than 3 levels of directories on a new site. If a page is hidden very deep in a directory structure it will struggle to get indexed. If you really want a low level page to get indexed, link to it from some higher level pages on the site with decent PageRank.
Most of these techniques are good practise for increasing visitors to a site, so they provide additional benefits apart from getting indexed by Google.
Posted by Tom on Wed 21st Jan 2009
