Subscribe to RSS

Using sitemaps to ensure frequent crawls

Sitemaps have been around for a very long time but it is only recently that the large search engines have begun to support sitemap submission to help towards the crawling of your site more frequently. It can also help to let search robots know where pages that may be hidden can be found. Typically, you want to make sure these pages are accessible throughout the site through organic linking; hopefully you haven’t failed your site in that aspect.There are several sites available that can help generate a sitemap for you but a lot of them are just the same as using a robot to crawl the site and can miss pages or not display their relevance in the right way. The best way to do make a sitemap for your site is to be sure that you are getting all of the pages and so a lot of it needs to be done by hand if you don’t have a content management system that lists out all of the pages on your site.

If you have a content management system they can typically tell you what all of the pages are on your site. You should be able to easily setup a page that will list out all of your pages in XML format and allow you to go through and add how often they are updated.

Making your sitemap with a little help

However, the best approach I have found is to use something like XML Sitemaps and go in and add any pages it may have missed or remove pages that I don’t want crawled. Typically I have done a pretty decent job at internal linking and the crawlers are able to find most of my pages. If you have a corporate site that may not have a good link hierarchy you could spend some time creating your sitemapsâ�� especially if the site is static.

The effort put into this process does pay out though, so if you have to spend an hour or two creating your sitemap the results from the search engines will definitely improve some of your more hidden pages to be listed in the search results and hopefully drive more relevant traffic to your site.

Popularity: 3%

Whats Next?

  • Save to Delicious! Save the page

One Response to “Using sitemaps to ensure frequent crawls”

  1. Ian Says:

    I like ping goat better

Leave a Reply


Recent comments

Top 10 search engine optimization techniques

On September 5, 2008, Q5 Webdesign wrote:

Great tips, already know the most of it but thanks!

Open source Dreamweaver alternatives

On September 4, 2008, WB wrote:

GIMP doesn’t count? GIMP rules!

Top 10 search engine optimization techniques

On September 1, 2008, George Cleanthous wrote:

Great list here. I think it is important that people understand they need to use ALL of these techniques. Any one on its own is practically useless, yet...

I’ve been on a vacation, normality will happen next week

On August 28, 2008, holiday travel wrote:

well this is useful… (at least for me) very thanks ————&# 8212;———R 12;—–...

Top 10 search engine optimization techniques

On August 26, 2008, ибп wrote:

А что, неплохо!