What is Robots.txt and why should I use it?
July 31st, 2007 by Dustin Brewer
I am here to answer just that question. There has been a lot of mystery I have gathered from fellow web developers about the benefits of using a robots.txt. Most are under the false impression that the only purpose for this text file is to tell search robots not to index a certain section of their site. This isn’t the case at all! There are much more useful reasons to be using a Robots.txt that have to do very much with Search Engine Optimization (SEO) that most people don’t realize. (more…)
Popularity: 4%
Posted in SEO
No Comments »





