Robots.txt

If you value SEO (search engine rankings), then your XF site should probably have solid robots.txt settings. I have spent a considerable amount of time learning about SEO and examining detailed crawl logs of my site. Based on my learnings I have updated my robots.txt file as shown below.

For those who do not know, a robots.txt file tells search engines what parts of the site they can and cannot explore. These are recommendations to the search engine crawlers which can be ignored, but...