Free SEO Tools offers to Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly. txt file consists of one or more rules. Each rule blocks (or allows) access for a given crawler to a specified file path on that website. Here is a simple robots.txt.