User-agent: * Disallow: Crawl-delay: 5 sitemap: http://www.newyorkhistoryblog.com/sitemap_index.xml User-agent: msnbot Crawl-delay: 5 User-Agent: AhrefsBot Crawl-Delay: 10 # start (J)ExR User-Agent: * Disallow: /jexr/ # end (J)ExR