Code Snippets robots.txt

Short texts robots.txt 🧑‍💻 Code snippets robots.txt ✍️ Reading lists robots.txt 👀

Sample robots.txt for Magento on Hypernode

User-agent: * Allow: / Crawl-delay: 25 sitemap: http://magentotestpakket.nl/sitemap.xml Disallow: /*?p= Disallow: /*&p= Disallow: /*?price= Disallow: /*&price= Disallow: /*?color= Disallow: /*&color= Disallow: /*?limit= Disallow: /*&limit= Disallow: /*?order= Disallow: /*&order= Disallow: /*?dir= Disallow: /*&dir= Disallow: /where/ Disallow: /w/ # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ Disallow: /js/ Disallow: /lib/ Disallow: […]