# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /thankyou.html Disallow: /cancel.html Disallow: /payment.html