Robots.txt: Similar to a stoplight, a robots.txt file offers search engine spiders or crawlers way. You’ll use this file to permit spiders to crawl your site. You can even use it to stop them from viewing URLs like your paid landing webpages. Search engine optimization: What you do to get https://www.youtube.com/watch?v=hgyQy18ZcIs