Men and women use DA to find out which Internet sites are reliable within their field and to figure out the place to obtain good hyperlinks. txt file is then parsed and will instruct the robotic concerning which web pages are certainly not being crawled. Being a search engine crawler https://www.youtube.com/watch?v=EGBkutWZZTk