I am curious about a website and want to do some web crawling at the /s path.
My questions are:
What does the dollar-sign mean is this case? And is it appropriate to crawl the url /s? with respect to the robots.txt file?
User-Agent: * Allow: /$ Allow: /debug/ Allow: /qa/ Allow: /wiki/ Allow: /cgi-bin/loginpage Disallow: /
Aucun commentaire:
Enregistrer un commentaire