dimanche 5 avril 2015

What does the dollar sign mean in robots.txt

I am curious about a website and want to do some web crawling at the /s path.


My questions are:


What does the dollar-sign mean is this case? And is it appropriate to crawl the url /s? with respect to the robots.txt file?


User-Agent: * Allow: /$ Allow: /debug/ Allow: /qa/ Allow: /wiki/ Allow: /cgi-bin/loginpage Disallow: /





Aucun commentaire:

Enregistrer un commentaire