A website I want to scrape has this robots.txt:
User-agent: Disallow: /wp-login.php
Disallow: /wp-admin/
Crawl-delay: 10
How do I interpret this when there seems to be no user-agent specified? Am I blanking on something obvious? Never seen something like this before.
Aucun commentaire:
Enregistrer un commentaire