jeudi 28 janvier 2021

robots.txt with no user-agent specified?

A website I want to scrape has this robots.txt:

User-agent: Disallow: /wp-login.php
Disallow: /wp-admin/
Crawl-delay: 10

How do I interpret this when there seems to be no user-agent specified? Am I blanking on something obvious? Never seen something like this before.




Aucun commentaire:

Enregistrer un commentaire