I am trying to crawl a website but the Robots.txt has just the following line
User-agent: *
Does it mean that it doesn't care if I crawl their website?
Aucun commentaire:
Enregistrer un commentaire