jeudi 25 février 2016

Is there any advantage of using X-Robot-Tag instead of robots.txt?

It looks like there are two mainstream solutions for instructing crawlers what to index and what not to index: adding an X-Robot-Tag HTTP header, or indicating a robots.txt.

Is there any advantage to using the former?




Aucun commentaire:

Enregistrer un commentaire