dimanche 27 mars 2016

A program to continuously crawl pages

I'm looking for a program that finds links on a page and follows them, making a map of everything it finds. I know things like this exist but I'm looking for one that will follow links inside of links and not stop until it reaches dead ends.

Does a software to do this already exist? If not, would it be relatively easy to create?




Aucun commentaire:

Enregistrer un commentaire