mercredi 27 mai 2015

Auto submit form for web crawling

I've got an old ASPX+XML website created by an external agency here. I only have access to sections of the XML as the web.config is locked.

I want to crawl this site to scrape the pages and capture the relational data. I can do a blank search which returns back all the data - from here a web crawler would be fine. However, I cannot find a web crawler that will hit search - I've tried a JavaScript that submits the form on page load but this still does not work (I guess it's not fast enough).

The URL does not contain the query string (so I cant just do a blank search and copy the results URL for example).

Any ideas?

Thanks in advance




Aucun commentaire:

Enregistrer un commentaire