I've got an old ASPX+XML website created by an external agency here. I only have access to sections of the XML as the web.config is locked.
I want to crawl this site to scrape the pages and capture the relational data. I can do a blank search which returns back all the data - from here a web crawler would be fine. However, I cannot find a web crawler that will hit search - I've tried a JavaScript that submits the form on page load but this still does not work (I guess it's not fast enough).
The URL does not contain the query string (so I cant just do a blank search and copy the results URL for example).
Any ideas?
Thanks in advance
Aucun commentaire:
Enregistrer un commentaire