mardi 17 février 2015

Programmatically do stuff on site using Web Crawling in c#.net

I have a requirement to do a web crawling on a site, and get the search results output links from the site, to do some stuff on my end. There are by default 20 page Size, which can extend up to 200 page size of the search result with the select Option on the site.


My requirement is to make a crawling of the site, with the search keyword, and get all the possible search result links, and then traverse each link and do some stuff on it.


The challenge is that on the crawling, while change the option for the page size, the page reloads, and after that it is changing the search results, also there is the option on the site for the set search result as an XML, and there is the anoter dropdown for the same,


So basically I need to make a console app, which is getting the search text as an input, manage the link for the search on site, with the search text, then make changes for the page size to 200 and change the search result as an xml (basically want to trigger the click event for both the dropdown) and then get the result.


Can anybody help me to achieve it.


-- Thanks





Aucun commentaire:

Enregistrer un commentaire