mardi 21 juillet 2015

Best Buy Web Crawler C#

I'm trying to make a crawler for the Best Buy website, however when I try to run the DownloadString function I get a 403 forbidden error. I've tried adding a header but the results remain the same. I've also run Fiddler and see what header is being sent by my browser and used the header there. What could possibly be causing the issue?

        WebClient client = new WebClient();
        client.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36");
        client.DownloadString("http://www.bestbuy.com");

Aucun commentaire:

Enregistrer un commentaire