jeudi 8 octobre 2015

HttpWebRequest against PickupHockey.com

I'm having troubles authenticated my first scraper against the login page for my hockey pool (just looking to grab the stats once a night for dislaying in a nicer format).

        //my variables
        string userName = "My Pool";
        string password = "hockey";

        ASCIIEncoding encoding = new ASCIIEncoding();
        string postData = "ctl00$MainContent$txtPoolName=" + userName + "&ctl00$MainContent$txtPassword=" + password;
        byte[] postDataBytes = encoding.GetBytes(postData);

        string url = "http://ift.tt/1LqS4fo";
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        request.Method = "POST";
        request.Headers.Add("Cache-Control: max-age=0");
        request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
        request.Headers.Add("Accept-Encoding: gzip,deflate");
        request.Headers.Add("Accept-Language: en-GB,en-US;q=0.8,en;q=0.6");
        request.Headers.Add("Upgrade-Insecure-Requests: 1");
        request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36";
        request.ContentLength = postDataBytes.Length;
        request.ContentType = "application/x-www-form-urlencoded";
        request.Headers.Add("Origin: http://ift.tt/1iG84xe");
        request.Referer = "http://ift.tt/1LqS4fo";
        request.Host = "www.pickuphockey.com";
        request.AllowAutoRedirect = false;

        request.CookieContainer = new CookieContainer();

        using (var stream = request.GetRequestStream())
        {
            stream.Write(postDataBytes, 0, postDataBytes.Length);
            stream.Close();
        }

        //cookie container to save for the next request
        var cookieContainer = new CookieContainer();

        //get the cookies from the login page
        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        {
            foreach (Cookie cookie in response.Cookies)
            {
                cookieContainer.Add(cookie);
            }
            Stream receiveStream = response.GetResponseStream();
            // Pipes the stream to a higher level stream reader with the required encoding format. 
            StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
            var r = readStream.ReadToEnd();
        }

I've been working with various examples throughout Stack Overflow and am using fiddler to analyse the web calls made; trying to match all the headers in the browser login with the headers that I'm adding to my code.

Right now when I code break and look at the variable r, I see the html for a page saying that I need to enter a value for the username/password. I assume this means that I'm not setting the parameters correctly in this call but nothing I read or tweak seems to get past this error. I've masked my actual user/password but I assume that even incorrect values (if someone tries the code) should result in a different error message saying something about a bad password. Any web call gurus out there that can spot what I'm doing wrong?

Once the code works I'll wrap it up into some nicer helper class, I know it's a tad messy at the moment.

Appreciate the time.




Aucun commentaire:

Enregistrer un commentaire