dimanche 1 septembre 2019

Python scrape number data website in every single date of url

I'm trying to scrape a website along the time from 2002 today. Each has its set of numbers. I turned on the inspect, it looks like this:

<tr>
    <td class="chu17 need_blank">42</td>
    <td class="chu17 need_blank">46</td>
    <td class="chu17 need_blank">46</td>
    <td class="chu17 need_blank">46</td>
    <td class="chu17 need_blank">49</td>
    <td class="chu17 need_blank">49</td>
    <td class="chu17 need_blank">61</td>
    <td class="chu17 need_blank">62</td>
    <td class="chu17 need_blank">69</td>
</tr>

I want to get those numbers before the </td> and print it in a .csv file. Here is the python code I am working on:

import datetime


urls = []

sdate = datetime.date(2002, 1, 5)
edate = datetime.date.today()

delta = edate - sdate

for i in range(delta.days + 1):
    day = sdate + datetime.timedelta(days=i)
    day = day.strftime("%d-%m-%Y")
    urls.append('https://website.php?date=' + day)

with open('output.csv', 'w') as data: 
    file = csv.writer(data)

The code will replace every single day on website url from 5-1-2002. I want it to be: In each day the program will gather (scrape) numbers on that HTML code like above, and once it's done the gathering it will go to the next day and do the gathering numbers set of that next day and so on.

How to do that?

Thank you.




Aucun commentaire:

Enregistrer un commentaire