mardi 27 janvier 2015

Python web-scrapping, problems

Hello so here is a small segment of my code:



import requests
from bs4 import BeautifulSoup

def mancity():
manclist = []
f = 'mancity.txt'
fo = open(f, 'w')
root_url = "http://ift.tt/JaHbUm"
index_url = root_url + "/Teams/167"
r = requests.get(index_url)
soup = BeautifulSoup(r.content)
playstyle = soup.find_all("div",{"class": "character-card singular"})
for item in playstyle:
chellist.append(item.text)
mstr = ''.join(map(str, manclist))
fo.write(mstr)
print(mstr)


The idea of the code is to scrape a website and extract the data to a text file. Now the code DOES work however not consistently. Sometimes it works, sometimes it returns no values at all. I am at a loss as to why this is? is it because my request is being denied? Its rather annoying as when nothing is returned the text file is overwritten by blank space.


Please run the code your self, and see your outputs. (this is for educational purposes)





Aucun commentaire:

Enregistrer un commentaire