I wanted to do some Broad Crawls for many shopping website. Such as amazon.com,macys.com and so on.
In amazon, If I want to get clothes' information, I did those code in python
MySpider msp = MySpider()
p = 1
while end_flag:
page = 'page=%s&' % str(p)
#Copy the URL form brower
search_url = 'http://ift.tt/2oPLtb6' % page
msp.GetInfo(search_url)
p = p + 1
However, when I did in another site , [macys.com][2]. The search URL will be different from the amazon. So I need to copy again.
Can I get the search URL by code or any way else for different site
Aucun commentaire:
Enregistrer un commentaire