lundi 3 avril 2017

How to get the Search URLs form different websites

I wanted to do some Broad Crawls for many shopping website. Such as amazon.com,macys.com and so on.

In amazon, If I want to get clothes' information, I did those code in python

MySpider msp = MySpider()
p = 1
while end_flag:
    page = 'page=%s&' % str(p)
    #Copy the URL form brower
    search_url = 'http://ift.tt/2oPLtb6' % page
    msp.GetInfo(search_url)
    p = p + 1

However, when I did in another site , [macys.com][2]. The search URL will be different from the amazon. So I need to copy again.

Can I get the search URL by code or any way else for different site

amazon's search




Aucun commentaire:

Enregistrer un commentaire