Search through a webstores all products

Maybe this is the wrong form but I try.

I have a question about how to search/scan through a webstore all products. I have no problem with send a search request to the website and the recieve the answer. But how can I get all products from the website.

It should be possible right? Because several pricecomparing sites can list all the products that a store has.

The solution I have right now is that I send a reqular search?q=cpu to the website and then parse the data that is received.