I am new to web scraping, and I use the following tool and method to scrap:
- I use R (with packages Curl, XML, etc) to read the web pages (with a url link), and htmlTreeParse function to parse the html page.
- Then in order to know get the data I want, I first use the developer tool i Chrome to insepct the code.
- When I know in which node the data are, I use xpathApply to get them.
Usually, it works well. But I had an issue with this site: http://www.sephora.fr/Parfum/Parfum-Femme/C309/2
- When you click on the link, you will load the page, and in fact it is the page 1 (of the products).
- You have to load the url again (by entering a second time the url), in order to get the page 2.
- When I use the usual process to read the data. The htmlTreeParse function always gives me the page1.
I tried to understand more this web site:
- It seems that it is built with Oracle commerce (ATG commerce).
- The "real" url is hidden, and when you click on the filter (for instance, you select a brand), you will get url with requestid: http://www.sephora.fr/Parfum/Parfum-Femme/C309?_requestid=285099
This doesn't help to know which selection I made.
Could you please help:
- How can I access to more products ?
Thank you
