results from Curl is different than results from P

2019-06-11 07:45发布

问题:

I'm trying to crawl Forever21 and it loads partials via AJAX requests. So, I'm trying to simulate the AJAX requests using Postman (Chrome Extension) and the result is totally different from the same request using Curl.

Here is the curl request

curl -X POST -H "Cache-Control: no-cache" -H "Postman-Token: 6d9cdb68-18f7-33c4-003a-3a6940d9d182" -H "Content-Type: application/x-www-form-urlencoded" -d 'ctl00%24MainContent%24ScriptManager1=ctl00%24MainContent%24upColorChart%7Cctl00%24MainContent%24dlColorChart%24ctl01%24imgColorImage&__EVENTARGUMENT=&__ASYNCPOST=true&__VIEWSTATE=' http://www.forever21.com/Product/Product.aspx?BR=f21&Category=outerwear_coats-and-jackets&ProductID=2000102135

And here is response:

69|dataItem||<script type="text/javascript">window.location="about:blank"</script>|55|pageRedirect||/Error/Default.aspx?aspxerrorpath=/Product/Product.aspx|%

Here are the Postman vars:

URL: http://www.forever21.com/Product/Product.aspx?BR=f21&Category=outerwear_coats-and-jackets&ProductID=2000102135
Request type: POST (x-www-form-urlencoded)
Variables
     ctl00$MainContent$ScriptManager1: ctl00$MainContent$upColorChart|ctl00$MainContent$dlColorChart$ctl01$imgColorImage
     __EVENTARGUMENT:
     __ASYNCPOST: true
     __VIEWSTATE: 

And here is the response:

384|updatePanel|ctl00_MainContent_upImageButtonList|

<div id="scroller_frame" style="left:-10000px; position:relative;">
    <ul id="scroller" class="clearfix" style="padding-left: 0px;"></ul>
</div>
                                    |440|updatePanel|ctl00_MainContent_upProductLayout|

<div id="productLayer">
    <a id="ctl00_MainContent_productZoomLink" class="jqzoom" title="" rel="gal1">
        <img id="ctl00_MainContent_productImage" class="ItemImage" title="" src="" style="border-width:0px;padding-left: 0px;" />
    </a>
    <div></div>
</div>
                                    |109|updatePanel|ctl00_MainContent_upColorChart|

                                                                                    |383|updatePanel|ctl00_MainContent_upColorList|

<select name="ctl00$MainContent$ddlColor" onchange="javascript:setTimeout('__doPostBack(\'ctl00$MainContent$ddlColor\',\'\')', 0)" id="ctl00_MainContent_ddlColor" class="input" style="width:175px;"></select>
                                                                                    |298|updatePanel|ctl00_MainContent_upSizeList|

<select name="ctl00$MainContent$ddlSize" id="ctl00_MainContent_ddlSize" class="input" onchange="fnChangeSize();"></select>
                                                                                    |0|hiddenField|__EVENTTARGET||0|hiddenField|__EVENTARGUMENT||0|hiddenField|__LASTFOCUS||1344|hiddenField|__VIEWSTATE|/wEPDwUJNjkyNTA5NjE0Dw9kZBYCZg9kFggCBA8WAh4JaW5uZXJodG1sBQEwZAIFDxYCHwAFBSQwLjAwZAIHDxYCHgRUZXh0BdMFPGRpdiBpZD0iR2xvYmFsTmF2Q29udGFpbmVyIj48ZGl2IGlkPSJHbG9iYWwxIj48YSBocmVmPSJqYXZhc2NyaXB0OnNob3dQb3BXaW4oJ2h0dHA6Ly93d3cuZm9yZXZlcjIxLmNvbS9IdG1scy9odG1sQ29udGVudHMuYXNweD9icj1mMjEmdXJsPS9Qb3B1cC9lbi1VUy8yME9GRlNIT0VTQkFHU19kaXNjbGFpbWVyLmh0bWwnLDQ1MCwxOTAsbnVsbCk7Ij48aW1nIHNyYz0iL2ltYWdlcy9lbi8yMDE0MTEyMi8xLmpwZyIgLz48L2E+PC9kaXY+PGRpdiBpZD0iR2xvYmFsMiI+PGEgaHJlZj0iaHR0cDovL3d3dy5mb3JldmVyMjEuY29tL1Byb2R1Y3QvQ2F0ZWdvcnkuYXNweD9icj1mMjEmY2F0ZWdvcnk9cHJvbW8tc2hvZXMtaGFuZGJhZ3Mtc2FsZSI+PGltZyBzcmM9Ii9pbWFnZXMvZW4vMjAxNDExMjIvMi5qcGciIC8+PC9hPjwvZGl2PjxkaXYgaWQ9Ikdsb2JhbDMiPjxhIGhyZWY9IiMiIG9uQ2xpY2s9InNob3dQb3B1cCgpOyI+PGltZyBzcmM9Ii9pbWFnZXMvZW4vMjAxNDExMjIvMy5qcGciIC8+PC9hPjwvZGl2PjwvZGl2PjxhIGhyZWY9Imh0dHA6Ly93d3cuZm9yZXZlcjIxLmNvbS9Qcm9kdWN0L0NhdGVnb3J5LmFzcHg/YnI9ZjIxJmNhdGVnb3J5PXByb21vLXNob2VzLWhhbmRiYWdzLXNhbGUiIGNsYXNzPSJtdy1nbG9iYWwtYmFyIiBzdHlsZT0iZGlzcGxheTogbm9uZTsiPjxpbWcgZGF0YS1tdy1zcmM9Ii9pbWFnZXMvZW4vMjAxNDExMjIvZ2xvYmFsLmpwZyIgLz48L2E+ZAIKD2QWAmYPZBYCAhYPZBYEZg9kFgJmD2QWAgIBDzwrAAkAZAIBD2QWAmYPZBYCAgEPEGRkFgBkGAEFHl9fQ29udHJvbHNSZXF1aXJlUG9zdEJhY2tLZXlfXxYDBR9jdGwwMCRNYWluQ29udGVudCRpYnRuU2VuZEVtYWlsBRxjdGwwMCRNYWluQ29udGVudCRpYnRuQWRkQmFnBR5jdGwwMCRNYWluQ29udGVudCRpYnRuV2lzaGxpc3Sy0RNtz8zunSVqrWHQd6XGHBYj3g==|8|hiddenField|__VIEWSTATEGENERATOR|3805503D|0|asyncPostBackControlIDs|||0|postBackControlIDs|||164|updatePanelIDs||tctl00$MainContent$upImageButtonList,tctl00$MainContent$upProductLayout,tctl00$MainContent$upColorChart,tctl00$MainContent$upColorList,tctl00$MainContent$upSizeList|0|childUpdatePanelIDs|||159|panelsToRefreshIDs||ctl00$MainContent$upImageButtonList,ctl00$MainContent$upProductLayout,ctl00$MainContent$upColorChart,ctl00$MainContent$upColorList,ctl00$MainContent$upSizeList|2|asyncPostBackTimeout||90|77|formAction||Product.aspx?BR=f21&Category=outerwear_coats-and-jackets&ProductID=2000102135|

回答1:

The problem was with "User-Agent" header, the Postman did not mention that it sends headers.

So, I appended the Curl request and added "User-Agent" header and it worked fine.