Scrapy and Selenium submit form that renders dynam

2019-06-01 04:29发布

I'm using selenium to first load the form which being generated via ajax.

Now I'm having troubles passing the selenium response to scrapy FormReuqest method

to send the form data values.

The form has jquery validation before user can submit it, does it make it harder to submit using scrapy?

any help is appreciated.

thanks

1条回答
乱世女痞
2楼-- · 2019-06-01 05:21

You don't need neither Scrapy, nor selenium here. Just make an underlying POST request and parse the json response. Example using requests:

import json
import requests


URL = 'http://calculator.shipito.com/en/rates'
data = {"location": "10", "country": "GB", "city": "London",
        "postalcode": "123456", 
        "packages": [{"dimensions_units": "in", "weight_units": "lbs", "dimension": {"width": "9", "height": "20.4", "length": "17"},
        "weight": "9", "value": "170"}]}

response = requests.post(URL, data=json.dumps(data))
print response.json()

Prints:

[
    {u'infoUrl': u'http://www.shipito.com/postage#tnt', u'isPromoted': False, u'popularity': 1, u'name': u'TNT Economy', u'deliveryTime': 6, u'insurable': True, u'usesDimWeight': True, u'shippingRate': 68.38, u'separateShippingRate': None, u'bonusShippingRate': 68.38, u'deliveryTimeInfo': u'4-6 business days', u'mps': False, u'insuranceRate': 0, u'logoImg': u'tnt.png'}, 
    ...
]
查看更多
登录 后发表回答