Web scrap with multiple inputs and collect Total M

Question

I have a web link as:

url = "zerodha.com/margin-calculator/SPAN"

Here input parameters with sample values for reference mentioned below:

Exchange - NFO
Product - Options
Symbol - DHFL 27-JUN-19
Option Type - Calls
Strike Price - 120
Net Qty appears automatically as 1500, 

and Use SELL Button then Click ADD Button.

I want to collect the Total Margin required (in above case its Rs 49,308) which appears in the right end.

Answer 1

You can just use requests. If you observe your network, you can see that it is making a POST requests with selected payload. This is how I would do it:

from requests import Session

BASE_URL = 'https://zerodha.com/margin-calculator/SPAN'

payload = {'action': 'calculate',
            'exchange[]': 'NFO',
            'product[]': 'FUT',
            'scrip[]': 'DHFL19AUG',
            'option_type[]': 'CE',
            'strike_price[]':120, 
            'qty[]': 4000,
            'trade[]': 'sell'
}



session = Session()

res = session.post(BASE_URL, data=payload)

data = res.json()

print(data)

I got the URL and Payload from observing network. This is what you will get as data in json form.

Results in chrome and python

Just observe how chrome or firefox send and receive data. And reverse engineer with your requests.

Answer 2

website link is dynamic rendering request table data. You should try automation selenium library. it allows you to scrap dynamic rendering request(js or ajax) page data.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
import time

driver = webdriver.Chrome("/usr/bin/chromedriver")

driver.get("https://zerodha.com/margin-calculator/SPAN")

# select  exchange   option of NFO
exchange = driver.find_element_by_name('exchange[]')
exchange.send_keys("NFO")

#  select product option of option
product = driver.find_element_by_name('product[]')
product.send_keys("OPT")

# select symbol  by option value
symbol = Select(driver.find_element_by_name("scrip[]"))
symbol.select_by_value("DHFL19JUN")

# select  option Type  CELL option
optionType = driver.find_element_by_name('option_type[]')
optionType.send_keys("CE")

#add Strike price
strikePrice = driver.find_element_by_name('strike_price[]')
strikePrice.clear()
strikePrice.send_keys("120")

# add Net quantity
netQty = driver.find_element_by_name('qty[]')
netQty.clear()
netQty.send_keys("1500")

# select sell radio button
driver.find_elements_by_css_selector("input[name='trade[]'][value='sell']")[0].click()

#submit form
submit = driver.find_element_by_css_selector("input[type='submit'][value='Add']")
submit.click()

time.sleep(2)

# scrap margin
margin = driver.find_element_by_css_selector(".val.total")
print(margin.text)

where '/usr/bin/chromedriver' selenium web driver path.

Download selenium web driver for chrome browser:

http://chromedriver.chromium.org/downloads

Install web driver for chrome browser:

https://christopher.su/2015/selenium-chromedriver-ubuntu/

Selenium tutorial:

https://selenium-python.readthedocs.io/