Unable to let my script perform all the clicks on

Question

I've created a script in python using pyppeteer to collect the names of different institutions traversing multiple pages from a website. What I wish to do is let my script rove different pages by clicking on the next page button while parsing the names from each page.

website address

What I've tried:

import asyncio
from pyppeteer import launch

url = "https://www.incometaxindia.gov.in/Pages/utilities/exempted-institutions.aspx"

async def fetch_table(link):
    browser = await launch(headless=False)
    [page] = await browser.pages()
    await page.goto(link)
    while True:
        await page.waitForSelector("h1.faqsno-heading", {'visible':True})
        for item in await page.querySelectorAll("h1.faqsno-heading"):
            name = await item.querySelectorEval("div[id^='arrowex']",'e => e.innerText')
            print(name)

        try:
            elem =  await page.querySelector("[title='Next Page']")
            await elem.click()
        except Exception: break

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(fetch_table(url))

The above script is doing it's job just fine until it encounters an error somewhere between 5 to 10 pages. Pages may vary, though.

Traceback (most recent call last):
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 23, in <module>
    loop.run_until_complete(fetch_table(url))
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 568, in run_until_complete
    return future.result()
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 11, in fetch_table
    await page.waitForSelector("h1.faqsno-heading", {'visible':True})
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\frame_manager.py", line 834, in __await__
    raise result
pyppeteer.errors.TimeoutError: Waiting for selector "h1.faqsno-heading" failed: timeout 30000ms exceeds.

However, when I bring about a minor change and try like this, I can see that the script also does it's job until it encounters the following error:

try:
    await page.click("[title='Next Page']")
except Exception: break

I get the following error:

Traceback (most recent call last):
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 48, in <module>
    loop.run_until_complete(fetch_table(url))
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 568, in run_until_complete
    return future.result()
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\demo.py", line 37, in fetch_table
    await page.waitForSelector("h1.faqsno-heading", {'visible':True})
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\frame_manager.py", line 832, in __await__
    result = yield from self.promise
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\frame_manager.py", line 859, in rerun
    *self._args,
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\execution_context.py", line 109, in evaluateHandle
    _rewriteError(e)
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\execution_context.py", line 239, in _rewriteError
    raise error
  File "C:\Users\WCS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pyppeteer\execution_context.py", line 106, in evaluateHandle
    'userGesture': True,
pyppeteer.errors.NetworkError: Protocol error Runtime.callFunctionOn: Target closed.

How can let my script keep going until all the clicks are performed?

Answer 1

Please note that the website you trying to scrape has hundreds of pages! I didn't want to my system get stuck for long running process. Instead I tried slots=20 pages, it appears to be working. You may change the number of slots to experiment yourself. I am using python 3.6, websockets 6.0. I am on windows 8.1. I have added few lines of code to limit number of pages. In addition to that I have added await page.waitForSelector("[title='Next Page']", {'visible':True}) in couple of places.

Here is code

import asyncio
from pyppeteer import launch

url = "https://www.incometaxindia.gov.in/Pages/utilities/exempted-institutions.aspx"

async def fetch_table(link):
    browser = await launch(headless=False)
    [page] = await browser.pages()
    await page.goto(link)
    slots=20 # change here for number of pages you want to scrape
    i=0
    while True:
        i=i+1
        if(i>slots):
           await page.waitForSelector("[title='Next Page']", {'visible':True})
           break
        await page.waitForSelector("h1.faqsno-heading", {'visible':True})
        for item in await page.querySelectorAll("h1.faqsno-heading"):
            name = await item.querySelectorEval("div[id^='arrowex']",'e => e.innerText')
            print(name)

        try:
            await page.waitForSelector("[title='Next Page']", {'visible':True})
            elem =  await page.querySelector("[title='Next Page']")
            await elem.click()
        except Exception: break


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(fetch_table(url))

-Output around 20th page

(testenv) C:\Py\pypuppeteer1>python stack5.py
....
....
SHREE SUBRAHMANYA VANGMAYEE PARISHAD, GOAAAPTS2410M
SHREE SUBRAHMANYA VANGMAYEE PARISHAD, GOAAAPTS2410M
WORD FOR THE WORLD FELLOWSHIPAAAAW6295Q
JANA SEVA TRUSTAACTJ0594Q
VAGDEVI VILAS EDUCATIONAL AND CHARITABLE TRUSTAABTV8264G
NCORE IMPACT FOUNDATIONAAFCN9985K
M V M EDUCATIONAL TRUSTAACTM5633K
SOCIETY FOR BETTERMENT OF EDUCATIONAAHAS9354D
SWASTIKAM CHARITABLE TRUSTAAJTS9298K
M/S SANKALP YUVA PRERIT SANVARDHAN BAHUUDDESHIYA SANSTHAAAITS8452J
TRAILOKYA BOUDHA MAHASANGHA SAHAYYAK GAN NAGPURAAABT2581K
MISSIONAL YATRA INDIA (MY INDIA) CHARITABLE TRUSTAAOTM9109M
VRUNDAVAN SHIKSHAN VA BAHUUDDESHIYA SANSTHAAABAV6403C
SHRI JAGDAMBA GOVIGYAN ANUSANDHAN KENDRAAAQTS8474C
SUSHILABAI DEUSKAR PRATISHTHANAALTS8647L
AMRAVATI DISTRICT OPTHALMIC SOCIETYAAETA8499F
ALUMNI ASSOCIATION OF INDIRA GANDHI GOVERNMENT MEDICAL COLLEGE NAGPURAAGTA1367C
VIDYA NIDHI NAGPURAABTN4351L
LATE RAJSINGH DUNGAPUR MEMORIAL FOUNDATIONAABTL5457B
ARTHIK DRUSTYA MAGASVARGIYA SAMAJ SHIKSHAN SANSTHAAACTA6288L
SPARSHAADAS4064Q
LATE PADMADEVI R. MALOO FOUNDATIONAAATL4181B
VISHWARACHNA GRAMINS VIKAS SANSTHAAAATV5359D