I'm trying to write an extension that scrapes a site that needs to be loaded as a webpage, due to the complexities of the pages (XHR isn't really an option). The best solution I've come up with is to do the scraping via an iframe placed in the background.html. The extension will provide the user with notifications when certain conditions on the target page are met.
There was a very similar question posted here a few years ago:
Chrome extension: loading a hidden page (without iframe)
I was able to load an iframe with the target page in the background page by removing the x-frame-options header, as suggested in the response to the question above. I can scrape the data via a content script when the page is loaded in a browser tab/window. However, I want to be able to do this from the background script, and of course it cannot access the iframe contentDocument directly due to cross-origin security.
Unfortunately, although the iframe loads the content from the target page just fine, it does not load the content script. It doesn't seem the background.html loads any content scripts at all. This makes sense, because things could get crazy when extensions start loading each other's content scripts.
Is there a manifest option that will allow a specific content script to run in the iframe on the background page?
If not, is there any way to relax cross-origin security in the extension to access the iframe contentDocument?
Please note, using the --disable-web-security chrome command line switch isn't an option. I want non-technical people to be able to use the extension, and I don't want to require users to disable web security to use it.
If there isn't any way to run a content script in a background page or relax security, what other possibilities are there for scraping a page DOM with a chrome extension without using XHR? Surely there must be a way.