bash script: word occurrences in web sites

Question

I need to make a script that counts the word occurrences on the web pages

what I'm doing is calculating random IPs (avoiding checking the same ips more than once), using nmap to see if the port 80 is open to know if it's a web server and then I use w3m to make the http page as file. After it's easy to count the word occurances

I got a few questions and problems

• This process takes A LOT, but I can't think any method to make it quicker
• Many of the ips with the port 80 open aren't web sites, they aren't up and they might have some problems, is there any ways to check only the sites that are up?
• this method only check the word occurrences in the index page of a web site, is there a way to check also the other public pages?

thanks a lot

Answer 1

I do similar stuff with awk. Awk is awesome for text parsing. What I do is analyze how many HTTP GETs each IP address done in the Apache log. So bots like yours would appear easily in my statistics :P With awk I've outperform all solutions made by all my collegues, in PHP, Ruby and bash script.

The problem is you are not generating statistics per file (or page). You are summing up all results, right? So I would use SQLite to keep track of how many times a word have appeared in all scanned texts. It is easy (and fast) to add data in SQLite with a shell script.

Also you should user wget --spider or other spider HTTP clients because they will download the content not only from the index page but from all pages that has links (HREFs) in the first page. So you can scan a website recursively.