configuring nutch regex-normalize.xml

问题:

I am using the Java-based Nutch web-search software. In order to prevent duplicate (url) results from being returned in my search query results, I am trying to remove (a.k.a. normalize) the expressions of 'jsessionid' from the urls being indexed when running the Nutch crawler to index my intranet. However my modifications to $NUTCH_HOME/conf/regex-normalize.xml (prior to running my crawl) do not seem to be having any effect.

1. How can I ensure that my regex-normalize.xml configuration is being engaged for my crawl? and,

2. What regular expression will successfully remove/normalize expressions of 'jsessionid' from the url during the crawl/indexing?

The following is the contents of my current regex-normalize.xml:

<?xml version="1.0"?>
<regex-normalize>
<regex>
 <pattern>(.*);jsessionid=(.*)$</pattern>
 <substitution>$1</substitution>
</regex>
<regex>
 <pattern>(.*);jsessionid=(.*)(\&amp;|\&amp;amp;)</pattern>
 <substitution>$1$3</substitution>
</regex>
<regex>
 <pattern>;jsessionid=(.*)</pattern>
 <substitution></substitution>
</regex>
</regex-normalize>

Here is the command that I am issuing to run my (test) 'crawl':

bin/nutch crawl urls -dir /tmp/test/crawl_test -depth 3 -topN 500

回答1:

What version of Nutch are you using? I'm not familiar with Nutch but the default download of Nutch 1.0 already contains a rule in regex-normalize.xml which seems to handle this problem.

<!-- removes session ids from urls (such as jsessionid and PHPSESSID) -->
<regex>
  <pattern>([;_]?((?i)l|j|bv_)?((?i)sid|phpsessid|sessionid)=.*?)(\?|&amp;|#|$)</pattern>
  <substitution>$4</substitution>
</regex>

Btw. regex-urlfilter.txt seems to contain something of relevance too

# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]

Then there are some settings in nutch-default.xml which you might want to check out

urlnormalizer.order
urlnormalizer.regex.file
plugin.includes

If that all doesn't help maybe this does: How can I force fetcher to use custom nutch-config?