Converting GeoLite2 data for use with xtables geoi

My apologies if this has been covered here or elsewhere. I read the postings back to 2016.

My debian system stopped updating the xtables geoip database. On investigation it developed that this is because Maxmind dropped support for legacy GeoIP databases. I have got as far as installing and configuring Maxmind's geoipupdate program for the GeoLite2 database and scheduling it weekly in crontab.

At this point I am stumped. geoipupdate returns a .mmdb database. This is not usable by the debian-supplied scripts which convert .CSV files to the country code files in /usr/share/xt_geoip/LE and /usr/share/xt_geoip/BE.

The debian package xtables-addons has not been updated to deal with this situation.

Assistance or a pointer to a solution will be gratefully received. At present I am still using the last valid database which is now getting to be over six months old.

回答1:

Have a look at GeoLite2xtables :- https://github.com/mschmitt/GeoLite2xtables

You can download a zip (or git clone). It has example workflow (shell commands) for legacy GeoLite CSV (which is probably what you have which stopped working in early Jan 2019) and GeoLite2 CSV (which you can use instead).

回答2:

You can also download the source from the xtable-addon's project (either directly or from the sid version of the xtables-addons-common package) and grab updated versions of the scripts.

https://sourceforge.net/projects/xtables-addons/files/Xtables-addons/

See the following askubuntu answer: https://askubuntu.com/questions/1117669/xtables-addons-issues-with-maxmind-geolite2

回答3:

I eventually ended up writing this script, which now runs weekly. So far (three months on) it appears to be satisfactory.

cat update-geoip.sh

#!/bin/bash -e

GEOLITE_URL="https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country-CSV.zip"
GEOLITE_ZIP="GeoLite2-Country-CSV.zip"
COUNTRY_URL="http://download.geonames.org/export/dump/countryInfo.txt"

#
# Switch to the GeoIP directory if not already there
#
echo "--> cd /usr/share/xt_geoip"
cd /usr/share/xt_geoip

#
# Remove anything remaining from previous failed runs
#
# Note:  DO NOT delete the existing BE and LE subfolders at this
#        time.  If the download fails the result would be no
#        database at all.
#
echo "--> rm -r GeoLite2*"
rm -r -f GeoLite2*
echo "--> rm countryInfo.txt"
rm -f countryInfo.txt
echo "--> rm GeoIP-legacy.csv"
rm -f GeoIP-legacy.csv

#
# Get the GeoIP ZIP file
#
echo "--> wget --no-check-certificate $GEOLITE_URL"
wget --no-check-certificate $GEOLITE_URL

#
# See if the ZIP file now exists
#
if [ ! -e $GEOLITE_ZIP ]; then
  echo "--> GeoIP ZIP file did not download"
  echo "--> Send email to root and stop here"
  /usr/sbin/sendmail root << EOM
From: Update_GeoIP
To: root
Subject: GeoIP update failed

GeoIP update failed.
Unable to download GeoIP ZIP file
$GEOLITE_ZIP
EOM
  exit
fi

#
# Unzip the ZIP file
#
echo "--> unzip $GEOLITE_ZIP"
unzip $GEOLITE_ZIP

#
# Delete the ZIP file
#
#echo "--> rm $GEOLITE_ZIP"
rm $GEOLITE_ZIP

#
# Move the received data directory to a standard name
#
echo "--> mv GeoLite2-Country-CSV_* GeoLite2"
mv GeoLite2-Country-CSV_* GeoLite2

#
# See if the critical GeoIP data files now exist
#
if [ ! -e "GeoLite2/GeoLite2-Country-Blocks-IPv4.csv" ] ||
   [ ! -e "GeoLite2/GeoLite2-Country-Blocks-IPv6.csv" ]; then
  echo "--> GeoIP data files are missing"
  echo "--> Send email to root and stop here"
  /usr/sbin/sendmail root << EOM
From: Update_GeoIP
To: root
Subject: GeoIP update failed

GeoIP update failed.
GeoIP data file(s) are missing
GeoLite2/GeoLite2-Country-Blocks-IPv4.csv
GeoLite2/GeoLite2-Country-Blocks-IPv6.csv
EOM
  exit
fi

#
# Get the country info data file
#
echo "--> wget --no-check-certificate $COUNTRY_URL"
wget --no-check-certificate $COUNTRY_URL

#
# See if the country info data file now exists
#
if [ ! -e "countryInfo.txt" ]; then
  echo "--> Country info data file did not download"
  echo "--> Send email to root and stop here"
  /usr/sbin/sendmail root << EOM
From: Update_GeoIP
To: root
Subject: GeoIP update failed

GeoIP update failed.
Unable to download country info data file
$COUNTRY_URL
EOM
  exit
fi

#
# Build an old format data file from the new format data files
#
echo "--> cat ./GeoLite2/GeoLite2-Country-Blocks-IPv{4,6}.csv | ./convert_GeoLite2.pl ./countryInfo.txt > /usr/share/xt_geoip/GeoIP-legacy.csv"
cat ./GeoLite2/GeoLite2-Country-Blocks-IPv{4,6}.csv | ./convert_GeoLite2.pl ./countryInfo.txt > /usr/share/xt_geoip/GeoIP-legacy.csv

#
# Delete the downloaded data files
#
echo "--> rm -r GeoLite2"
rm -r GeoLite2
echo "--> rm countryInfo.txt"
rm country_Info.txt

#
# Preserve the old BE and LE directories just in case
#
echo "--> rm -r -f LastBE LastLE"
rm -r -f LastBE LastLE
echo "--> mv BE LastBE"
mv BE LastBE
echo "--> mv LE LastLE"
mv LE LastLE

#
# Convert the generated database to the xtables GeoIP format
#
echo "--> /usr/lib/xtables-addons/xt_geoip_build -D /usr/share/xt_geoip ./GeoIP-legacy.csv"
/usr/lib/xtables-addons/xt_geoip_build -D /usr/share/xt_geoip ./GeoIP-legacy.csv

#
# Delete the remaining data files
#
echo "--> rm countryInfo.txt"
rm countryInfo.txt
echo "--> rm GeoIP-legacy.csv"
rm GeoIP-legacy.csv

#
# Notify root that the update succeeded
#
echo "--> Send notification email to root"
/usr/sbin/sendmail root << EOM
From: Update_GeoIP
To: root
Subject: Weekly update of xtables GeoIP completed

Weekly update of xtables GeoIP database successful.
EOM
echo "xtables GeoIP database update completed"