fail2ban regular to find 403 request in nginx

Question

I want to find the 403 requests and ban them,here is my log format

112.253.6.182 - - [08/Sep/2014:17:42:56 -0400] "GET / HTTP/1.1" 403 579 "baidu" "Mozilla/4.0" 50.117.86.72
106.37.177.251 - - [08/Sep/2014:17:42:56 -0400] "GET /index.php HTTP/1.1" 404 576 "baidu" "Mozilla/4.0" 204.44.65.173
190.254.173.14 - - [08/Sep/2014:17:42:56 -0400] "GET /index.php HTTP/1.1" 404 576 "baidu" "Mozilla/4.0" 204.44.65.173
41.222.196.37 - - [08/Sep/2014:17:42:56 -0400] "GET / HTTP/1.1" 403 579 "baidu" "Mozilla/4.0" 50.117.86.72

and my failreg is:

failregex = ^<HOST> -.*"(GET|POST).*.php.*\ 403\ .*$

ignoreregex =

but when I test it using fail2ban-regex command , it returns below

Failregex: 32 total
|-  #) [# of hits] regular expression
|   1) [32] ^<HOST> -.*"(GET|POST).*.php.*\ 403\ .*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [3266] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 3266 lines, 0 ignored, 32 matched, 3234 missed
Missed line(s): too many to print.  Use --print-all-missed to print all 3234 lines

could you help me to make a regex to match 403 requests and print the ip out . Thanks in advance

Answer 1

As per @saaj's answer, while the four lines of log don't have a match, it is obvious from the test results on the full log that there were 32 matches. That said, the new pattern (excluding the php extension) does catch more 403s, obviously.

A simpler pattern to match would be: ^<HOST> .* 403

This can be tested from the command line with:

fail2ban-regex '112.253.6.182 - - [08/Sep/2014:17:42:56 -0400] "GET / HTTP/1.1" 403 579 "baidu" "Mozilla/4.0" 50.117.86.72' '^<HOST> .* 403'

for a positive result, and

fail2ban-regex '106.37.177.251 - - [08/Sep/2014:17:42:56 -0400] "GET /index.php HTTP/1.1" 404 576 "baidu" "Mozilla/4.0" 204.44.65.173' '^<HOST> .* 403'

for a negative result.

Answer 2

First, your example log entrys are 403s for /, and 404s for /index.php, whereas your regex tries to match php extension and 403 code. There's no wonder you have no match.

So if your interest is only a 403 error entry regardless of path, this should work.

^<HOST> .* "(GET|POST) [^"]+" 403

To debug your regular expression you can use this snippet. Note that <HOST> is preprocessed to (?:::f{4,6}:)?(?P<host>\S+) by fail2ban.