In my locale (et_EE) [a-z]
means:
abcdefghijklmnopqrsšz
So, 6 ASCII chars (tuvwxy
) and one from Estonian alphabet (ž
) are not included. I see a lot modules which are still using regexes like
/\A[0-9A-Z_a-z]+\z/
For me it seems wrong way to define range of ASCII alphanumeric chars and i think it should be replaced with:
/\A\p{PosixAlnum}+\z/
Is the first one still considered idiomatic way? Or accepted solution? Or a bug?
Or has last one some caveats?
As this question goes beyond Perl, I was interested to find out how it goes in general. Testing this on popular programming languages with native regular expression support, Perl, PHP, Python, Ruby, Java and Javascript, the conclusions are:
[a-z]
will match ASCII-7 a-z range in each and every one of those languages, always, and locale settings do not impact it in any way. Characters likež
andš
are never matched.\w
might or might not match charactersž
andš
, depending on the programming language and parameters given on creation of regular expression. For this expression the variety is greatest, as in some languages they are never matched, irrelevant of options, in others they are always matched and in some it depends.[[:alpha:]]
and Unicode\p{Alpha}
and\p{L}
, if they are supported by the regular expression system of the programming language in question and appropriate configuration is used, will match characters like likež
andš
.Note that "Appropriate configuration" did not require locale change: change of locale did not have impact on results in any of the tested systems.
To be on the safe side, I also tested command line Perl, grep and awk. From there, command line Perl behaves identically to the above. However, grep and awk seem to have behaviour different from others in that for them, locale matters also for
[a-z]
. Behaviour is also version and implementation specific.In that context - grep, awk or similar command line tools - I'd agree that using
a-z
range without the defintion of locale could be considered a bug as you can't really know what you end up with.If we go to more details per language, the status seems to be:
Java
In java,
\p{Alpha}
works like[a-z]
if unicode class is not specified, and unicode alphabetic character if it is, matchingž
.\w
will match characters likež
if unicode flag is present and not if it is not, and\p{L}
will match regardless of unicode flag. There are no locale-aware regular expressions or support for[[alpha]]
.PHP
In PHP
\w
,[[:alpha:]]
and\p{L}
will match characters likež
if unicode switch is present, and not if it is not.\p{Alpha}
is not supported. Locale has no impact on regular expressions.Python
\w
will match mentioned characters if unicode flag is present and locale flag is not present. For unicode strings, unicode flag is assumed by default if Python 3 is used, but not with Python 2. Unicode\p{Alpha}
,\p{L}
or POSIX[[:alpha:]]
are not supported in Python.The modifier to use locale-specific regular expressions apparently only works for character sets with 1 byte per character, making it unusable for unicode.
Perl
\w
matches previously mentioned characters in addition to matching[a-z]
. Unicode\p{Letter}
,\p{Alpha}
and POSIX[[:alpha:]]
are supported and work as expected. Unicode and locale flags for regular expression didn't change the results, and neither did change of locale oruse locale;
/no locale;
.Behavour does not change if we run tests using commandline Perl.
Ruby
[a-z]
and\w
detect just the characters[a-z]
, irrelevant of options. Unicode\p{Letter}
,\p{Alpha}
and POSIX[[:alpha:]]
are supported and working as expected. Locale does not have impact.Javascript
[a-z]
and\w
always detects just the characters[a-z]
. There is support for/u
unicode switch in ECMA2015, which is mostly supported by major browsers, but it does not bring support for[[:alpha:]]
,\p{Alpha}
or\p{L}
or change the behaviour of\w
. The unicode switch does add treatment of unicode characters as one character, which has been a problem before.Situation is the same for client-side javascript as well as Node.js.
AWK
For AWK, there is a longer description of the status posted in article A.8 Regexp Ranges and Locales: A Long Sad Story. It details that in the old world of unix tools,
[a-z]
was the correct way to detect lowercase letters and this is how the tools of the time worked. However, 1992 POSIX introduced locales, and changed the interpretation of character classes so that order of characters was defined per collation order, binding it to locale. This was adopted by AWK of the time as well (3.x series), which resulted in several issues. When 4.x series was developed, POSIX 2008 had defined the order to be undefined, and maintainer reverted back to original behaviour.Nowadays mostly 4.x version of AWK is used. When that is used,
[a-z]
matches a-z ignoring any locale changes, and\w
and[[:alpha:]]
will match locale-specific characters. Unicode \p{Alpha} and \p{L} are not supported.grep
Grep (as well as sed, ed) use GNU Basic Regular Expressions, which is an old flavor. It doesn't have support for unicode character classes.
At least gnu grep 2.16 and 2.25 seems to follow 1992 posix in that locale matters also for
[a-z]
, as well as for\w
and[[:alpha:]]
. This means for example that [a-z] only matches z in set xuzvöä if estonian locale is used.Test code used listed below for each language.
Java (1.8.0_131)
PHP (7.1.5)
Python (3.5.3)
Perl (v5.22.3)
Ruby (ruby 2.2.6p396 (2016-11-15 revision 56800) [x64-mingw32])
Javascript (node.js) (v6.10.3)
Javascript (web browsers)
AWK (GNU Awk 4.1.3)
AWK (GNU Awk 3.1.8)
grep (GNU grep 2.25)
Possible Locale Bugs
The problem you're facing is not with POSIX character classes per se, but with the fact that the classes are dependent on locale. For example, regex(7) says:
The emphasis is mine, but the manual page is clearly saying that the character classes are dependent on locale. Further, wctype(3) says:
In other words, if your locale incorrectly defines a character class, then it's a bug that should be filed against the specific locale. On the other hand, if the character class simply defines the character set in a way that you are not expecting, then it may not be a bug; it may just be a problem that needs to be coded around.
Character Classes as Shortcuts
Character classes are shortcuts for defining sets. You certainly aren't restricted to the pre-defined sets for your locale, and you are free to use the Unicode character sets defined by perlre(1), or simply create the sets explicitly if that provides greater accuracy.
You already know this, so I'm not trying to be pedantic. I'm just pointing out that if you can't or won't fix the locale (which is the source of the problem here) then you should use an explicit set, as you have done.
A convenience class is only convenient if it works for your use case. If it doesn't, toss it overboard!
Back in the old Perl 3.0 days, everything was ASCII, and Perl reflected that.
\w
meant the same thing as[0-9A-Z_a-z]
. And, we liked it!However, Perl is no longer bound to ASCII. I've stopped using
[a-z]
a while ago because I got yelled at when programs I wrote didn't work with languages that weren't English. You must have imagined my surprise as an American to discover that there are at least several thousand people in this world who don't speak English.Perl has better ways of handling
[0-9A-Z_a-z]
anyway. You can use the[[:alnum:]]
set or simply use\w
which should do the right thing. If you must only have lowercase characters, you can use[[:lower:]]
instead of[a-z]
(Which assumes an English type of language). (Perl goes to some lengths to get [a-z] mean just the 26 characters a, b, c, ... z even on EBCDIC platforms.)If you need to specify ASCII only, you can add the
/a
qualifier. If you mean locale specific, you should compile the regular expression within the lexical scope of a 'use locale'. (Avoid the /l modifier, as that applies only to the regular expression pattern, and nothing else. For example in 's/[[:lower:]]/\U$&/lg', the pattern is compiled using locale, but the \U is not. This probably should be considered a bug in Perl, but it is the way things currently work. The /l modifier is really only intended for internal bookkeeping, and should not be typed-in directly.) Actually, it is better to translate your locale data upon input into the program, and translate it back on output, while using Unicode internally. If your locale is one of the new-fashioned UTF-8 ones, a new feature in 5.16 'use locale ":not_characters"' is available to allow the other portions of your locale work seamlessly in Perl.Now, is this a bug? If the program doesn't work as intended, it is a bug plain and simple. If you really want the ASCII sequence,
[a-z]
, then the programmer should have used[[:lower:]]
with the/a
qualifier. If you want all possible lowercase characters including those in other languages, you should simply use[[:lower:]]
.