I'm looking for what the standard, approved, and robust way of stripping invalid characters from strings before writing them to an XML file. I'm talking here about blocks of text containing backspace (^H) and formfeed characters etc.
There has to be a standard library/module function for doing this but I can't find it.
I'm using XML::LibXML to build a DOM tree that I then serialize to disk.
The complete regex for removal of invalid xml-1.0 characters is:
# #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
$str =~ s/[^\x09\x0A\x0D\x20-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]//go;
for xml-1.1 it is:
# allowed: [#x1-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
$str =~ s/[^\x01-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]//go;
# restricted:[#x1-#x8][#xB-#xC][#xE-#x1F][#x7F-#x84][#x86-#x9F]
$str =~ s/[\x01-\x08\x0B-\x0C\x0E-\x1F\x7F-\x84\x86-\x9F]//go;
As almost everyone else has said, use a regular expression. It's honestly not complex enough to be worth adding to a library. Preprocess your text with a substitution.
Your comment about linefeeds above suggests that the formatting is of some importance to you so you will possibly have to decide exactly what you want to replace some characters with.
The list of invalid characters is clearly defined in the XML spec (here - http://www.w3.org/TR/REC-xml/#charsets - for example). The disallowed characters are the ASCII control characters bar carriage return, linefeed and tab. So, you are looking at a 29 character regular expression character class. That's not too bad surely.
Something like:
$text =~ s/[\x00-\x08 \x0B \x0C \x0E-\x19]//g;
should do it.
I've found a solution, but it uses the iconv command instead of perl.
$ iconv -c -f UTF-8 -t UTF-8 invalid.utf8 > valid.utf8
The solutions given above based on regular expressions do not work!!, consider the following example:
$ perl -e 'print "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<root>\x{A0}\x{A0}</root>"' > invalid.xml
$ perl -e 'use XML::Simple; XMLin("invalid.xml")'
invalid.xml:2: parser error : Input is not proper UTF-8, indicate encoding !
Bytes: 0xA0 0xA0 0x3C 0x2F
$ perl -ne 's/[^\x09\x0A\x0D\x20-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]//go; print' invalid.xml > valid.xml
$ perl -e 'use XML::Simple; XMLin("valid.xml")'
invalid.xml:2: parser error : Input is not proper UTF-8, indicate encoding !
Bytes: 0xA0 0xA0 0x3C 0x2F
In fact, the two files invalid.xml and valid.xml are identical.
The thing is that the range "\x20-\x{D7FF}" matches valid representations of those unicode characters, but not e.g. the invalid character sequence "\x{A0}\x{A0}".
Translate is a lot faster than regex substitution. Especially if all you want to do delete characters. Using newt's set:
$string_to_clean =~ tr/\x00-\x08\x0B\x0C\x0E-\x19//d;
A test like this:
cmpthese 1_000_000
, { translate => sub {
my $copy = $text;
$copy =~ tr/\x00-\x08\x0B\x0C\x0E-\x19//d;
}
, substitute => sub {
my $copy = $text;
$copy =~ s/[\x00-\x08\x0B\x0C\x0E-\x19]//g;
}
};
yeilded:
Rate substitute translate
substitute 287770/s -- -86%
translate 2040816/s 609% --
And the more characters I needed to delete the faster tr got in relation.
If you use an XML library to build your XML (as opposed to string concatenation, simple templates, etc), then it should take care of that for you. There is no point in reinventing the wheel.
Okay, this seems to be already answered, but what the hey. If you want to author XML documents, you must use an XML library.
#!/usr/bin/perl
use strict;
use XML::LibXML;
my $doc = XML::LibXML::Document->createDocument('1.0');
$doc->setURI('http://example.com/myuri');
$doc->setDocumentElement($doc->createElement('root-node'));
$doc->documentElement->appendTextChild('text-node',<<EOT);
This node contains &, ñ, á, <, >...
EOT
print $doc->toString;
This produces the following:
$ perl test.pl
<?xml version="1.0"?>
<root-node><text-node> This node contains &, 񬠡, <, >...
</text-node></root-node>
Edit: I now see that you are already using XML::LibXML. This should do the trick.
You could use a Regular expression to remove control characters for example \cH will match \cL or \x08 and \x0C both match backspace and Formfeed respectively.
You can use a simple regex to find and replace all control characters in your chunk of text replacing them either with a space or removing them altogether-
# Replace all control characters with a space
$text =~ s/[[:cntrl:]]/ /g;
# or remove them
$text =~ s/[[:cntrl:]]//g;
I haven't done a lot of work with XML containing "invalid" characters before, but it seems to me you have two completely separate problems here.
First, there are characters in your data that you may not want. You should decide what those are and how you want to remove/replace them independent of any XML restrictions. For instance, you may have things like x^H_y^H_z^H_ where you decide you want to strip both the backspace and the following character. Or it's possible that you in fact don't want to adjust your data but feel forced to by the need to represent it in XML.
Update: I've preserved the following paragraphs for posterity, but they are based on a misunderstanding: I thought you could include any character in XML data so long as you encoded it properly, but it seems there are some characters that are outright verboten, even encoded? XML::LibXML strips these out (at least the current version does so), except for the nul character, which it treats as the end of the string, discarding it and anything that follows :(
Second, you may have characters in your data that you've kept that need encoding in XML. Ideally, whatever XML module you use would do this for you, but if it isn't, you should be able to do it manually, with something like:
use HTML::Entities "encode_entities_numeric";
$encoded_string = encode_entities_numeric( $string, "\x00-\x08\x0B\x0C\x0E-\x19");
But that's really just a stopgap measure. Use a proper XML module; see for instance this answer.
Axeman's right about using tr, but he and newt made a little mistake inverting the XML spec's range of legal characters. http://www.w3.org/TR/REC-xml/#charsets gives
Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
and since the hexadecimal number before \x20 is \x1F (not \x19!) you should use
$string_to_clean =~ tr/\x00-\x08\x0B\x0C\x0E-\x1F//d;
