Perl: Handle malformed UTF-8 strings with Encode::encode
Having the error message “Malformed UTF-8 character (fatal)” in my log files, I tried to handle this properly without letting the process die nor throwing away the whole string.
Having some research on Google I came up with following solution:
sub encode_utf_8 { my $string = @_;
my $utf8_encoded = ''; eval { $utf8_encoded = Encode::encode('UTF-8', $string, Encode::FB_CROAK); }; if ($@) { # sanitize malformed UTF-8 $utf8_encoded = ''; my @chars = split(//, $string); foreach my $char (@chars) { my $utf_8_char = eval { Encode::encode('UTF-8', $char, Encode::FB_CROAK) } or next; $utf8_encoded .= $utf_8_char; } } return $utf8_encoded;}See also:
http://perldoc.perl.org/Encode.html#Handling-Malformed-Data
http://www.perlmonks.org/?node_id=839519
