Perl: Handle malformed UTF-8 strings with Encode::encode

Web Developer in Hamburg, Germany

Perl: Handle malformed UTF-8 strings with Encode::encode

Having the error message “Malformed UTF-8 character (fatal)” in my log files, I tried to handle this properly without letting the process die nor throwing away the whole string.
Having some research on Google I came up with following solution:

sub encode_utf_8 { my $string = @_; my $utf8_encoded = ''; eval { $utf8_encoded = Encode::encode('UTF-8', $string, Encode::FB_CROAK); }; if ($@) { # sanitize malformed UTF-8 $utf8_encoded = ''; my @chars = split(//, $string); foreach my $char (@chars) { my $utf_8_char = eval { Encode::encode('UTF-8', $char, Encode::FB_CROAK) } or next; $utf8_encoded .= $utf_8_char; } } return $utf8_encoded; }

Written by Hinnerk

August 31st, 2011 at 4:58 pm

Posted in English

Tagged with gist, GitHub, OpenSource, Perl, PerlIDS, Web

Hinnerk Altenburg