utf8 on in dancer but not script

Question

This is a follow up to my previous question on showing unicode string differences. As it turns out the strings appear to be the same, however in one of them the UTF8 flag is on.

SV = PVMG(0x4cca750) at 0x4b3fc90
 REFCNT = 1
 FLAGS = (PADMY,POK,pPOK,UTF8)
 IV = 0
 NV = 0
 PV = 0x1eda410 "flurbe"\0 [UTF8 "flurbe"]
 CUR = 6
 LEN = 16

vs

SV = PV(0xf28090) at 0xf4b6a0
 REFCNT = 1
 FLAGS = (PADMY,POK,pPOK)
 PV = 0xf37b90 "flurbe"\0
 CUR = 6
 LEN = 16

This appears to make a difference between the resulting sha512 hashes when I encrypt the string. Dancer is what is causing the first result to have utf8 as far as I can tell, my other script is simply a command line one, without using dancer in that how can I force it to behave in the same way?

Answer 1

You have an encoding problem, namely the lack thereof. The digest functions operate on octets. You give it characters, which is wrong.

Course of action: encode your characters into octets. UTF-8 is a suitable encoding.

my $octets = Encode::encode('UTF-8', $characters, Encode::FB_CROAK);
# add salt to octets
# produce digest

Answer 2

(This is more of a comment than an answer, but it's too big.)

I just ran this program:

#!/usr/bin/perl -w

use warnings;
use strict;

use Devel::Peek ();
use Digest::SHA ();

my $x = 'flurbe';

Devel::Peek::Dump $x;

print Digest::SHA::sha512_hex($x), "\n\n";

utf8::upgrade $x;

Devel::Peek::Dump $x;

print Digest::SHA::sha512_hex($x), "\n";

__END__

and it gave this output:

SV = PV(0x10441040) at 0x10491638
  REFCNT = 1
  FLAGS = (PADMY,POK,pPOK)
  PV = 0x10449ca0 "flurbe"\0
  CUR = 6
  LEN = 8
1cd2e71e55653caeb6c9bffa47a66ff1c9b526bbb732dcff28412090601e9b5e34d36be6a0267527347cd94039b383d4bc45653d786d1041debe7faa0716bdf1

SV = PV(0x10441040) at 0x10491638
  REFCNT = 1
  FLAGS = (PADMY,POK,pPOK,UTF8)
  PV = 0x10449ca0 "flurbe"\0 [UTF8 "flurbe"]
  CUR = 6
  LEN = 8
1cd2e71e55653caeb6c9bffa47a66ff1c9b526bbb732dcff28412090601e9b5e34d36be6a0267527347cd94039b383d4bc45653d786d1041debe7faa0716bdf1

As you can see, Devel::Peek::Dump correctly identifies that the string has been upgraded to UTF-8, but this doesn't affect the SHA-512 hash computed by Digest::SHA.

Edited to add: In a comment above, you mention that your "hashes are random salted". Can these salts include bytes outside the ASCII range? If so, concatenation with a UTF-8-upgraded string can affect their contents. I just ran this modified program:

#!/usr/bin/perl -w

use warnings;
use strict;

use Devel::Peek ();
use Digest::SHA ();

my $x = 'flurbe';
my $y = "\xA0";      # a single byte, hex 00A0
my $z = "\xC2\xA0";  # UTF-8 representation of U+00A0, as a byte-string

Devel::Peek::Dump "$x$y";
print Digest::SHA::sha512_hex("$x$y"), "\n\n";

Devel::Peek::Dump "$x$z";
print Digest::SHA::sha512_hex("$x$z"), "\n\n";

utf8::upgrade $x;

Devel::Peek::Dump "$x$y";

print Digest::SHA::sha512_hex("$x$y"), "\n";

__END__

and it gave this output:

SV = PV(0x104410e8) at 0x104d68d8
  REFCNT = 1
  FLAGS = (PADTMP,POK,pPOK)
  PV = 0x10449ca0 "flurbe\240"\0
  CUR = 7
  LEN = 8
1901f989ed76143697ecc6683fd03ec793bc126d51cdbee0a72241933136c144f2e602828abddc7e4843df5542a099be92313fa5874d1d2dc54ecdd1ff308c5e

SV = PV(0x104d80b8) at 0x104ec098
  REFCNT = 1
  FLAGS = (PADTMP,POK,pPOK)
  PV = 0x10489170 "flurbe\302\240"\0
  CUR = 8
  LEN = 12
072f7b54c80fa8062ca1d17727a88c9ff4815f83c1166471331c6398b9140a06812eff341c98453f4c51356926dbe9694cbcbebfe4cda7e77cf68008ab838c6d

SV = PV(0x104d80a8) at 0x104f0f98
  REFCNT = 1
  FLAGS = (PADTMP,POK,pPOK,UTF8)
  PV = 0x104896c8 "flurbe\302\240"\0 [UTF8 "flurbe\x{a0}"]
  CUR = 8
  LEN = 12
072f7b54c80fa8062ca1d17727a88c9ff4815f83c1166471331c6398b9140a06812eff341c98453f4c51356926dbe9694cbcbebfe4cda7e77cf68008ab838c6d

As you can see, the SHA-512 hash of "$x$y" depends on whether $x was UTF-8-upgraded. "$x$y" with a UTF-8-upgraded $x gives the same SHA-512 hash as does "$x$z" with a non-UTF-8-upgraded $x. This is because SHA-512 operates on bytes, not characters, and the concatenation of a UTF-8-upgraded string with a byte-string causes the byte-string to be UTF-8-upgraded.