I am using this line to generate a sha1 id for node.js:
crypto.createHash('sha1').digest('hex');
The problem is that it's returning the same id every time.
Is it possible to have it generate a random id each time so I can use it as a database document id?
243,583,606,221,817,150,598,111,409x more entropy
I'd recommend using crypto.randomBytes. It's not
sha1
, but for id purposes, it's quicker, and just as "random".The resulting string will be twice as long as the random bytes you generate; each byte encoded to hex is 2 characters. 20 bytes will be 40 characters of hex.
Using 20 bytes, we have
256^20
or 1,461,501,637,330,902,918,203,684,832,716,283,019,655,932,542,976 unique output values. This is identical to SHA1's 160-bit (20-byte) possible outputs.Knowing this, it's not really meaningful for us to
shasum
our random bytes. It's like rolling a die twice but only accepting the second roll; no matter what, you have 6 possible outcomes each roll, so the first roll is sufficient.Why is this better?
To understand why this is better, we first have to understand how hashing functions work. Hashing functions (including SHA1) will always generate the same output if the same input is given.
Say we want to generate IDs but our random input is generated by a coin toss. We have
"heads"
or"tails"
If
"heads"
comes up again, the SHA1 output will be the same as it was the first timeOk, so a coin toss is not a great random ID generator because we only have 2 possible outputs.
If we use a standard 6-sided die, we have 6 possible inputs. Guess how many possible SHA1 outputs? 6!
It's easy to delude ourselves by thinking just because the output of our function looks very random, that it is very random.
We both agree that a coin toss or a 6-sided die would make a bad random id generator, because our possible SHA1 results (the value we use for the ID) are very few. But what if we use something that has a lot more outputs? Like a timestamp with milliseconds? Or JavaScript's
Math.random
? Or even a combination of those two?!Let's compute just how many unique ids we would get ...
The uniqueness of a timestamp with milliseconds
When using
(new Date()).valueOf().toString()
, you're getting a 13-character number (e.g.,1375369309741
). However, since this a sequentially updating number (once per millisecond), the outputs are almost always the same. Let's take a lookTo be fair, for comparison purposes, in a given minute (a generous operation execution time), you will have
60*1000
or60000
uniques.The uniqueness of
Math.random
Now, when using
Math.random
, because of the way JavaScript represents 64-bit floating point numbers, you'll get a number with length anywhere between 13 and 24 characters long. A longer result means more digits which means more entropy. First, we need to find out which is the most probable length.The script below will determine which length is most probable. We do this by generating 1 million random numbers and incrementing a counter based on the
.length
of each number.By dividing each counter by 1 million, we get the probability of the length of number returned from
Math.random
.So, even though it's not entirely true, let's be generous and say you get a 19-character-long random output;
0.1234567890123456789
. The first characters will always be0
and.
, so really we're only getting 17 random characters. This leaves us with10^17
+1
(for possible0
; see notes below) or 100,000,000,000,000,001 uniques.So how many random inputs can we generate?
Ok, we calculated the number of results for a millisecond timestamp and
Math.random
That's a single 6,000,000,000,000,000,060,000-sided die. Or, to make this number more humanly digestible, this is roughly the same number as
Sounds pretty good, right ? Well, let's find out ...
SHA1 produces a 20-byte value, with a possible 256^20 outcomes. So we're really not using SHA1 to it's full potential. Well how much are we using?
A millisecond timestamp and Math.random uses only 4.11e-27 percent of SHA1's 160-bit potential!
Holy cats, man! Look at all those zeroes. So how much better is
crypto.randomBytes(20)
? 243,583,606,221,817,150,598,111,409 times better.Notes about the
+1
and frequency of zeroesIf you're wondering about the
+1
, it's possible forMath.random
to return a0
which means there's 1 more possible unique result we have to account for.Based on the discussion that happened below, I was curious about the frequency a
0
would come up. Here's a little script,random_zero.js
, I made to get some dataThen, I ran it in 4 threads (I have a 4-core processor), appending the output to a file
So it turns out that a
0
is not that hard to get. After 100 values were recorded, the average wasCool! More research would be required to know if that number is on-par with a uniform distribution of v8's
Math.random
implementationDo it in the browser, too !
You can do this client side in modern browsers, if you'd like
Ok, let's check it out !
Browser requirements
Have a look here: How do I use node.js Crypto to create a HMAC-SHA1 hash? I'd create a hash of the current timestamp + a random number to ensure hash uniqueness: