De Bruijn algorithm binary digit count 64bits C#

问题:

Im using the "De Bruijn" Algorithm to discover the number of digits in binary that a big number (up to 64bits) has.

For example:

• 1022 has 10 digits in binary.
• 130 has 8 digits in binary.

I found that using a table lookup based on De Bruijn give me the power to calculate this x100 times faster than conventional ways (power, square, ...).

According to this website, 2^6 has the table to calculate the 64 bits numbers. this would be the table exposed in c#

static readonly int[] MultiplyDeBruijnBitPosition2 = new int[64]
{
  0,1,2,4,8,17,34,5,11,23,47,31,63,62,61,59,
  55,46,29,58,53,43,22,44,24,49,35,7,15,30,60,57,
  51,38,12,25,50,36,9,18,37,10,21,42,20,41,19,39,
  14,28,56,48,33,3,6,13,27,54,45,26,52,40,16,32
};

(I dont know if i brought the table from that website correctly) Then, based on the R.. comment here. I should use this to use the table with the input uint64 number.

public static int GetLog2_DeBruijn(ulong v)
{
return MultiplyDeBruijnBitPosition2[(ulong)(v * 0x022fdd63cc95386dull) >> 58];
}

But the c# compiler doesnt allow me to use "0x022fdd63cc95386dull" because it overflows 64bits. And i have to use "0x022fdd63cc95386d" instead.

Using those codes. The problem is that i am not getting the correct result for the input given.

For example, doing 1.000.000 calculations of the number: 17012389719861204799 (64bits used) This is the result:

• Using pow2 method i get the result 64 1 Million times in 1380ms.
• Using DeBruijn method i get the result 40 1 Million times in 32ms. (Dont know why 40)

Im trying to understand how "De Bruijn" works, and how can i fix this and create a final code for c# to calculate up to 64bits numbers.

UDPATE and benchmarks of different solutions

I was looking for the fastest algorithm to get the number of digits in binary that a unsigned given number of 64bits has in c# (known as ulong).

For example:

• 1024 has 11 binary digits. (2^10+1) or (log2[1024]+1)
• 9223372036854775808 has 64 binary digits. (2^63+1) or (log2[2^63]+1)

The conventional power of 2 and square is extremely slow. and just for 10000 calculations it needs 1500ms to get the answer. (100M calculations needs hours).

Here, Niklas B., Jim Mischel, and Spender brought differents methods to make this faster.

• SIMD and SWAR Techniques //Provided by Spender (Answer here)
• De_Bruijn Splited 32bits //Provided by Jim Mischel (Answer here)
• De_Bruijn 64bits version //Provided by Niklas B. (Answer here)
• De_Bruijn 128bits version //Also provided by Niklas B. (Answer here)

Testing this Methods with a CPU Q6600 overclocked to 3Ghz using Windows 7 (64bits) Gives the following results.

As you can see, it takes just a few seconds to find correctly 100,000,000 of request given, being De_Bruijn 128bits version the fastest.

Thanks a lot to all of you, you help me a lot with this. I hope this helps you too.

回答1:

You should check R..'s answer and his resource again. The question that he responded to was how to find the log2 for powers of two.

The bit twiddling website says that the simple multiplication + shift only works "If you know that v is a power of 2". Otherwise you need to round up to the next power of two first:

static readonly int[] bitPatternToLog2 = new int[64] { 
    0, // change to 1 if you want bitSize(0) = 1
    1,  2, 53,  3,  7, 54, 27, 4, 38, 41,  8, 34, 55, 48, 28,
    62,  5, 39, 46, 44, 42, 22,  9, 24, 35, 59, 56, 49, 18, 29, 11,
    63, 52,  6, 26, 37, 40, 33, 47, 61, 45, 43, 21, 23, 58, 17, 10,
    51, 25, 36, 32, 60, 20, 57, 16, 50, 31, 19, 15, 30, 14, 13, 12
}; // table taken from http://chessprogramming.wikispaces.com/De+Bruijn+Sequence+Generator
static readonly ulong multiplicator = 0x022fdd63cc95386dUL;

public static int bitSize(ulong v) {
    v |= v >> 1;
    v |= v >> 2;
    v |= v >> 4;
    v |= v >> 8;
    v |= v >> 16;
    v |= v >> 32;
    // at this point you could also use popcount to find the number of set bits.
    // That might well be faster than a lookup table because you prevent a 
    // potential cache miss
    if (v == (ulong)-1) return 64;
    v++;
    return MultiplyDeBruijnBitPosition2[(ulong)(v * multiplicator) >> 58];
}

Here is a version with a larger lookup table that avoids the branch and one addition. I found the magic number using random search.

static readonly int[] bitPatternToLog2 = new int[128] {
    0, // change to 1 if you want bitSize(0) = 1
    48, -1, -1, 31, -1, 15, 51, -1, 63, 5, -1, -1, -1, 19, -1, 
    23, 28, -1, -1, -1, 40, 36, 46, -1, 13, -1, -1, -1, 34, -1, 58,
    -1, 60, 2, 43, 55, -1, -1, -1, 50, 62, 4, -1, 18, 27, -1, 39, 
    45, -1, -1, 33, 57, -1, 1, 54, -1, 49, -1, 17, -1, -1, 32, -1,
    53, -1, 16, -1, -1, 52, -1, -1, -1, 64, 6, 7, 8, -1, 9, -1, 
    -1, -1, 20, 10, -1, -1, 24, -1, 29, -1, -1, 21, -1, 11, -1, -1,
    41, -1, 25, 37, -1, 47, -1, 30, 14, -1, -1, -1, -1, 22, -1, -1,
    35, 12, -1, -1, -1, 59, 42, -1, -1, 61, 3, 26, 38, 44, -1, 56
};
static readonly ulong multiplicator = 0x6c04f118e9966f6bUL;

public static int bitSize(ulong v) {
    v |= v >> 1;
    v |= v >> 2;
    v |= v >> 4;
    v |= v >> 8;
    v |= v >> 16;
    v |= v >> 32;
    return bitPatternToLog2[(ulong)(v * multiplicator) >> 57];
}

You should definitely check other tricks to compute the log2 and consider using the MSR assembly instruction if you are on x86(_64). It gives you the index of the most significant set bit, which is exactly what you need.

回答2:

After perusing various bit-twiddling info, this is how I'd do it... don't know how this stacks up next to DeBruijn, but should be considerably faster than using powers.

ulong NumBits64(ulong x)
{
    return (Ones64(Msb64(x) - 1ul) + 1ul);
}

ulong Msb64(ulong x)
{  
    //http://aggregate.org/MAGIC/
    x |= (x >> 1);
    x |= (x >> 2);
    x |= (x >> 4);
    x |= (x >> 8);
    x |= (x >> 16);
    x |= (x >> 32);
    return(x & ~(x >> 1));
}

ulong Ones64(ulong x)
{
    //https://chessprogramming.wikispaces.com/SIMD+and+SWAR+Techniques
    const ulong k1 = 0x5555555555555555ul;
    const ulong k2 = 0x3333333333333333ul;
    const ulong k4 = 0x0f0f0f0f0f0f0f0ful;
    x = x - ((x >> 1) & k1);
    x = (x & k2) + ((x >> 2) & k2);
    x = (x + (x >> 4)) & k4;
    x = (x * 0x0101010101010101ul) >> 56;
    return x;
}

回答3:

When I looked into this a while back for 32 bits, the DeBruijn sequence method was by far the fastest. See https://stackoverflow.com/a/10150991/56778

What you could do for 64 bits is split the number into two 32-bit values. If the high 32 bits is non-zero, then run the DeBruijn calculation on it, and then add 32. If the high 32 bits is zero, then run the DeBruijn calculation on the low 32 bits.

Something like this:

int NumBits64(ulong val)
{
    if (val > 0x00000000FFFFFFFFul)
    {
        // Value is greater than largest 32 bit number,
        // so calculate the number of bits in the top half
        // and add 32.
        return 32 + GetLog2_DeBruijn((int)(val >> 32));
    }
    // Number is no more than 32 bits,
    // so calculate number of bits in the bottom half.
    return GetLog2_DeBruijn((int)(val & 0xFFFFFFFF));
}

int GetLog2_DeBruijn(int val)
{
    uint32 v = (uint32)val;
    int r;      // result goes here

    static const int MultiplyDeBruijnBitPosition[32] = 
    {
        0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30,
        8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31
    };

    v |= v >> 1; // first round down to one less than a power of 2 
    v |= v >> 2;
    v |= v >> 4;
    v |= v >> 8;
    v |= v >> 16;

    r = MultiplyDeBruijnBitPosition[(uint32_t)(v * 0x07C4ACDDU) >> 27];
    return r;
}

回答4:

Edit: This solution is not recommanded as it requires branching for Zero.

After reading Niklas B's answer I spent a few hours on researching this, and realize all magic multiplicator has to be in the last nth in order to suit for 64-elements lookup table (I don't have the necessary knowledge to explain why).

So I used exactly the same generator mentioned by that answer to find the last sequence, here is the C# code:

// used generator from http://chessprogramming.wikispaces.com/De+Bruijn+Sequence+Generator
static readonly byte[] DeBruijnMSB64table = new byte[]
{
    0 , 47, 1 , 56, 48, 27, 2 , 60,
    57, 49, 41, 37, 28, 16, 3 , 61,
    54, 58, 35, 52, 50, 42, 21, 44,
    38, 32, 29, 23, 17, 11, 4 , 62,
    46, 55, 26, 59, 40, 36, 15, 53,
    34, 51, 20, 43, 31, 22, 10, 45,
    25, 39, 14, 33, 19, 30, 9 , 24,
    13, 18, 8 , 12, 7 , 6 , 5 , 63,
};
// the cyclc number has to be in the last 16th of all possible values
// any beyond the 62914560th(0x03C0_0000) should work for this purpose
const ulong DeBruijnMSB64multi = 0x03F79D71B4CB0A89uL; // the last one
public static byte GetMostSignificantBit(this ulong value)
{
    value |= value >> 1;
    value |= value >> 2;
    value |= value >> 4;
    value |= value >> 8;
    value |= value >> 16;
    value |= value >> 32;

    return DeBruijnMSB64table[value * DeBruijnMSB64multi >> 58];
}