Improving performance of for loop when decimating

问题:

I am collecting 16384 double values from a hardware device using a signal processing library. The 16384 values I receive are the output of a low pass filter. I want to down sample by a factor of 10. In other words, I want to keep 1 point out of every 10. So in general it works like this:

1. I get a buffer from my library when the low pass filter has completed.

2. I then collect 10 of these buffers.

3. When all 10 buffers are collected into one buffer that has 10*16384 doubles I then loop collecting every 10th double from the buffer. The result is one buffer with 16384 doubles. This will sent off for the rest of the data processing.

Here is the code:

double[] rawData = new double[163840];
int bufferCount = 0;

private void genericReal2_ProcessData(object Sender, Mitov.SignalLab.ProcessRealNotifyArgs Args)
{
    var realbuffer = Args.InBuffer; //get the buffer of processed doubles

    var myData = realbuffer.ToArray(); //must be converted to an array since the data type is not quite an array of doubles.

    Array.Copy(myData, 0, rawData, bufferCount * 16384, 16384);

    bufferCount++;

    if (bufferCount == 10)
    {
        bufferCount = 0;

        int j = 0;

        for (int i = 0; i < 163840; i += 10) //this loop takes 20ms
        {
            realbuffer[j] = rawData[i];                   
            j++;
        }

        genericReal2.SendData(realbuffer); //send data off for further processing
    }
}

The for loop takes about 20ms to run whereas everything else is around 20uS.
So, is there some way I could improve the overall performance of this without using the for loop?

Update************************** I have determined that all the processing time in the loop is taken assigning realbuffer to rawData. So I changed this as follows:

    private void genericReal2_ProcessData(object Sender, Mitov.SignalLab.ProcessRealNotifyArgs Args)
    {
        double[] finalBuffer = new double[16384];

        var realbuffer = Args.InBuffer; //get the buffer of processed doubles

        var myData = realbuffer.ToArray(); //must be converted to an array since the data type is not quite an array of doubles.

        Array.Copy(myData, 0, rawData, bufferCount * 16384, 16384);

        bufferCount++;

        if (bufferCount == 10)
        {
            bufferCount = 0;

            int j = 0;

            for (int i = 0; i < 163840; i += 10)
            {
                finalBuffer[j] = rawData[i];
                j++;
            }

             var pointer= realbuffer.Read();
             //I can get a pointer to the realbuffer.
             //It stores 8 bytes for every double value
             how can I copy 16384 8 byte values from finalbuffer to realbuffer?


            genericReal2.SendData(realbuffer); //send data off for further processing
        }

回答1:

This is solved. The libraries I use have a function which will assign all values from a double[] to it's internal real buffer

 realbuffer.Equals(finalBuffer);

This takes 50us... Thanks for all the help.

回答2:

You are actually getting everything and then drop the unnecessary ones. How about dropping those without copying at first step. I mean rather than using Array.Copy just copy the needed ones like;

int j = bufferCount * 16384;
for (int i = 0; i < 16384; i += 10)
{
    realbuffer[j++] = rawData[i];
}

By doing so you will not need to run the loop inside if statement.

If you cannot change the overall structure of you code due to any condition, you can increase the loop performance by a technique that is used in sorting algorithms . Define your array with the size 163840 + 1. Assign some value to the last position of your array such as -1 which will not be inside the received data. And change the loop as follows which decreases the number of comparison in the execution of the loop;

for (int i = 0; i < rawData[i+=10 ] !=-1; ) 
{
    realbuffer[j] = rawData[i];
    j++;
}

Hope it helps.

回答3:

You could try putting each buffer into a dictionary and processing them in a for each. Theoretically this should be faster because the arrays will be processed in parallel. 20ms is pretty fast, why the need for such speed?

var buffers = new Dictionary<int, double[]>();

Then process something like this:

var myData = realbuffer.ToArray();

buffers.Add(bufferCount, myData);

if (bufferCount == 10)
{
   Parallel.ForEach(buffers, (buffer) =>
   {
      //process buffer independently
   });
}