Efficiently detect missing dates in array and inje

Question

I'm using highcharts.js to visualize data series from a database. There's lots of data series and they can potantially change from the database they are collected from with ajax. I can't guarantee that they are flawless and sometimes they will have blank gaps in the dates, which is a problem. Highcharts simply draws a line through the entire gap to the next available date, and that's bad in my case.

The series exists in different resolutions. Hours, Days and Weeks. Meaning that a couple of hours, days or weeks can be missing. A chart will only show 1 resolution at a time on draw, and redraw if the resolution is changed.

The 'acutal' question is how to get highcharts to not draw those gaps in an efficient way that works for hous, days and weeks

I know highcharts (line type) can have that behaviour where it doesn't draw a single line over a gap if the gap begins with a null.

What I tried to do is use the resolution (noted as 0, 1, 2 for hour day or week), to loop through the array that contains the values for and detect is "this date + 1 != (what this date + 1 should be)

The code where I need to work this out is here. Filled with psudo

for (var k in data.values) {
    //help start, psudo code.
    if(object-after-k != k + resolution){ //The date after "this date" is not as expected
       data.values.push(null after k) 
    }

    //help end
    HC_datamap.push({ //this is what I use to fill the highchart later, so not important
        x: Date.parse(k),
        y: data.values[k]
    });
}

the k objects in data.values look like this

2015-05-19T00:00:00
2015-05-20T00:00:00
2015-05-21T00:00:00
...and more dates

as strings. They can number in thousands, and I don't want the user to have to wait forever. So performance is an issue and I'm not an expert here either

Please ask away for clarifications.

Answer 1

Personally, object with property names as dates may be a bit problematic, I think. Instead I would create an array of data. Then simple loop to fill gaps shouldn't be very slow. Example: http://jsfiddle.net/4mxtvotv/ (note: I'm changing format to array, as suggested).

var origData = {
        "2015-05-19T00:00:00": 20,
        "2015-05-20T00:00:00": 30,
        "2015-05-21T00:00:00": 50,
        "2015-06-21T00:00:00": 50,
        "2015-06-22T00:00:00": 50
};

// let's change to array format
var data = (function () {
    var d = [];
    for (var k in origData) {
        d.push([k, origData[k]]);
    }
    return d;
})();
var interval = 'Date'; //or Hour or Month or Year etc.

function fillData(data, interval) {
    var d = [],
        now = new Date(data[0][0]), // first x-point
        len = data.length,
        last = new Date(data[len - 1][0]), // last x-point
        iterator = 0,
        y;

    while (now <= last) { // loop over all items
        y = null;
        if (now.getTime() == new Date(data[iterator][0]).getTime()) { //compare times 
            y = data[iterator][1]; // get y-value 
            iterator++; // jump to next date in the data
        }
        d.push([now.getTime(), y]); // set point
        now["set" + interval](now.getDate() + 1); // jump to the next period
    }
    return d;
}

var chart = new Highcharts.StockChart({
    chart: {
        renderTo: 'container'
    },
    series: [{
        data: fillData(data, interval)
    }]
}); 

Second note: I'm using Date.setDay() or Date.setMonth(), of course if your data is UTC-based, then should be: now["setUTC" + interval].

Answer 2

I wrote this loop.

In my case my data is always keyed to a date (12am) and it moves either in intervals of 1 day, 1 week or 1 month. Its designed to work on an already prepared array of points ({x,y}). Thats what dataPoints is, these are mapped to finalDataPoints which also gets the nulls. finalDataPoints is what is ultimately used as the series data. This is using momentjs, forwardUnit is the interval (d, w, or M).

It assumes that the data points are already ordered from earliest x to foremost x.

dataPoints.forEach(function (point, index) {
    var plotDate = moment(point.x);
    finalDataPoints.push(point);

    var nextPoint = dataPoints[index+1];
    if (!nextPoint) {
        return;
    }

    var nextDate = moment(nextPoint.x);
    while (plotDate.add(1, forwardUnit).isBefore(nextDate)) {
        finalDataPoints.push({x: plotDate.toDate(), y: null});
    }
});