How to filter a very large bootstrap table using p

Question

I've built a large table in bootstrap, about 5,000 rows x 10 columns, and I need to filter the table for specific attributes, fast, using only JavaScript. The table has both an id column and an attribute column, i.e.

id | attr | ...
---------------
2  |  X   | ...
3  |  Y   | ...
4  |  X   | ...

To make the filtering process fast, I built a hashtable table that maps the attributes back to the column ids. So for example, I have a mapping:

getRowIds["X"] = [2,4]

The user can enter the attribute "X" in a search box, the hashtable then looks up the corresponding rows that contain "X" (2 and 4 in this case), and then calls the following functions via a map operation:

this.hideRow = function(id) {
    document.getElementById(id).style.display="none"
}

this.showRow = function(id) {
    document.getElementById(id).style.display=""
}

This process is still quite slow, as the user is allowed to select multiple attributes (say X,Y).

Is there a faster way of hiding the rows?

Would it be faster if I could somehow detach the table from the DOM, make the changes, and then re-attach? How do I do this in javascript?

Are there other more efficient/smarter ways of doing the filtering?

Thanks :)

Answer 1

Here is a on the fly filter solution, that filter the table using letters typed in input box on keypress event.

Though right now I am using DataTables in my current project development, yet if you want a strict javascript solution here is it. It may not be the best optimized but works good.

function SearchRecordsInTable(searchBoxId, tableId) {
    var searchText = document.getElementById(searchBoxId).value;
    searchText = searchText.toLowerCase();
    var targetTable = document.getElementById(tableId);
    var targetTableColCount;

    //Loop through table rows
    for (var rowIndex = 0; rowIndex < targetTable.rows.length; rowIndex++) {
        var rowData = '';

        //Get column count from header row
        if (rowIndex == 0) {
            targetTableColCount = targetTable.rows.item(rowIndex).cells.length;
            continue; //do not execute further code for header row.
        }

        //Process data rows. (rowIndex >= 1)
        for (var colIndex = 0; colIndex < targetTableColCount; colIndex++) {
            rowData += targetTable.rows.item(rowIndex).cells.item(colIndex).textContent;
            rowData = rowData.toLowerCase();
        }
        console.log(rowData);

        //If search term is not found in row data
        //then hide the row, else show
        if (rowData.indexOf(searchText) == -1)


            targetTable.rows.item(rowIndex).style.display = 'none';
        else
            targetTable.rows.item(rowIndex).style.display = '';
    }
}

Cheers!!

Answer 2

see this link it might help, the only problem is its not in pure javascript it also uses angularjs.

    app.service("NameService", function($http, $filter){

  function filterData(data, filter){
    return $filter('filter')(data, filter)
  }

  function orderData(data, params){
    return params.sorting() ? $filter('orderBy')(data, params.orderBy()) : filteredData;
  }

  function sliceData(data, params){
    return data.slice((params.page() - 1) * params.count(), params.page() * params.count())
  }

  function transformData(data,filter,params){
    return sliceData( orderData( filterData(data,filter), params ), params);
  }

  var service = {
    cachedData:[],
    getData:function($defer, params, filter){
      if(service.cachedData.length>0){
        console.log("using cached data")
        var filteredData = filterData(service.cachedData,filter);
        var transformedData = sliceData(orderData(filteredData,params),params);
        params.total(filteredData.length)
        $defer.resolve(transformedData);
      }
      else{
        console.log("fetching data")
        $http.get("data.json").success(function(resp)
        {
          angular.copy(resp,service.cachedData)
          params.total(resp.length)
          var filteredData = $filter('filter')(resp, filter);
          var transformedData = transformData(resp,filter,params)

          $defer.resolve(transformedData);
        });  
      }

    }
  };
  return service;  
});

Answer 3

Using AngularJS can indeed be a good idea, which lets us render your rows as simple as

<tr ng-repeat="row in rowArray">
  <td>{{row.id}}</td>
  <td>{{row.attr}}</td>
</tr>

where you only need to supply your rowArray as array of objects like {id: 1, attr: 'X'}, see the documentation for ng-repeat directive. One of Angular's big powers lies in its extremely compact code.

Among other things, Angular also has powerful filter building library to filter and sort your rows on the fly right inside your HTML:

<tr ng-repeat="row in rowArray | yourCustomFilter:parameters">
  <td>{{row.id}}</td>
  <td>{{row.attr}}</td>
</tr>

Having said that, it'll clearly be a performance drag to throw 5K rows into your array. That would create a huge HTML in your browser memory that, however, will not fit into your viewport. Then there is no point to have it in the memory if you can't show it anyway. Instead you only want to have the viewable part in your memory plus possibly a few more rows around.

Have a look at the directive "Scroll till you drop" provided by Angular UI Utils - it does exactly that!

Pagination as mentioned in another answer is surely a valid alternative to the infinite scroll. There is lot written on the web about strengths and weaknesses of pagination vs infinite scroll if you want to dig into that.

---

Speaking of your code specifically, it has other performance drags. For instance, on each call, this function

document.getElementById(id).style.display="none"  

will look up the DOM for the element by its id, and then will look up its property .style (which can be a drag if the JavaScript needs to go high up in the Prototype chain). You could do much better performance wise by caching direct reference links to the display properties, which are the ones you really need.

---

EDIT. By caching here I mean pre-compiling a hash linking id with the interesting properties:

hash[id] = document.getElementById(id).style.display

Then you switch the style by simple setting:

hash[id] = 'none'
hash[id] = 'block'

This way of calculating hash assumes that your elements are all inside the DOM, which is bad for performance, but there are better ways!

Libraries like jQuery and, of course, Angular :) will let you create your HTML elements with their complete style properties but without attaching them to the DOM. That way you are not overloading your browser's capacity. But you can still cache them! So you will cache your HTML (but not DOM) Elements and their Display like that:

elem[id] = $('<tr>' +
  '<td>' + id + '</td>' +
  '<td>' + attr + '</td>' +
</tr>');

display[id] = elem[id].style.display;

and then attach/ detach your elements to the DOM as you go and update their display properties using the display cache.

Finally note that for better performance, you want to concatenate your rows in a bundle first, and only then attach in a single jump (instead of attaching one-by-one). The reason is, every time your change the DOM, the browser has to do a lot of recalculation to adjust all other DOM elements correctly. There is a lot going on there, so you want to minimize those re-calculations as much as possible.

---

POST EDIT.

To illustrate by an example, if parentElement is already in your DOM, and you want to attach an array of new elements

elementArray = [rowElement1, ..., rowElementN]

the way you want to do it is:

var htmlToAppend = elementArray.join('');

parentElement.append(htmlToAppend);

as opposed to running a loop attaching one rowElement at a time.

Another good practice is to hide your parentElement before attaching, then only show when everything is ready.

Answer 4

I would ask

• Why you want to write this code for yourself? From personal experience, trying to filter efficiently and on all browsers is a non-trivial task.
• If you are doing this as a learning experience, then look at source of the packages listed below as examples.
• With 5000 rows, it would be more efficient to do server side filtering and sorting. Then use ajax to update the displayed table.

I would suggest that you look at using one of the several JavaScript packages that already do this. There are many more packages that the two below. I'm showing these two as examples of what is available.

• http://datatables.net/ - This is a very full featured package that handles both client and server side filtering and sorting.
• http://www.listjs.com/ - is a lightweight client side filtering and sorting package.

Answer 5

I whipped up a filtering solution that you might want to check out.

Features

• can process a 5000 row table almost instantly*
• uses plain old JavaScript; no need for libraries
• no new syntax to learn; using it is as easy as calling a function
• works fine with your preexisting table; no need to start from scratch
• no data structures or caching required
• supports multiple values per filter and multiple filters
• supports inclusive and exclusive filtering
• works just as well on a table that's detached from the DOM if you want to apply filters before displaying it.

---

How it works

The JavaScript is very simple. All it does is create a unique class name for each filter and add it to every row that matches the filter parameters. The class names can be used to determine which rows a given filter is currently filtering, so there's no need to store that information in a data structure. The classes share a common prefix, so they can all be targeted by the same CSS selector for applying the display: none declaration. Removing a filter is as simple as removing its associated class name from the rows that have it.

---

The Code

If you want to show only rows that have a value of "X" or "Y" in column 2, the function call would look something like this:

addFilter(yourTable, 2, ['X','Y']);

That's all there is to it! Instructions on removing a filter can be found in the demo code below.

---

Demo

The demo in the code snippet below allows you to apply any number of filters with any number of values to a 5000 row table like the one the OP described, and remove them afterward. It may look like a lot of code, but most of it is just for setting up the demo interface. If you were to use this solution in your own code, you'd probably just copy over the first two js functions (addFilter and removeFilter), and the first CSS rule (the one with display: none).

/*
The addFilter function is ready to use and should work with any table. You just need
to pass it the following arguments:
  1) a reference to the table
  2) the numeric index of the column to search
  3) an array of values to search for
Optionally, you can pass it a boolean value as the 4th argument; if true, the filter
will hide rows that DO contain the specified values rather than those that don't (it
does the latter by default). The return value is an integer that serves as a unique
identifier for the filter. You'll need to save this value if you want to remove the
filter later.
*/
function addFilter(table, column, values, exclusive) {
  if(!table.hasAttribute('data-filtercount')) {
    table.setAttribute('data-filtercount', 1);
    table.setAttribute('data-filterid', 0);
    var filterId = 0;
  }
  else {
    var
      filterCount = parseInt(table.getAttribute('data-filtercount')) + 1,
      filterId = filterCount === 1 ?
        0 : parseInt(table.getAttribute('data-filterid')) + 1;
    table.setAttribute('data-filtercount', filterCount);
    table.setAttribute('data-filterid', filterId);
  }
  exclusive = !!exclusive;
  var
    filterClass = 'filt_' + filterId,
    tableParent = table.parentNode,
    tableSibling = table.nextSibling,
    rows = table.rows,
    rowCount = rows.length,
    r = table.tBodies[0].rows[0].rowIndex;
  if(tableParent)
    tableParent.removeChild(table);
  for(; r < rowCount; r++) {
    if((values.indexOf(rows[r].cells[column].textContent.trim()) !== -1) === exclusive)
      rows[r].classList.add(filterClass);
  }
  if(tableParent)
    tableParent.insertBefore(table, tableSibling);
  return filterId;
}

/*
The removeFilter function takes two arguments:
  1) a reference to the table that has the filter you want to remove
  2) the filter's ID number (i.e. the value that the addFilter function returned)
*/
function removeFilter(table, filterId) {
  var
    filterClass = 'filt_' + filterId,
    tableParent = table.parentNode,
    tableSibling = table.nextSibling,
    lastId = table.getAttribute('data-filterid'),
    rows = table.querySelectorAll('.' + filterClass),
    r = rows.length;
  if(tableParent)
    tableParent.removeChild(table);
  for(; r--; rows[r].classList.remove(filterClass));
  table.setAttribute(
    'data-filtercount',
    parseInt(table.getAttribute('data-filtercount')) - 1
  );
  if(filterId == lastId)
    table.setAttribute('data-filterid', parseInt(filterId) - 1);
  if(tableParent)
    tableParent.insertBefore(table, tableSibling);
}

/*
THE REMAINING JS CODE JUST SETS UP THE DEMO AND IS NOT PART OF THE SOLUTION, though it
does provide a simple example of how to connect the above functions to an interface.
*/
/* Initialize interface. */
(function() {
  var
    table = document.getElementById('hugeTable'),
    addFilt = function() {
      var
        exclusive = document.getElementById('filterType').value === '0' ? true : false,
        colSelect = document.getElementById('filterColumn'),
        valInputs = document.getElementsByName('filterValue'),
        filters = document.getElementById('filters'),
        column = colSelect.value,
        values = [],
        i = valInputs.length;
      for(; i--;) {
        if(valInputs[i].value.length) {
          values[i] = valInputs[i].value;
          valInputs[i].value = '';
        }
      }
      filters.children[0].insertAdjacentHTML(
        'afterend',
        '<div><input type="button" value="Remove">'
        + colSelect.options[colSelect.selectedIndex].textContent.trim()
        + (exclusive ? '; [' : '; everything but [') + values.toString() + ']</div>'
      );
      var
        filter = filters.children[1],
        filterId = addFilter(table, column, values, exclusive);
      filter.children[0].addEventListener('click', function() {
        filter.parentNode.removeChild(filter);
        removeFilter(table, filterId);
      });
    },
    addFiltVal = function() {
      var input = document.querySelector('[name="filterValue"]');
      input.insertAdjacentHTML(
        'beforebegin',
        '<input name="filterValue" type="text" placeholder="value">'
      );
      input.previousElementSibling.focus();
    },
    remFiltVal = function() {
      var input = document.querySelector('[name="filterValue"]');
      if(input.nextElementSibling.name === 'filterValue')
        input.parentNode.removeChild(input);
    };
  document.getElementById('addFilterValue').addEventListener('click', addFiltVal);
  document.getElementById('removeFilterValue').addEventListener('click', remFiltVal);
  document.getElementById('addFilter').addEventListener('click', addFilt);
})();

/* Fill test table with 5000 rows of random data. */
(function() {
  var
    tbl = document.getElementById('hugeTable'),
    num = 5000,
    dat = [
      'a','b','c','d','e','f','g','h','i','j','k','l','m',
      'n','o','p','q','r','s','t','u','v','w','x','y','z'
    ],
    len = dat.length,
    flr = Math.floor,
    rnd = Math.random,
    bod = tbl.tBodies[0],
    sib = bod.nextSibling,
    r = 0;
  tbl.removeChild(bod);
  for(; r < num; r++) {
    bod.insertAdjacentHTML(
      'beforeend',
      '<tr><td>' + r + '</td><td>' + dat[flr(rnd() * len)] + '</td></tr>');
  }
  tbl.insertBefore(bod, sib);
})();

[class*="filt_"] {display: none;} /* THIS RULE IS REQUIRED FOR THE FILTERS TO WORK!!! */

/* THE REMAINING CSS IS JUST FOR THE DEMO INTERFACE AND IS NOT PART OF THE SOLUTION. */
h3 {margin: 0 0 .25em 0;}
[name="filterValue"] {width: 2.5em;}
[class*="filt_"] {display: none;}
#addFilter {margin-top: .5em;}
#filters {margin-left: .5em;}
#filters > div {margin-bottom: .5em;}
#filters > div > input, select {margin-right: .5em;}
#filters, #hugeTable {
  float: left;
  border: 1px solid black;
  padding: 0 .5em 0 .5em;
  white-space: nowrap;
}
#hugeTable {border-spacing: 0;}
#hugeTable > thead > tr > th {
  padding-top: 0;
  text-align: left;
}
#hugeTable > colgroup > col:first-child {min-width: 4em;}

<h3>Add Filter</h3>
Column:
<select id="filterColumn">
  <option value="1">attr</option>
  <option value="0">id</option>
</select>
Action:
<select id="filterType">
  <option value="0">filter out</option>
  <option value="1">filter out everything but</option>
</select>
Value(s):
<input id="addFilterValue" type="button" value="+"
><input id="removeFilterValue" type="button" value="-"
><input name="filterValue" type="text" placeholder="value">
<br>
<input id="addFilter"  type="button" value="Apply">
<hr>
<table id="hugeTable">
  <col><col>
  <thead>
    <tr><th colspan="2"><h3>Huge Table</h3></th></tr>
    <tr><th>id</th><th>attr</th></tr>
  </thead>
  <tbody>
  </tbody>
</table>
<div id="filters">
  <h3>Filters</h3>
</div>

---

*Performance will vary depending on how much CSS is being applied to the table rows and cells, and whether that CSS was written with performance in mind. Whatever filtering strategy you use, there's not much you can do to make a heavily- or inefficiently-styled table perform well, other than load less of it (as others have suggested).

Answer 6

Your best option is to not render all those things and store object versions of them and only show a max of 50 rows at a time via pagination. Storing that many objects in memory, in JS is no problem. Storing all of those in DOM on the other hand will bring browsers to their knees. 5000 is at around the upper bound of what a browser can do on a good machine while maintaining decent performance. If you start modifying some of those rows and tweaking things ('hiding', 'showing') things definitely will get even slower.

The steps would look something like:

1. Organize the data into an array of objects, your hash map is great for supplementary and quick access purposes.
2. Write some sorting and filtering functions that will give you the subsets of data you need.
3. Write a paginator so you can grab sets of data and then get the next set based on some modified params
4. Replace your "draw/render" or "update" method with something that displays the current set of 50 that meets the criteria entered.

The following code should be considered pseudo code that probably works:

// Represents each row in our table
function MyModelKlass(attributes) {
    this.attributes = attributes;
}

// Represents our table
function CollectionKlass() {
    this.children = [];
    this.visibleChildren = [];
    this.limit = 50;
}

CollectionKlass.prototype = {
    // accepts a callback to determine if things are in or out
    filter: function(callback) {
        // filter doesn't work in every browser
        // you can loop manually or user underscorejs
        var filteredObjects = this.children.filter(callback);

        this.visibleChildren = filteredObjects;
        this.filteredChildren = filteredObjects;
        this.showPage(0);
    },
    showPage: function(pageNumber) {
        // TODO: account for index out of bounds
        this.visibleChildren = this.filteredChildren.slice(
           pageNumber * this.limit,
           (pageNumber + 1) * this.limit
        );
    },
    // Another example mechanism, comparator is a function
    // sort is standard array sorting in JS
    sort: function(comparator) {
        this.children.sort(comparator);
    }
}

function render(el, collection, templateContent) {
    // this part is hard due to XSS
    // you need to sanitize all data being written or
    // use a templating language. I'll opt for 
    // handlebars style templating for this example.
    //
    // If you opt for no template then you need to do a few things.
    // Write then read all your text to a detached DOM element to sanitize
    // Create a detached table element and append new elements to it
    // with the sanitized data. Once you're done assembling attach the
    // element into the DOM. By attach I mean 'appendChild'.
    // That turns out to be mostly safe but pretty slow. 
    //
    // I'll leave the decisions up to you.
    var template = Handlebars.compile(templateContent);
    el.innerHTML(template(collection));
}

// Lets init now, create a collection and some rows
var myCollection = new CollectionKlass();

myCollection.children.push(new MyModelKlass({ 'a': 1 }));
myCollection.children.push(new MyModelKlass({ 'a': 2 }));

// filter on something...
myCollection.filter(function(child) {
    if (child.attributes.a === 1) {
        return false;
    }

    return true;
});

// this will throw an out of bounds error right now
// myCollection.showPage(2); 

// render myCollection in some element for some template
render(
    document.getElementById('some-container-for-the-table'), 
    myCollection,
    document.getElementById('my-template').innerHTML()
);

// In the HTML:

<script type="text/x-handlebars-template" id="my-template">
    <ul>
        {{#each visibleChildren}}
            <li>{{a}}</li>
        {{/each}}
    </ul>
</script>