I'm struggling to understand why this code generates a memory leak, or at least what appears to be a memory leak.
I'm attempting to write a portfolio of quiz objects to an indexedDB database. At this stage, I'm just trying to test the speed of writing different numbers of quizzes of different sizes.
Included is only the portion of the code that writes to the database. The port.quiz[] object is defined globally. The code is set up to display messages as each quiz is successfully written and a final message when the full transaction is completed.
The part that I've been unable to understand is why the statement "delete port.quiz[ code[c] ]" in the request.onsuccess block of the store_data function is required to avoid a memory leak. If it is commented out, the memory usage grows as each quiz is written and is not freed at the end of the transaction.
If the delete statement is included, then the memory usage rises for four or five quiz stores and then drops down; but, the maximum amount it rises to increases after every drop. At the close of the transaction, the memory usage returns to what it was before the write_quizzes function was invoked.
The testing is somewhat ridiculous in the sense that I'm writing 100 quizzes each containing 500 questions, and the components of each question are filled with more text than anyone would ever use. But I'm trying to test for a more than reasonable expected maximum limit. Apart from the memory leak, it works fairly well.
If the memory usage is viewed in windows task manager, it starts out at about 2MB and rises to 2GB by the 100th quiz if the delete port.quiz statement is not included. If the statement is included, memory usage rises to 5-6MB and then drops down to 4MB, and then rises to 7-8MB and then drops to 5MB, and so on until it maxes out around 1.1-1.2GB and then drops back to 2MB at the close of the transaction.
I thought maybe the issue was as described in these two links but I haven't been able to figure it out.
https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/
https://blog.meteor.com/an-interesting-kind-of-javascript-memory-leak-8b47d2e7f156
I tried moving the store_data function outside of the write_quizzes function, in case the issue is related to a shared scope; and it appears to reduce the maximum memory usage reached, but the same behavior overall exists.
I also came across an issue noted for Edge browser concerning indexedDB generating a memory leak; but I've been testing in Firefox only, and no solution is provided in the link anyway.
https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/7122372/
Even with the delete statement, the memory usage is too high. The object containing the 100 quizzes is about 2MB. So, why memory usage increases so much with the writing of each individual quiz, has me confused. Perhaps, I am just overlooking something obvious and fundamental.
Thank you for any guidance or suggestions you may be able to provide.
function write_quizzes() {
// Separating the transaction from the objectStore allows for
// separate onsuccess and onerror events.
var qz_tran = db.transaction( ["quiz data"], "readwrite" ),
store = qz_tran.objectStore("quiz data"),
code = new Array(),
i = 0, q, l,
request;
// Do something when all the data is added to the database.
qz_tran.oncomplete = function(event) {
$("#msg").text('Finished writing the ' + i + ' quizzes to database!');
}; // close oncomplete
qz_tran.onerror = function(event) {
// Don't forget to handle errors!
alert("Error writing data " + event.target.errorCode + " .");
}; // close onerror
// Writing quiz object references to an array in order to write recursively.
for ( q in port.quiz ) {
code[++i] = q;
}; // next q
l = code.length;
store_data(1);
function store_data(c) {
request = store.add( port.quiz[ code[c] ], code[c] );
request.onsuccess = function( event ) {
$('#msg').text('Writing quiz ' + code[c]);
delete port.quiz[ code[c] ];
// Why does this need to be deleted and how is it leaking memory?
if ( c + 1 < l ) { store_data(++c); };
}; // close onsuccess
request.onerror = function( event ) {
alert('write error : ' + event.target.errorCode );
}; // close onerror
} // close store_data
EDIT with EXAMPLE:
Below is html code with script that demonstrates the problem encountered. If click the 'Build Portfolio' button, it will build a sample portfolio of junk quiz data and open a database with one object store at the quiz level. After the message reads that the database has been opened, each click of the 'Write Quiz' button will write one of the fifty quiz objects from the portfolio to the object store. If observe memory usage, even in windows task manager, it will increase with each click after message reads 'Done writing quiz n to the database', and it will not be released upon reload of the page.
I experimented for a few hours and in one trial I noticed that store.put for an existing quiz key would increase the memory usage for a second or so and then return to the before-click level; but for a new key, put would take up memory and keep it just like store.add does here.
I'm pretty much a novice at indexedDB but can only conclude that the issue is with store.add or because the portfolio object is global. I admit I don't understand the information at MDN web docs about store.add creating a structured clone and serialization of the value, but wonder if something at that step doesn't free the memory.
https://developer.mozilla.org/en-US/docs/Web/API/IDBObjectStore/add
https://html.spec.whatwg.org/multipage/structured-data.html#structured-clone
These two links have report dates from 2015 but updates as recent as 29 days ago; and the issue they describe, especially that it occurs on larger objects and not smaller ones, seems to be similar to this situation since I was trying to test the writing of multiple large quiz objects. Or maybe my test cases are a bit unrealistic for a single object and the test size is just too large.
Race condition can cause redundant IDBFS store sync operations. https://github.com/kripken/emscripten/issues/3908
Erratic IndexedDB large file store operations cause IndexedDB consume a large amount of memory that is not freed. https://bugzilla.mozilla.org/show_bug.cgi?id=1223782
Also related and more recent, and yet to be answered..
Indexeddb seems to not free memory
Interestingly, if the k loop in the gen_port function is commented out completely and what was the text property of the answer_choice objects (the variable t = 5000 question marks) is increased by a factor of 26 to be 130,000 question marks and written as the only data element of each question object, such that each question object still contains about the same amount of data as it would when a collection of 26 separate answer-choice objects generated in the k loop, the memory leak does not occur. (Please see gen_port_2 in the code example below.) It takes place only when there is an answer-choice object level: portfolio.quiz[].question[].answer_choice[].text. The memory usage jumps up while writing each quiz to the database, but when complete it returns to the pre-write level.
This leads me to suspect that the issue is very much related to that of the bugzilla link above entitled 'Erratic IndexedDB large file store operations cause IndexedDB consume a large amount of memory that is not freed.' Perhaps, fixing the blob issue doesn't fix this issue when the blob is just a large chunk of data without the nested object levels that require serialization in the database. The memory leak is observed in windows task manager in the firefox.exe, as the link first described three years ago.
As I wrote earlier, I'm very much a novice in this area and could be greatly confused. I also am not familiar with bugzilla and whether or not I could add this to their group but wll investigate. Perhaps, my example is a misuse of what indexedDB is intended to provide; but maybe not.
For my specific purposes, I think I can just stringify the quiz-level object before writing to the database. I just want to store and retrieve the quizzes as a block and don't require the serialization step to search or query the database by individual data elments within the quiz objects. So, I think I have my answer in terms of being able to accomplish what I need for my project without leaking memory, but this could still be an issue for others.
UPDATE
Unfortunately, using JSON.stringify doesn't work either because, as I should have expected, the serialization step seems to allocate RAM and not release it when complete. The memory usage increases as the quiz objects are converted to strings instead of when written to the database. So, all this attempt accomplished was altering at which point the memory issue occurs. If the quiz objects are converted to string without using JSON.stringify, the memory usage stays flat during conversion but increases when the strings are written to the database. In both cases, the memory is almost always released, and it takes place only when the delete port.quiz[c] statement is not commented out; but the release is slow to take place and the memory usage reaches a high level at around 1GB to complete the task before dropping, either while the quiz objects are being written to the database or when they're being converted to strings. And, in those instances in which the memory is not released, a refresh won't release it, only closing the page does. If the delete port.quiz[c] statement is commented out, the memory usage grows until it crashes my machine which has only 4GB of RAM.
The only way I could write a large portfolio of quizzes to the database without having a memory usage issue, was to create a separate object store for each quiz, and then write the questions for each quiz to that store. The smaller objects don't cause the memory usage issue so long as the delete port.quiz[c] statement is active. So, I could write larger amounts of data without reaching half the memory usage of the former cases.
There doesn't appear to be much interest in this issue; but if you understand why this happens and/or how to correct it, please explain. Thank you.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<script src="jquery-3.3.1.js"></script>
<style>
html { background-color: rgb(73,110,147); }
</style>
</head>
<header>
</header>
<body>
<div style='width: 750px; min-height: 100px; margin: 0 auto; border: 1px solid black; background-color: white;'>
<p id="msg" style="margin: 25px auto; text-align: center;">IndexedDB Testing</p>
<button id="build_port">Build Portfolio</button>
<button id="write_quiz">Write Quiz</button>
</div>
</body>
<script>
"use strict";
$(document).ready( function() {
var db, c = 0, port;
$("#build_port").click( load_port );
$("#write_quiz").click( function() { add_quiz(++c); } );
function load_port() {
$("#msg").text('Generating quiz portfolio');
setTimeout( function() { port = gen_port(50); open_db(); }, 10 );
} // close load_port
function add_quiz(c) {
var qz_tran = db.transaction( ["quiz data"], "readwrite" ),
store = qz_tran.objectStore("quiz data"),
request;
qz_tran.oncomplete = function(event) {
$("#msg").text('Done writing quiz ' + c + ' to the database!');
};
qz_tran.onerror = function(event) {
alert("Error Writing data " + event.target.errorCode + " .");
};
request = store.add( port.quiz[ c ], c );
request.onsuccess = function( event ) {
$('#msg').text( 'Writing quiz ' + c );
//delete port.quiz[ c ];
};
request.onerror = function( event ) { alert('write error : ' + event.target.errorCode ); };
} // close add_quiz
function open_db() {
if ( !window.indexedDB ) {
alert("Your browser doesn't support a stable version of IndexedDB. Such and such feature will not be available.");
}; // end if
// Temporarily deleting each time for testing purposes.
// Will later need to test for an existing databases just like did for localStorage.
indexedDB.deleteDatabase("quizDB");
// Attempt to open a database.
var status = 'Existing',
req = window.indexedDB.open( "quizDB", 1 );
req.onerror = function(event) {
alert('Attempt to open the portfolio database failed. Error code : ' + event.target.errorCode + '.' );
}; // close onerror
req.onsuccess = function(event) {
db = this.result;
db.onerror = function(event) {
// Generic error handler for all errors targeted at this database's requests!
alert( 'Database error: ' + event.target.errorCode + '.' );
}; // close onerror
$("#msg").text('opened/created database as : ' + status );
}; // close onsuccess
req.onupgradeneeded = function(event) {
var db = event.target.result;
status = 'New';
db.createObjectStore( "quiz data" );
}; // close onupgradeneeded
} // close open_db
function gen_port(n) {
var i, j, k, port = new Object(),
t = new Array(5000).join('?');
port.quiz = new Object();
for ( i = 1; i <= n; i++ ) {
port.quiz[i] = new Object();
port.quiz[i].name = 'Quiz ' + i;
port.quiz[i].question = new Object();
for ( j = 1; j <= 500; j++ ) {
port.quiz[i].question[j] = new Object();
for ( k = 1; k <= 26; k++ ) {
port.quiz[i].question[j]['answer_choice_' + k] = new Object();
port.quiz[i].question[j]['answer_choice_' + k].text = 'Quiz ' + j + ', Question ' + j + ' AC ' + k + ' : ' + t;
port.quiz[i].question[j]['answer_choice_' + k].checked = true;
port.quiz[i].question[j]['answer_choice_' + k].pos = k;
}; // next k
}; // next j question
}; // next i quiz
$("#msg").text('Done generationg quiz portfolio');
return port;
} // close gen_port
function gen_port_2(n) {
var i, j, k, port = new Object(),
t = new Array(130000).join('?');
port.quiz = new Object();
for ( i = 1; i <= n; i++ ) {
port.quiz[i] = new Object();
port.quiz[i].name = 'Quiz ' + i;
port.quiz[i].question = new Object();
for ( j = 1; j <= 500; j++ ) {
port.quiz[i].question[j] = new Object();
port.quiz[i].question[j] = t;
}; // next j question
}; // next i quiz
$("#msg").text('Done generationg quiz portfolio');
return port;
} // close gen_port
}); // close document ready
</script>
</html>
