Lua memory leak in C process

Question

I have a C program running with Lua.

Although I try to use lua_gc() to get and control memory usage of Lua, C process memory usage is still high. The C process uses over 150MB memory even though Lua say that it uses only 4MB memory.

I also try to use my l_alloc() function to track Lua memory allocation, but the result is the same with the memory usage Lua told by calling lua_gc(LUA_GCCOUNT) and lua_gc(LUA_GCCOUNTB).

After calling lua_close() to close the Lua environment, the process memory is down and looks fine. Therefore, I think the 'lost memory' is still controlled by Lua not C program.

Here is the sample C code. It creates a Lua environment, calls Lua function to purge data and then check memory usage.

int main()
{
    int rc;
    uint64_t gc_mem_usage;

    lua_State* Lua = luaL_newstate();
    luaL_openlibs(Lua);

    lua_gc(Lua, LUA_GCSTOP, 0);
    luaL_dofile(Lua, "test.lua");

    gc_mem_usage = ((uint64_t)lua_gc(Lua, LUA_GCCOUNT, 0) << 10) + lua_gc(Lua, LUA_GCCOUNTB, 0);
    printf("Lua mem usage: [%" PRIu64 "] Bytes\n", gc_mem_usage);

    lua_getglobal(Lua, "command_handler");
    lua_pushstring(Lua, "CC");
    rc = lua_pcall(Lua, 1, 0, 0);
    if (rc != 0 ) {
        printf("function error\n");
        return;
    }

    lua_settop(Lua, 0);

    // do full gc
    lua_gc(Lua, LUA_GCCOLLECT, 0);
    lua_gc(Lua, LUA_GCCOLLECT, 0); // I don't know why it has different result by calling full gc twice
    sleep(1);

    printf("-------------After GC ----------------------\n");
    gc_mem_usage = ((uint64_t)lua_gc(Lua, LUA_GCCOUNT, 0) << 10) + lua_gc(Lua, LUA_GCCOUNTB, 0);
    printf("Lua mem usage: [%" PRIu64 "] Bytes\n", gc_mem_usage);

    // infinite-loop
    while(1);
}

Lua sample code:

local abc = {}

function command_handler(cmd)
    if (cmd == "CC") then
        abc = {}
    end
end

for i =0, 2000000 do
    abc[i] = "ABC" .. i .. "DEF"
end

output:

Lua mem usage: [204913817] Bytes
-------------After GC ----------------------
Lua mem usage: [4219342] Bytes

The output told me that Lua memory usage is down after GC, but the memory usage of this C process is still very high (193.7MB) by checking atop continuously.

 PID MINFLT MAJFLT      VSTEXT  VSIZE  RSIZE  VGROW  RGROW  MEM CMD     1/1
4622      1      0          3K 193.7M 183.9M     0K     4K  18% a.out

Is there any solution to reduce the C process memory usage?

My environment is Lua 5.1.4 running in Ubuntu/CentOS.

Answer 1

Lua faithfully frees unreachable objects by calling the supplied deallocation function (by default realloc(block, 0)). It looks like libc allocator is struggling to return unused memory, possibly due to high fragmentation. Looking at the strace output (I've got roughly the same numbers with Lua 5.1.4 on 64-bit Debian 6), C runtime chooses to allocate using brk with small increments, but no deallocation (calling brk with lower value) follows. However, if you insert malloc_trim(M_TOP_PAD) before entering infinite loop, you'll see in the top output that resident size drops drastically to ~5M and strace reveals that data segment was indeed trimmed with brk. Using custom allocator (e.g. pool-based) or tuning malloc parameters will probably help in this situation.