why is my c program suddenly using 30g of virtual

Question

In top, I noticed that my c program (using CUDA 3.2) has a virtual size of 28g or more (looking at VIRT), on every run right from the beginning. This doesn't make ANY sense to me. The resident memory makes sense and is only around 2g on my largest data set. I know at some point in the past the virtual size was not so large, but I'm not sure when the change occurred.

Why would my process use 28g of virtual memory (or why would top's VIRT be so large)? I understand that VIRT includes the executable binary (only 437K), shared libraries, and "data area". What is the "data area"? How can I find out how much memory the shared libraries require? What about other elements of my process's total memory?

contents of /proc/< pid >/smaps (1022 lines) here: http://pastebin.com/fTJJneXr

One of the entries from smaps show that one of them accounts for MOST of it, but has no label... how can I find out what this "blank" entry is that has 28gb?

200000000-900000000 ---p 00000000 00:00 0 
Size:           29360128 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

--

ubuntu 11.04 64-bit
16 GB RAM

Answer 1

These two regions would be the culprit:

200000000-900000000 ---p 00000000 00:00 0
Size:           29360128 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
7f2e9deec000-7f2f131ec000 rw-s 33cc0c000 00:05 12626                     /dev/nvidia0
Size:            1920000 kB
Rss:             1920000 kB
Pss:             1920000 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1920000 kB
Referenced:      1920000 kB
Anonymous:             0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

The first segment is a 30GB anonymous private segment, with no access to it allowed, mapped from 0x200000000-0x900000000. A bit mysterious, indeed - probably something to do with the nvidia driver's internal workings (maybe it wants to prevent allocations with those specific addresses?). It's not actually occupying any memory though - Rss is zero, and the access flags (---p) are set to deny all access, so (at the moment) actually allocating any memory to it won't happen. It's just a reserved section in your address space.

The other bit is the /dev/nvidia0 mapping, of two gigabytes. This is likely a direct mapping of part of the video card's RAM. It's not occupying memory as such - it's just reserving part of your address space to use to communicate with hardware.

So it's not really something to worry about. If you want to know how much memory you're really using, add up the Rss figures for all other memory segments (use the Private_* entries instead if you want to skip shared libraries and such).

Answer 2

UVA requires CUDA to allocate enough virtual memory to map all of both GPU and system memory. Please see post #5 in the following thread on the NVIDIA forums: