memstat: Identify what is using up virtual memory

October 19th, 2008 edited by Vicho

Article submitted by Todd Troxell. Guess what? We still need you to submit good articles about software you like!

This tool lets you discover what libraries and programs are using up memory. It is very simple to use. Here is an example of its output (the -w makes memstat not truncate lines at 80 columns):

gaius% memstat -w
    256k: PID  5465 (/lib/
    368k: PID 13019 (/var/db/nscd/passwd)
   3352k: PID 13914 (/usr/lib/gconv/gconv-modules.cache)
      8k: /usr/bin/memstat 5465
     12k: /lib/ 13019
    256k: /lib/ 13914
     88k: /lib/ 5465 13019
    256k: /lib/ 13019
   1212k: /lib/tls/ 13914
     32k: /lib/tls/ 13914
     24k: /lib/tls/ 13914
     12k: /lib/tls/ 13914
    144k: /lib/tls/ 13914
     76k: /lib/tls/ 13914
     40k: /lib/tls/ 13914
     36k: /lib/tls/ 13914
     60k: /lib/tls/ 13914
     28k: /lib/tls/ 13914
     88k: /lib/ 13914
   1212k: /lib/tls/ 5465 13019
     12k: /lib/tls/ 13019
    144k: /lib/tls/ 13019
     76k: /lib/tls/ 13019
    480k: /bin/zsh-beta 13019
    212k: /var/db/nscd/passwd 13019
    788k: /usr/bin/irssi 13914
    148k: /usr/lib/ 13019
    176k: /usr/lib/perl5/auto/Irssi/ 13914
     80k: /usr/lib/perl5/auto/Irssi/Irc/ 13914
     80k: /usr/lib/perl5/auto/Irssi/UI/ 13914
     12k: /usr/lib/gconv/ 13914
     24k: /usr/lib/gconv/gconv-modules.cache 13914
     76k: /usr/lib/ 13914
    584k: /usr/lib/ 13914
   1128k: /usr/lib/ 13914
     12k: /usr/lib/ 13914
   1240k: /usr/lib/i686/cmov/ 13914
    248k: /usr/lib/i686/cmov/ 13914
      8k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/ 13019
     24k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/ 13019
     56k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/ 13019
    116k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/ 13019
    196k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/ 13019

This output lists many libraries and processes loaded into memory and their sizes. First of all, processes and the size of their private memory are listed. This does not include their shared memory. Afterwards, shared objects are listed, and finally the total is listed.

In case you are wondering, shared object are libraries like /lib/tls/ that are shared across all processes that need them to save memory and make things run faster. Instead of loading this library into memory for every process, Linux loads one copy and uses this for any process that wants to use the library. Therefore, you may notice that these values sometimes do not add up to the amount of memory you see used on your system. If you look at ps(1) you will see two columns related to memory: RSS and VSZ. For each process, RSS is the amount of memory used by the process, and VSZ is the amount used counting shared objects. To add up memory correctly, you must count each shared object only once for the system.

It is possible to output per user statistics by running memstat as an unprivileged user. When run as root, memstat will list everything on the system.

Memstat works by scanning files in /proc and then searching for binaries in the paths listed at /etc/memstat.conf. The default file should be sufficient for most cases. If you have libraries or binaries in a non-standard place, you may need to modify this file to get accurate results.

Memstat was authored by Joshua Yelon. It is available in all current Debian and Ubuntu releases.

Posted in Debian, Ubuntu |

9 Responses

  1. horst Says:

    I dont’t see RSS and VSZ neither in the output or the manpage of top.
    Did you mean RES and SHR?

  2. Vicho Says:

    That’s right, RSS and VSZ are from the manpage of ps(1). I’ve corrected the article.

    In top(1) they are RES and VIRT respectively.

  3. Ingo Wagener Says:

    Thanks for posting this. It would be useful to get the output sorted in some way or another (like heaviest users first). Unfortunately my command of the CLI is not as advanced as that :( Can you/somebody help out?

  4. oliver Says:

    Btw. do you know of any apps that make use of the new pagemap stuff ( that has finally landed in 2.6.26? It sounds pretty neat, but the example apps seem outdated, and I haven’t been able to make sense of the kernel interface either, and Google doesn’t give much useful content for the whole topic… Anyone knows more?

  5. Benoit Says:

    To sort output (ascending):
    memstat -w | sort -n

  6. Shai Says:

    Thanks for posting it, I didn’t know about it, and it seems very useful.

    However, it could be a little more useful if it printed process executables correctly. Even in your example, process 13914, which (judging by its list of libraries) looks like Irssi, is listed as “/usr/lib/gconv/gconv-modules.cache”. When I look at my KDE session, a process listed as “libssl” turned out to be kded, and iceweasel was listed as “/usr/share/icons/hicolor/icon-theme.cache”.

    Does anybody know why this happens and how to fix it?

  7. thefool Says:

    I don’t think memstat really works, at least on my system (sid with kernel 2.6.24). It shows the names of truetype font files on my system, and names like /SYSV00000000, instead of program names that correspond to the PID it lists.

    Does anyone know if it also tries to use LD_LIBRARY_PATH or PATH environment variables? I can’t tell because all the program names are wrong.

    And a sort option would be useful, or options to only show process private memory or shared memory so you can sort them seperately with sort.

  8. Manski Says:

    I’ve installed it today but find it rather useless. I’ve 1 GB Ram in my Debian box and “memstat” shows me that 2.5 GB are used. I guess that’s virtual memory so this is not a problem but this information is IMHO not very helpful. I prefer “htop” which displays the amount of memory that is really used (400 MB in my case, so this is the information I wanted to get).

  9. Shai Says:

    Well, my (and thefool’s) complaint is now Debian bug #503026.

    Thanks again for bringing this tool to my attention,