memstat: Identify what is using up virtual memory
October 19th, 2008 edited by VichoArticle submitted by Todd Troxell. Guess what? We still need you to submit good articles about software you like!
This tool lets you discover what libraries and programs are using up memory. It is very simple to use. Here is an example of its output (the -w makes memstat not truncate lines at 80 columns):
gaius% memstat -w 256k: PID 5465 (/lib/ld-2.3.6.so) 368k: PID 13019 (/var/db/nscd/passwd) 3352k: PID 13914 (/usr/lib/gconv/gconv-modules.cache) 8k: /usr/bin/memstat 5465 12k: /lib/libcap.so.1.10 13019 256k: /lib/libncurses.so.5.5 13914 88k: /lib/ld-2.3.6.so 5465 13019 256k: /lib/libncurses.so.5.5 13019 1212k: /lib/tls/libc-2.3.6.so 13914 32k: /lib/tls/libnss_compat-2.3.6.so 13914 24k: /lib/tls/libcrypt-2.3.6.so 13914 12k: /lib/tls/libdl-2.3.6.so 13914 144k: /lib/tls/libm-2.3.6.so 13914 76k: /lib/tls/libnsl-2.3.6.so 13914 40k: /lib/tls/libnss_files-2.3.6.so 13914 36k: /lib/tls/libnss_nis-2.3.6.so 13914 60k: /lib/tls/libpthread-2.3.6.so 13914 28k: /lib/tls/librt-2.3.6.so 13914 88k: /lib/ld-2.3.6.so 13914 1212k: /lib/tls/libc-2.3.6.so 5465 13019 12k: /lib/tls/libdl-2.3.6.so 13019 144k: /lib/tls/libm-2.3.6.so 13019 76k: /lib/tls/libnsl-2.3.6.so 13019 480k: /bin/zsh-beta 13019 212k: /var/db/nscd/passwd 13019 788k: /usr/bin/irssi 13914 148k: /usr/lib/libpcre.so.3.12.0 13019 176k: /usr/lib/perl5/auto/Irssi/Irssi.so 13914 80k: /usr/lib/perl5/auto/Irssi/Irc/Irc.so 13914 80k: /usr/lib/perl5/auto/Irssi/UI/UI.so 13914 12k: /usr/lib/gconv/CP1252.so 13914 24k: /usr/lib/gconv/gconv-modules.cache 13914 76k: /usr/lib/libz.so.1.2.3 13914 584k: /usr/lib/libglib-2.0.so.0.1200.4 13914 1128k: /usr/lib/libperl.so.5.8.8 13914 12k: /usr/lib/libgmodule-2.0.so.0.1200.4 13914 1240k: /usr/lib/i686/cmov/libcrypto.so.0.9.8 13914 248k: /usr/lib/i686/cmov/libssl.so.0.9.8 13914 8k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/terminfo.so 13019 24k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/zutil.so 13019 56k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/compctl.so 13019 116k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/complete.so 13019 196k: /usr/lib/zsh-beta/4.3.2-dev-1/zsh/zle.so 13019 -------- 13480k
This output lists many libraries and processes loaded into memory and their sizes. First of all, processes and the size of their private memory are listed. This does not include their shared memory. Afterwards, shared objects are listed, and finally the total is listed.
In case you are wondering, shared object are libraries like /lib/tls/libc-2.3.6.so
that are shared across all processes that need them to save memory and make things run faster. Instead of loading this library into memory for every process, Linux loads one copy and uses this for any process that wants to use the library. Therefore, you may notice that these values sometimes do not add up to the amount of memory you see used on your system. If you look at ps(1) you will see two columns related to memory: RSS and VSZ. For each process, RSS is the amount of memory used by the process, and VSZ is the amount used counting shared objects. To add up memory correctly, you must count each shared object only once for the system.
It is possible to output per user statistics by running memstat as an unprivileged user. When run as root, memstat will list everything on the system.
Memstat works by scanning files in /proc
and then searching for binaries in the paths listed at /etc/memstat.conf
. The default file should be sufficient for most cases. If you have libraries or binaries in a non-standard place, you may need to modify this file to get accurate results.
Memstat was authored by Joshua Yelon. It is available in all current Debian and Ubuntu releases.
October 19th, 2008 at 12:46 pm
I dont’t see RSS and VSZ neither in the output or the manpage of top.
Did you mean RES and SHR?
October 19th, 2008 at 1:30 pm
That’s right, RSS and VSZ are from the manpage of ps(1). I’ve corrected the article.
In top(1) they are RES and VIRT respectively.
October 19th, 2008 at 4:07 pm
Thanks for posting this. It would be useful to get the output sorted in some way or another (like heaviest users first). Unfortunately my command of the CLI is not as advanced as that :( Can you/somebody help out?
October 19th, 2008 at 7:32 pm
Btw. do you know of any apps that make use of the new pagemap stuff (http://lwn.net/Articles/230975/) that has finally landed in 2.6.26? It sounds pretty neat, but the example apps seem outdated, and I haven’t been able to make sense of the kernel interface either, and Google doesn’t give much useful content for the whole topic… Anyone knows more?
October 20th, 2008 at 8:29 am
To sort output (ascending):
memstat -w | sort -n
October 20th, 2008 at 8:57 am
Thanks for posting it, I didn’t know about it, and it seems very useful.
However, it could be a little more useful if it printed process executables correctly. Even in your example, process 13914, which (judging by its list of libraries) looks like Irssi, is listed as “/usr/lib/gconv/gconv-modules.cache”. When I look at my KDE session, a process listed as “libssl” turned out to be kded, and iceweasel was listed as “/usr/share/icons/hicolor/icon-theme.cache”.
Does anybody know why this happens and how to fix it?
October 20th, 2008 at 3:11 pm
I don’t think memstat really works, at least on my system (sid with kernel 2.6.24). It shows the names of truetype font files on my system, and names like /SYSV00000000, instead of program names that correspond to the PID it lists.
Does anyone know if it also tries to use LD_LIBRARY_PATH or PATH environment variables? I can’t tell because all the program names are wrong.
And a sort option would be useful, or options to only show process private memory or shared memory so you can sort them seperately with sort.
October 21st, 2008 at 8:14 am
I’ve installed it today but find it rather useless. I’ve 1 GB Ram in my Debian box and “memstat” shows me that 2.5 GB are used. I guess that’s virtual memory so this is not a problem but this information is IMHO not very helpful. I prefer “htop” which displays the amount of memory that is really used (400 MB in my case, so this is the information I wanted to get).
October 21st, 2008 at 10:10 pm
Well, my (and thefool’s) complaint is now Debian bug #503026.
Thanks again for bringing this tool to my attention,
Shai.