Running updatedb (sys-apps/slocate 2.7-r2) from CLI or as nightly cron job causes massive memory leak. This seems to occur (only?) with ext3, as reiserfs seems unaffected. Tested on 2.4.22-gentoo-r2 and 2.6.0. Reproducible: Always Steps to Reproduce: 1.free -mt 2.updatedb 3.free -mt Actual Results: memory usage (not cached or buffered) went from 30MB to 500MB. Expected Results: memory usage unchanged will attach when I get access to affected computer
Rsync operations appears to have an impact as well. I suspect the real culprit might be the new dir_index filesystem feature that was recently introduced with ext3fs.
I can confirm this as well. I have tried these kernels: gentoo-sources all 2.4.20 versions up to r8 2.4.22-r1 vanilla-sources 2.4.23 ac-sources 2.4.22-r4 aa-sources 2.4.23_pre6 All has the same efect, when updatedb is executed.
This problem is not ext3 specific. I'm experiencing it too and I'm using reiserfs.
I've been experiencing this as well. My partitions are all reiserfs, my kernel is vanilla 2.4.24_pre3 (not in portage) with no patches. It also seems to be caused by any large amounts of file transfers, since compiling programs on my system now takes considerably more memory than before. MAKE_OPTS=-j4 will now cause the system to use several hundred megs (sometimes up to 600M or more) while compiling a relatively small program such as fluxbox.
It's been suggested that 2.6.1 or 2.4.23 or later will be a lot better.
I don't know why this was assigned to me... reassigning to x86-kernel
Try this patch and let us know if this works! diff -Nru a/fs/ext3/dir.c b/fs/ext3/dir.c --- a/fs/ext3/dir.c Wed Apr 23 21:37:34 2003 +++ b/fs/ext3/dir.c Wed Apr 23 21:37:34 2003 @@ -501,7 +501,7 @@ static int ext3_release_dir (struct inode * inode, struct file * filp) { - if (is_dx(inode) && filp->private_data) + if (filp->private_data) ext3_htree_free_dir_info(filp->private_data); return 0;
I want to point out that this bug does affect reiserfs, I see it using 2.4.22-gentoo-r2 kernel. One intersting thing is that after several runs of updatedb the leak seem to stabilize and kernel is able to reclaim some memory (about 100MB in 24 hrs on my 512MB system) so that even after two weeks my box is usable and did not crash.
Okay, can someone show me a vmstat 5 while running rsync or something that chews through memory fast? Also a /proc/slabinfo before and after.
Created attachment 23361 [details] vmstat 5 while running rsync... 1 of 2 For this I did an 'emerge sync' since I needed to do that anyway.
Created attachment 23362 [details] vmstat 5 while running rsync or something that ... 2/2 For this I did an 'updatedb' since that seems to be the program everyone runs everyday which makes this problem so annoying... :) The database is quite big on this system as there are mirrors of two other system here.
Created attachment 23431 [details] vmstat 5 output Here's what I did: reboot cat /proc/slabinfo > slabinfo.before start vmstat 5 running emerge --sync cat /proc/slabinfo > slabinfo.after MARK vmstat output updatedb cat /proc/slabinfo > slabinfo.after_updatedb I've attached the vmstat output, I'll attach the other three files of the slabinfo as well.
Created attachment 23432 [details] slabinfo.before
Created attachment 23433 [details] slabinfo.after
Created attachment 23434 [details] slabinfo.after_updatedb
Created attachment 23438 [details] Post-patch /proc/slabinfo and vmstat 5 For comparison with attachment 23362 [details] (again using an updatedb command, this one starting from just after a system boot). My impression post patch is that though after the command has finished, memory is not immediatly released, it /is/ made available upon demand. Is that what is supposed to happen - I have not taken much notice of the way memory management works in Linux, since to this point it just worked for me :) RG
Patch I gave really does fixes a memory leak problem with htree in ext3 filesystem. As for reiserfs, this perhaps should be reported to the reiserfs people. It might be that we're seeing two different problems causing the same thing.
A FYI. This forum thread seems to be related to the memory leak. http://forums.gentoo.org/viewtopic.php?t=123309
Is this bug also for reiserfs problem (same symptoms, but on reiser) and is NOT fixed by the proposed patch or should I submit a new bug report?
If your system is able to reclaim the memory it's not a bug. That's normal fs caching behavior. The kernel will aggresively cache stuff as long as there is free memory. It's common for me to only see 10MB free on my 2GB workstation. In fact my memory is almost purely taken by the cache (and X).
Maybe someone can describe this in dumb-man terms for me. I use gkrellm2 which has 3 marks on the memory, the main one being used, second buffers, third cache. After running updatedb the used memory goes up to about halfway (512MB) and doesn't seem to lower until I reboot the machine - I've left it about a day or so. This all happens on a reisersfs system using kernel 2.6.2-gentoo.
OK, here is what it looks on my system with Reiser: Freshly rebooted system # free -mt total used free shared buffers cached Mem: 503 49 453 0 7 21 -/+ buffers/cache: 20 483 Swap: 980 0 980 Total: 1484 49 1434 #updatedb # free -mt total used free shared buffers cached Mem: 503 494 9 0 235 25 -/+ buffers/cache: 233 270 Swap: 980 0 980 Total: 1484 494 990 I know about caching behavior and according to my understanding available memory is basically free+cached+buffers. The output I posted above shows me that update db ate more than 200MB of memory. As for as reclaiming letting this stand overnight improves the situation very little (maybe because something were moved to swap) and more than 200MB is missing. I think this is a bug and not caching behavior.
It's not a bug, that's what every system I've ever seen does. I'm closing this bug as I've applied the ext3 fix to -r5.
Surely the point is that it doesn't do it with the fix applied for ext3, so it shouldn't do it for reiserfs either...
ran "emerge sync" (uses rsync).. and the ram used up was _not_ able to be reclaimed... when i went to play a game i just simply hit swap space.. and after exitting the game.. the same amount of ram used after rsyncing was still un-reclaimable... same for updatedb... can't reclaim it..