Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 36855 - slocate/updatedb causes massive memory leak with ext3 filesystem
Summary: slocate/updatedb causes massive memory leak with ext3 filesystem
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: x86 Linux
: High major (vote)
Assignee: x86-kernel@gentoo.org (DEPRECATED)
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2003-12-30 16:53 UTC by Adam Collins
Modified: 2005-01-22 14:01 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
vmstat 5 while running rsync... 1 of 2 (bug36855.1of2.gz,5.35 KB, application/octet-stream)
2004-01-07 20:45 UTC, r1chard
Details
vmstat 5 while running rsync or something that ... 2/2 (bug36855.2of2.gz,5.46 KB, application/octet-stream)
2004-01-07 20:47 UTC, r1chard
Details
vmstat 5 output (vmstat.out,34.12 KB, text/plain)
2004-01-08 19:07 UTC, Mr. Bones. (RETIRED)
Details
slabinfo.before (slabinfo.before,3.40 KB, text/plain)
2004-01-08 19:07 UTC, Mr. Bones. (RETIRED)
Details
slabinfo.after (slabinfo.after,3.40 KB, text/plain)
2004-01-08 19:08 UTC, Mr. Bones. (RETIRED)
Details
slabinfo.after_updatedb (slabinfo.after_updatedb,3.40 KB, text/plain)
2004-01-08 19:08 UTC, Mr. Bones. (RETIRED)
Details
Post-patch /proc/slabinfo and vmstat 5 (bug36855.post_ext_patch,19.29 KB, text/plain)
2004-01-08 20:47 UTC, r1chard
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Adam Collins 2003-12-30 16:53:20 UTC
Running updatedb (sys-apps/slocate 2.7-r2) from CLI or as nightly cron job causes massive memory leak.  This seems to occur (only?) with ext3, as reiserfs seems unaffected.  Tested on 2.4.22-gentoo-r2 and 2.6.0.

Reproducible: Always
Steps to Reproduce:
1.free -mt
2.updatedb
3.free -mt

Actual Results:  
memory usage (not cached or buffered) went from 30MB to 500MB.

Expected Results:  
memory usage unchanged

will attach when I get access to affected computer
Comment 1 Alex Buell 2003-12-30 17:18:42 UTC
Rsync operations appears to have an impact as well. I suspect the real culprit might be the new dir_index filesystem feature that was recently introduced with ext3fs. 
Comment 2 Sami Näätänen 2003-12-30 18:53:04 UTC
I can confirm this as well.

I have tried these kernels:

gentoo-sources
all 2.4.20 versions up to r8
2.4.22-r1

vanilla-sources
2.4.23

ac-sources
2.4.22-r4

aa-sources
2.4.23_pre6

All has the same efect, when updatedb is executed.
Comment 3 Tom Kiermaier 2004-01-02 06:07:15 UTC
This problem is not ext3 specific. I'm experiencing it too and I'm using reiserfs. 
Comment 4 Jayson 2004-01-03 18:47:52 UTC
I've been experiencing this as well. My partitions are all reiserfs, my kernel is vanilla 2.4.24_pre3 (not in portage) with no patches. It also seems to be caused by any large amounts of file transfers, since compiling programs on my system now takes considerably more memory than before. MAKE_OPTS=-j4 will now cause the system to use several hundred megs (sometimes up to 600M or more) while compiling a relatively small program such as fluxbox. 
Comment 5 Alex Buell 2004-01-03 22:38:56 UTC
It's been suggested that 2.6.1 or 2.4.23 or later will be a lot better. 
Comment 6 Aron Griffis (RETIRED) gentoo-dev 2004-01-06 07:23:31 UTC
I don't know why this was assigned to me... reassigning to x86-kernel
Comment 7 Alex Buell 2004-01-07 01:00:53 UTC
Try this patch and let us know if this works!

diff -Nru a/fs/ext3/dir.c b/fs/ext3/dir.c
--- a/fs/ext3/dir.c     Wed Apr 23 21:37:34 2003
+++ b/fs/ext3/dir.c     Wed Apr 23 21:37:34 2003
@@ -501,7 +501,7 @@
 
 static int ext3_release_dir (struct inode * inode, struct file * filp)
 {
-       if (is_dx(inode) && filp->private_data)
+       if (filp->private_data)
                ext3_htree_free_dir_info(filp->private_data);
 
        return 0;
Comment 8 hodak 2004-01-07 18:28:35 UTC
I want to point out that this bug does affect reiserfs, I see it using   2.4.22-gentoo-r2 kernel. One intersting thing is that after several runs of updatedb the leak seem to stabilize and kernel is able to reclaim some memory (about 100MB in 24 hrs on my 512MB system) so that even after two weeks my box is usable and did not crash.
Comment 9 Brian Jackson (RETIRED) gentoo-dev 2004-01-07 19:18:40 UTC
Okay, can someone show me a vmstat 5 while running rsync or something that chews through memory fast? Also a /proc/slabinfo before and after.
Comment 10 r1chard 2004-01-07 20:45:05 UTC
Created attachment 23361 [details]
vmstat 5 while running rsync... 1 of 2

For this I did an 'emerge sync' since I needed to do that anyway.
Comment 11 r1chard 2004-01-07 20:47:19 UTC
Created attachment 23362 [details]
vmstat 5 while running rsync or something that ... 2/2

For this I did an 'updatedb' since that seems to be the program everyone runs
everyday which makes this problem so annoying... :) The database is quite big
on this system as there are mirrors of two other system here.
Comment 12 Mr. Bones. (RETIRED) gentoo-dev 2004-01-08 19:07:12 UTC
Created attachment 23431 [details]
vmstat 5 output

Here's what I did:

reboot
cat /proc/slabinfo > slabinfo.before
start vmstat 5 running
emerge --sync
cat /proc/slabinfo > slabinfo.after
MARK vmstat output
updatedb
cat /proc/slabinfo > slabinfo.after_updatedb

I've attached the vmstat output, I'll attach the other three files of the
slabinfo as well.
Comment 13 Mr. Bones. (RETIRED) gentoo-dev 2004-01-08 19:07:45 UTC
Created attachment 23432 [details]
slabinfo.before
Comment 14 Mr. Bones. (RETIRED) gentoo-dev 2004-01-08 19:08:08 UTC
Created attachment 23433 [details]
slabinfo.after
Comment 15 Mr. Bones. (RETIRED) gentoo-dev 2004-01-08 19:08:29 UTC
Created attachment 23434 [details]
slabinfo.after_updatedb
Comment 16 r1chard 2004-01-08 20:47:05 UTC
Created attachment 23438 [details]
Post-patch /proc/slabinfo and vmstat 5 

For comparison with attachment 23362 [details] (again using an updatedb command, this one
starting from just after a system boot). 

My impression post patch is that though after the command has finished, memory
is not immediatly released, it /is/ made available upon demand. Is that what is
supposed to happen - I have not taken much notice of the way memory management
works in Linux, since to this point it just worked for me :)

RG
Comment 17 Alex Buell 2004-01-09 01:48:51 UTC
Patch I gave really does fixes a memory leak problem with htree in ext3 filesystem. As for reiserfs, this perhaps should be reported to the reiserfs people. It might be that we're seeing two different problems causing the same thing. 
Comment 18 Anders Hellgren gentoo-dev 2004-01-12 00:29:26 UTC
A FYI. This forum thread seems to be related to the memory leak. http://forums.gentoo.org/viewtopic.php?t=123309
Comment 19 hodak 2004-01-13 01:17:39 UTC
Is this bug also for reiserfs problem (same symptoms, but on reiser) and is NOT fixed by the proposed patch or should I submit a new bug report?
Comment 20 Brian Jackson (RETIRED) gentoo-dev 2004-01-13 17:05:52 UTC
If your system is able to reclaim the memory it's not a bug. That's normal fs caching behavior. The kernel will aggresively cache stuff as long as there is free memory. It's common for me to only see 10MB free on my 2GB workstation. In fact my memory is almost purely taken by the cache (and X).
Comment 21 Tom Wesley 2004-01-14 00:51:32 UTC
Maybe someone can describe this in dumb-man terms for me.  I use gkrellm2 which has 3 marks on the memory, the main one being used, second buffers, third cache.
After running updatedb the used memory goes up to about halfway (512MB) and doesn't seem to lower until I reboot the machine - I've left it about a day or so.

This all happens on a reisersfs system using kernel 2.6.2-gentoo.
Comment 22 hodak 2004-01-14 03:02:04 UTC
OK, here is what it looks on my system with Reiser:
Freshly rebooted system

 # free -mt
             total       used       free     shared    buffers     cached
Mem:           503         49        453          0          7         21
-/+ buffers/cache:         20        483
Swap:          980          0        980
Total:        1484         49       1434
#updatedb
# free -mt
             total       used       free     shared    buffers     cached
Mem:           503        494          9          0        235         25
-/+ buffers/cache:        233        270
Swap:          980          0        980
Total:        1484        494        990

I know about caching behavior and according to my understanding available memory is basically free+cached+buffers. The output I posted above shows me that update db ate more than 200MB of memory. As for as reclaiming letting this stand overnight improves the situation very little (maybe because something were moved to swap) and more than 200MB is missing. 
I think this is a bug and not caching behavior. 
Comment 23 Brian Jackson (RETIRED) gentoo-dev 2004-01-14 09:45:19 UTC
It's not a bug, that's what every system I've ever seen does. I'm closing this bug as I've applied the ext3 fix to -r5.
Comment 24 Tom Wesley 2004-01-14 09:47:36 UTC
Surely the point is that it doesn't do it with the fix applied for ext3, so it shouldn't do it for reiserfs either...
Comment 25 Erik Anderson 2005-01-22 14:01:45 UTC
 ran "emerge sync" (uses rsync).. and the ram used up was _not_ able to be reclaimed...  when i went to play a game i just simply hit swap space.. and after exitting the game.. the same amount of ram used after rsyncing was still un-reclaimable...
same for updatedb... can't reclaim it..