A Gentoo server has been running for quite a while and during the last 2 days has had "page allocation failure" messages in /var/log/messages. Finally it froze completely. Only ping kept responding; all other services were unusable. Also, the console did not respond so I could not login nor do anything except reset/reboot the server (pushing the button). Reproducible: Didn't try Actual Results: # uname -a Linux INF-BL07 2.6.15-gentoo-r5 #1 SMP Mon Mar 6 12:09:37 CET 2006 x86_64 Intel(R) Xeon(TM) CPU 3.20GHz GenuineIntel GNU/Linux Sep 18 10:44:04 INF-BL07 kswapd0: page allocation failure. order:0, mode:0x20 Sep 18 10:44:04 INF-BL07 Sep 18 10:44:04 INF-BL07 Call Trace: <IRQ> <ffffffff8015a823>{__alloc_pages+666} <ffffffff8015db19>{cache_alloc_refill+661} Sep 18 10:44:04 INF-BL07 <ffffffff8015d464>{__kmalloc+104} <ffffffff8048b1fd>{__alloc_skb+104} Sep 18 10:44:04 INF-BL07 <ffffffff88165170>{:e1000:e1000_alloc_rx_buffers+117} Sep 18 10:44:04 INF-BL07 <ffffffff881674c8>{:e1000:e1000_clean_rx_irq+1216} <ffffffff80491017>{process_backlog+156} Sep 18 10:44:04 INF-BL07 <ffffffff88166f9b>{:e1000:e1000_intr+143} <ffffffff80154aaf>{handle_IRQ_event+41} Sep 18 10:44:04 INF-BL07 <ffffffff80154b94>{__do_IRQ+178} <ffffffff8011088a>{do_IRQ+47} Sep 18 10:44:04 INF-BL07 <ffffffff8010e080>{ret_from_intr+0} <EOI> <ffffffff8016bf45>{page_check_address+40} Sep 18 10:44:04 INF-BL07 <ffffffff8057e8bf>{.text.lock.spinlock+2} <ffffffff8016c19d>{page_referenced_file+75} Sep 18 10:44:04 INF-BL07 <ffffffff8016c317>{page_referenced+242} <ffffffff80160f12>{shrink_zone+612} Sep 18 10:44:04 INF-BL07 <ffffffff8057d283>{thread_return+0} <ffffffff8015d392>{kmem_freepages+194} Sep 18 10:44:04 INF-BL07 <ffffffff8014aabd>{autoremove_wake_function+0} <ffffffff8016204d>{balance_pgdat+618} Sep 18 10:44:04 INF-BL07 <ffffffff801622f4>{kswapd+301} <ffffffff8014aabd>{autoremove_wake_function+0} Sep 18 10:44:04 INF-BL07 <ffffffff8014aabd>{autoremove_wake_function+0} <ffffffff8010e9ce>{child_rip+8} Sep 18 10:44:04 INF-BL07 <ffffffff801621c7>{kswapd+0} <ffffffff8010e9c6>{child_rip+0} Sep 18 10:44:04 INF-BL07 Sep 18 10:44:04 INF-BL07 Mem-info: Sep 18 10:44:04 INF-BL07 DMA per-cpu: Sep 18 10:44:04 INF-BL07 cpu 0 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 0 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 1 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 1 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 2 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 2 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 3 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 3 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 DMA32 per-cpu: Sep 18 10:44:04 INF-BL07 cpu 0 hot: low 0, high 186, batch 31 used:31 Sep 18 10:44:04 INF-BL07 cpu 0 cold: low 0, high 62, batch 15 used:12 Sep 18 10:44:04 INF-BL07 cpu 1 hot: low 0, high 186, batch 31 used:99 Sep 18 10:44:04 INF-BL07 cpu 1 cold: low 0, high 62, batch 15 used:55 Sep 18 10:44:04 INF-BL07 cpu 2 hot: low 0, high 186, batch 31 used:97 Sep 18 10:44:04 INF-BL07 cpu 2 cold: low 0, high 62, batch 15 used:61 Sep 18 10:44:04 INF-BL07 cpu 3 hot: low 0, high 186, batch 31 used:30 Sep 18 10:44:04 INF-BL07 greylist[29295]: SMTP: HELO: 250 hello Sep 18 10:44:04 INF-BL07 cpu 3 cold: low 0,s+40} Sep 18 10:44:04 INF-BL07 <ffffffff8057e8bf>{.text.lock.spinlock+2} <ffffffff8016c19d>{page_referenced_file+75} Sep 18 10:44:04 INF-BL07 <ffffffff8016c317>{page_referenced+242} <ffffffff80160f12>{shrink_zone+612} Sep 18 10:44:04 INF-BL07 <ffffffff8057d283>{thread_return+0} <ffffffff8015d392>{kmem_freepages+194} Sep 18 10:44:04 INF-BL07 <ffffffff8014aabd>{autoremove_wake_function+0} <ffffffff8016204d>{balance_pgdat+618} Sep 18 10:44:04 INF-BL07 <ffffffff801622f4>{kswapd+301} <ffffffff8014aabd>{autoremove_wake_function+0} Sep 18 10:44:04 INF-BL07 <ffffffff8014aabd>{autoremove_wake_function+0} <ffffffff8010e9ce>{child_rip+8} Sep 18 10:44:04 INF-BL07 <ffffffff801621c7>{kswapd+0} <ffffffff8010e9c6>{child_rip+0} Sep 18 10:44:04 INF-BL07 Sep 18 10:44:04 INF-BL07 Mem-info: Sep 18 10:44:04 INF-BL07 DMA per-cpu: Sep 18 10:44:04 INF-BL07 cpu 0 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 0 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 1 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 1 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 2 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 2 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 3 hot: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 cpu 3 cold: low 0, high 0, batch 1 used:0 Sep 18 10:44:04 INF-BL07 DMA32 per-cpu: Sep 18 10:44:04 INF-BL07 cpu 0 hot: low 0, high 186, batch 31 used:31 Sep 18 10:44:04 INF-BL07 cpu 0 cold: low 0, high 62, batch 15 used:12 Sep 18 10:44:04 INF-BL07 cpu 1 hot: low 0, high 186, batch 31 used:121 Sep 18 10:44:04 INF-BL07 cpu 1 cold: low 0, high 62, batch 15 used:55 Sep 18 10:44:04 INF-BL07 cpu 2 hot: low 0, high 186, batch 31 used:128 Sep 18 10:44:04 INF-BL07 cpu 2 cold: low 0, high 62, batch 15 used:61 Sep 18 10:44:04 INF-BL07 cpu 3 hot: low 0, high 186, batch 31 used:30 Sep 18 10:44:04 INF-BL07 cpu 3 cold: low 0, high 62, batch 15 used:54 Sep 18 10:44:04 INF-BL07 Normal per-cpu: empty Sep 18 10:44:04 INF-BL07 HighMem per-cpu: empty Sep 18 10:44:04 INF-BL07 Free pages: 11908kB (0kB HighMem) Sep 18 10:44:04 INF-BL07 Active:547228 inactive:234449 dirty:0 writeback:2 unstable:0 free:2977 slab:43641 mapped:701398 pagetables:26466 Sep 18 10:44:04 INF-BL07 DMA free:9108kB min:16kB low:20kB high:24kB active:0kB inactive:0kB present:8660kB pages_scanned:5543358 all_unreclaimable? yes Sep 18 10:44:04 INF-BL07 lowmem_reserve[]: 0 3511 3511 3511 Sep 18 10:44:04 INF-BL07 DMA32 free:2800kB min:7572kB low:9464kB high:11356kB active:2188912kB inactive:937796kB present:3596256kB pages_scanned:34487 all_unreclaimable? no Sep 18 10:44:04 INF-BL07 lowmem_reserve[]: 0 0 0 0 Sep 18 10:44:04 INF-BL07 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 18 10:44:04 INF-BL07 lowmem_reserve[]: 0 0 0 0 Sep 18 10:44:04 INF-BL07 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Sep 18 10:44:04 INF-BL07 lowmem_reserve[]: 0 0 0 0 Sep 18 10:44:04 INF-BL07 DMA: 5*4kB 6*8kB 3*16kB 5*32kB 4*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 9108kB Sep 18 10:44:04 INF-BL07 DMA32: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2800kB Sep 18 10:44:04 INF-BL07 Normal: empty Sep 18 10:44:04 INF-BL07 HighMem: empty Sep 18 10:44:04 INF-BL07 Swap cache: add 380349, delete 380224, find 1315769/1327977, race 0+571 Sep 18 10:44:04 INF-BL07 Free swap = 0kB Sep 18 10:44:04 INF-BL07 Total swap = 1052248kB Sep 18 10:44:04 INF-BL07 Free swap: 0kB Sep 18 10:44:04 INF-BL07 1048576 pages of RAM Sep 18 10:44:04 INF-BL07 166132 reserved pages Sep 18 10:44:04 INF-BL07 1196901 pages shared Sep 18 10:44:04 INF-BL07 128 pages swap cached Sep 18 10:44:04 INF-BL07 kswapd0: page allocation failure. order:0, mode:0x20 I know it's an "old" kernel but does someone know what could have happened? (after reboot everything seems to be ok although I wrote a script so that I get notified of such failures and reboot programmatically before it freezes and users get "stuck")
Please note that this is not a support forum. Also, your kernel version is unsupported. If you can reproduce with uptodate kernels (>=gentoo-sources-2.6.22-r5), feel free to reopen.
