In the past week I have had 3 instances of total machine lockup with load rising to 30 and loss of keyboard and mouse and lots of deadlocked processes. I am running kernel-3.2.5 with automatic cpu cgroups so I suspected a rouge program depleting memory. One of the programs I suspected was liferea because it is not always running but was running the last time I locked up. I therefore decided to put liferea in a memory cgroup and limit it to 1G of memory. After several hours I had an OOM killer. See the syslog attached Reproducible: Always
Created attachment 301331 [details] syslog showing OOM caused by liferea
Extra Information for net-news/liferea-1.8.0::installed: >>> Running ebuild phase killold as harrisl:harrisl... >>> Starting builtin_killold >>> Done builtin_killold >>> Completed ebuild phase killold >>> Running ebuild phases initmisc infovars info as harrisl:harrisl... >>> Starting builtin_initmisc bunzip2 < /var/db/pkg/net-news/liferea-1.8.0/environment.bz2 > /var/tmp/paludis/environment-net-news-liferea-1.8.0-7017 ebuild_scrub_environment /var/tmp/paludis/environment-net-news-liferea-1.8.0-7017 ebuild_safe_source /var/tmp/paludis/environment-net-news-liferea-1.8.0-7017 rm /var/tmp/paludis/environment-net-news-liferea-1.8.0-7017 >>> Done builtin_initmisc >>> Starting builtin_infovars ACCEPT_KEYWORDS=amd64 CBUILD=x86_64-pc-linux-gnu CFLAGS=-march=native -O2 -pipe CHOST=x86_64-pc-linux-gnu CONFIG_PROTECT=/etc CONFIG_PROTECT_MASK=/etc/env.d /etc/gconf CPPFLAGS= CTARGET= CXXFLAGS=-march=native -O2 -pipe DISTDIR=/var/db/pkg/net-news/liferea-1.8.0 FEATURES= FFLAGS= GENTOO_MIRRORS= INSTALL_MASK= LANG= LC_ALL=C LDFLAGS=-Wl,-O1 -Wl,--as-needed LINGUAS=en MAKEOPTS=-j13 PORTAGE_COMPRESS= PORTAGE_COMPRESS_FLAGS= PORTAGE_CONFIGROOT= PORTAGE_RSYNC_EXTRA_OPTS= PORTAGE_RSYNC_OPTS= PORTAGE_TMPDIR=/var/tmp/paludis PORTDIR=/var/db/pkg/net-news/liferea-1.8.0 PORTDIR_OVERLAY= SYNC= USE=libnotify amd64 alsa_cards_ali5451 alsa_cards_als4000 alsa_cards_atiixp alsa_cards_atiixp-modem alsa_cards_bt87x alsa_cards_ca0106 alsa_cards_cmipci alsa_cards_emu10k1x alsa_cards_ens1370 alsa_cards_ens1371 alsa_cards_es1938 alsa_cards_es1968 alsa_cards_fm801 alsa_cards_hda-intel alsa_cards_intel8x0 alsa_cards_intel8x0m alsa_cards_maestro3 alsa_cards_trident alsa_cards_usb-audio alsa_cards_via82xx alsa_cards_via82xx-modem alsa_cards_ymfpci alsa_pcm_plugins_adpcm alsa_pcm_plugins_alaw alsa_pcm_plugins_asym alsa_pcm_plugins_copy alsa_pcm_plugins_dmix alsa_pcm_plugins_dshare alsa_pcm_plugins_dsnoop alsa_pcm_plugins_empty alsa_pcm_plugins_extplug alsa_pcm_plugins_file alsa_pcm_plugins_hooks alsa_pcm_plugins_iec958 alsa_pcm_plugins_ioplug alsa_pcm_plugins_ladspa alsa_pcm_plugins_lfloat alsa_pcm_plugins_linear alsa_pcm_plugins_meter alsa_pcm_plugins_mmap_emul alsa_pcm_plugins_mulaw alsa_pcm_plugins_multi alsa_pcm_plugins_null alsa_pcm_plugins_plug alsa_pcm_plugins_rate alsa_pcm_plugins_route alsa_pcm_plugins_share alsa_pcm_plugins_shm alsa_pcm_plugins_softvol apache2_modules_actions apache2_modules_alias apache2_modules_auth_basic apache2_modules_authn_alias apache2_modules_authn_anon apache2_modules_authn_dbm apache2_modules_authn_default apache2_modules_authn_file apache2_modules_authz_dbm apache2_modules_authz_default apache2_modules_authz_groupfile apache2_modules_authz_host apache2_modules_authz_owner apache2_modules_authz_user apache2_modules_autoindex apache2_modules_cache apache2_modules_cgi apache2_modules_cgid apache2_modules_dav apache2_modules_dav_fs apache2_modules_dav_lock apache2_modules_deflate apache2_modules_dir apache2_modules_disk_cache apache2_modules_env apache2_modules_expires apache2_modules_ext_filter apache2_modules_file_cache apache2_modules_filter apache2_modules_headers apache2_modules_include apache2_modules_info apache2_modules_log_config apache2_modules_logio apache2_modules_mem_cache apache2_modules_mime apache2_modules_mime_magic apache2_modules_negotiation apache2_modules_rewrite apache2_modules_setenvif apache2_modules_speling apache2_modules_status apache2_modules_unique_id apache2_modules_userdir apache2_modules_usertrack apache2_modules_vhost_alias calligra_features_braindump calligra_features_flow calligra_features_karbon calligra_features_kexi calligra_features_krita calligra_features_plan calligra_features_stage calligra_features_tables calligra_features_words cameras_ptp2 collectd_plugins_df collectd_plugins_interface collectd_plugins_irq collectd_plugins_load collectd_plugins_memory collectd_plugins_rrdtool collectd_plugins_swap collectd_plugins_syslog elibc_glibc gpsd_protocols_aivdm gpsd_protocols_ashtech gpsd_protocols_earthmate gpsd_protocols_evermore gpsd_protocols_fv18 gpsd_protocols_garmin gpsd_protocols_garmintxt gpsd_protocols_gpsclock gpsd_protocols_itrax gpsd_protocols_mtk3301 gpsd_protocols_navcom gpsd_protocols_nmea gpsd_protocols_ntrip gpsd_protocols_oceanserver gpsd_protocols_oldstyle gpsd_protocols_oncore gpsd_protocols_rtcm104v2 gpsd_protocols_rtcm104v3 gpsd_protocols_sirf gpsd_protocols_superstar2 gpsd_protocols_timing gpsd_protocols_tnt gpsd_protocols_tripmate gpsd_protocols_tsip gpsd_protocols_ubx input_devices_evdev input_devices_wacom kernel_linux lcd_devices_bayrad lcd_devices_cfontz lcd_devices_cfontz633 lcd_devices_glk lcd_devices_hd44780 lcd_devices_lb216 lcd_devices_lcdm001 lcd_devices_mtxorb lcd_devices_ncurses lcd_devices_text linguas_en php_targets_php5-3 ruby_targets_ruby18 ruby_targets_ruby19 userland_GNU video_cards_fbdev video_cards_nv video_cards_nvidia video_cards_vesa xtables_addons_account xtables_addons_chaos xtables_addons_condition xtables_addons_delude xtables_addons_dhcpmac xtables_addons_fuzzy xtables_addons_geoip xtables_addons_iface xtables_addons_ipmark xtables_addons_ipp2p xtables_addons_ipset xtables_addons_ipv4options xtables_addons_length2 xtables_addons_logmark xtables_addons_lscan xtables_addons_pknock xtables_addons_psd xtables_addons_quota2 xtables_addons_rawnat xtables_addons_steal xtables_addons_sysrq xtables_addons_tarpit xtables_addons_tee amd64 >>> Done builtin_infovars >>> Starting pkg_info >>> Done pkg_info >>> Completed ebuild phases initmisc infovars info >>> Running ebuild phase tidyup as harrisl:harrisl... >>> Starting builtin_tidyup rm -fr /var/tmp/paludis/net-news-liferea-1.8.0-info >>> Done builtin_tidyup >>> Completed ebuild phase tidyup Extra Information for net-news/liferea-1.8.0:0::gentoo: >>> Running ebuild phase killold as harrisl:harrisl... >>> Starting builtin_killold >>> Done builtin_killold >>> Completed ebuild phase killold >>> Running ebuild phases initmisc infovars info as harrisl:harrisl... >>> Starting builtin_initmisc >>> Done builtin_initmisc >>> Starting builtin_infovars ACCEPT_KEYWORDS=amd64 CBUILD=x86_64-pc-linux-gnu CFLAGS=-march=native -O2 -pipe CHOST=x86_64-pc-linux-gnu CONFIG_PROTECT= CONFIG_PROTECT_MASK= CPPFLAGS= CTARGET= CXXFLAGS=-march=native -O2 -pipe DISTDIR=/usr/portage/distfiles FEATURES= FFLAGS= GENTOO_MIRRORS= INSTALL_MASK= LANG= LC_ALL=C LDFLAGS=-Wl,-O1 -Wl,--as-needed LINGUAS=en MAKEOPTS=-j13 PORTAGE_COMPRESS= PORTAGE_COMPRESS_FLAGS= PORTAGE_CONFIGROOT= PORTAGE_RSYNC_EXTRA_OPTS= PORTAGE_RSYNC_OPTS= PORTAGE_TMPDIR=/var/tmp/paludis PORTDIR=/usr/portage PORTDIR_OVERLAY= SYNC= USE=libnotify amd64 alsa_cards_ali5451 alsa_cards_als4000 alsa_cards_atiixp alsa_cards_atiixp-modem alsa_cards_bt87x alsa_cards_ca0106 alsa_cards_cmipci alsa_cards_emu10k1x alsa_cards_ens1370 alsa_cards_ens1371 alsa_cards_es1938 alsa_cards_es1968 alsa_cards_fm801 alsa_cards_hda-intel alsa_cards_intel8x0 alsa_cards_intel8x0m alsa_cards_maestro3 alsa_cards_trident alsa_cards_usb-audio alsa_cards_via82xx alsa_cards_via82xx-modem alsa_cards_ymfpci alsa_pcm_plugins_adpcm alsa_pcm_plugins_alaw alsa_pcm_plugins_asym alsa_pcm_plugins_copy alsa_pcm_plugins_dmix alsa_pcm_plugins_dshare alsa_pcm_plugins_dsnoop alsa_pcm_plugins_empty alsa_pcm_plugins_extplug alsa_pcm_plugins_file alsa_pcm_plugins_hooks alsa_pcm_plugins_iec958 alsa_pcm_plugins_ioplug alsa_pcm_plugins_ladspa alsa_pcm_plugins_lfloat alsa_pcm_plugins_linear alsa_pcm_plugins_meter alsa_pcm_plugins_mmap_emul alsa_pcm_plugins_mulaw alsa_pcm_plugins_multi alsa_pcm_plugins_null alsa_pcm_plugins_plug alsa_pcm_plugins_rate alsa_pcm_plugins_route alsa_pcm_plugins_share alsa_pcm_plugins_shm alsa_pcm_plugins_softvol apache2_modules_actions apache2_modules_alias apache2_modules_auth_basic apache2_modules_authn_alias apache2_modules_authn_anon apache2_modules_authn_dbm apache2_modules_authn_default apache2_modules_authn_file apache2_modules_authz_dbm apache2_modules_authz_default apache2_modules_authz_groupfile apache2_modules_authz_host apache2_modules_authz_owner apache2_modules_authz_user apache2_modules_autoindex apache2_modules_cache apache2_modules_cgi apache2_modules_cgid apache2_modules_dav apache2_modules_dav_fs apache2_modules_dav_lock apache2_modules_deflate apache2_modules_dir apache2_modules_disk_cache apache2_modules_env apache2_modules_expires apache2_modules_ext_filter apache2_modules_file_cache apache2_modules_filter apache2_modules_headers apache2_modules_include apache2_modules_info apache2_modules_log_config apache2_modules_logio apache2_modules_mem_cache apache2_modules_mime apache2_modules_mime_magic apache2_modules_negotiation apache2_modules_rewrite apache2_modules_setenvif apache2_modules_speling apache2_modules_status apache2_modules_unique_id apache2_modules_userdir apache2_modules_usertrack apache2_modules_vhost_alias calligra_features_braindump calligra_features_flow calligra_features_karbon calligra_features_kexi calligra_features_krita calligra_features_plan calligra_features_stage calligra_features_tables calligra_features_words cameras_ptp2 collectd_plugins_df collectd_plugins_interface collectd_plugins_irq collectd_plugins_load collectd_plugins_memory collectd_plugins_rrdtool collectd_plugins_swap collectd_plugins_syslog elibc_glibc gpsd_protocols_aivdm gpsd_protocols_ashtech gpsd_protocols_earthmate gpsd_protocols_evermore gpsd_protocols_fv18 gpsd_protocols_garmin gpsd_protocols_garmintxt gpsd_protocols_gpsclock gpsd_protocols_itrax gpsd_protocols_mtk3301 gpsd_protocols_navcom gpsd_protocols_nmea gpsd_protocols_ntrip gpsd_protocols_oceanserver gpsd_protocols_oldstyle gpsd_protocols_oncore gpsd_protocols_rtcm104v2 gpsd_protocols_rtcm104v3 gpsd_protocols_sirf gpsd_protocols_superstar2 gpsd_protocols_timing gpsd_protocols_tnt gpsd_protocols_tripmate gpsd_protocols_tsip gpsd_protocols_ubx input_devices_evdev input_devices_wacom kernel_linux lcd_devices_bayrad lcd_devices_cfontz lcd_devices_cfontz633 lcd_devices_glk lcd_devices_hd44780 lcd_devices_lb216 lcd_devices_lcdm001 lcd_devices_mtxorb lcd_devices_ncurses lcd_devices_text linguas_en php_targets_php5-3 ruby_targets_ruby18 ruby_targets_ruby19 userland_GNU video_cards_fbdev video_cards_nv video_cards_nvidia video_cards_vesa xtables_addons_account xtables_addons_chaos xtables_addons_condition xtables_addons_delude xtables_addons_dhcpmac xtables_addons_fuzzy xtables_addons_geoip xtables_addons_iface xtables_addons_ipmark xtables_addons_ipp2p xtables_addons_ipset xtables_addons_ipv4options xtables_addons_length2 xtables_addons_logmark xtables_addons_lscan xtables_addons_pknock xtables_addons_psd xtables_addons_quota2 xtables_addons_rawnat xtables_addons_steal xtables_addons_sysrq xtables_addons_tarpit xtables_addons_tee amd64
I disagree with your analysis. In the middle of the log you posted, it is decided that chrome ( www-client/{google-chrome,chromium} ?) is responsible for the fact that liferea ran out of memory - liferea merely triggered it. Feb 8 18:50:09 harrisl-desktop kernel: Out of memory: Kill process 8365 (chrome) score 304 or sacrifice child Feb 8 18:50:09 harrisl-desktop kernel: Killed process 8365 (chrome) total-vm:1305660kB, anon-rss:247792kB, file-rss:23880kB Not liferea, then. If you could tell us which package provided the executable for the `chrome' process, we might be able to put you through to its maintainers (but running out of RAM is usually not a software problem).
I am running chromium-17.0.963.46-r1 You could be right but I am still very suspicious of liferea. Why would liferea have launched 95 threads? BTW, this machine has 24G of ram and 40G of swap. The swap wasn't touched because the memory cgroup limit launched the OOM-killer when liferea went over. Given how much RAM this machine has, it is likely a software bug is the cause. It is very hard for me to run out of memory.
OK, then try either: 1) not running liferea for a few days, or 2) monitoring liferea's memory usage properly for a few days.
I started liferea this time in a shell which is in a memory cgroup so all of its threads will be jailed. It started with 6 threads. Lets see what happens over night.
Last night the OOM-killer ran at 4:38 which took down X. Since liferea is triggering the OOM in a memory cgroup, I don't understand why the OOM-killer is running over the entire task list even though the machine is clearly not out of memory (swap wasn't touched). Either I am not understanding cgroups or there is some sort of bug. Today I ran my usual load of browsers and vms without liferea and without incident. For tonight, I turned off the OOM-killer both for the memory cgroup and the entire machine. Liferea is running in a memory limited cgroup with memory.limit_in_bytes set to 500M. The program launched with 6 threads, using 48M of ram. I am writing memory.usage_in_bytes to a file once a minute from the liferea cgroup. In the last hour memory usage has grown to 53M. It is my understanding that if liferea hits 500M the program will freeze and the rest of the machine should be ok. If something else is causing the problem we should find out tonight.
After running for several hours without interaction, under the memory cgroup, liferea exited all threads on a sig 11 segfault. The segfault message was repeated in the shell liferea was running in 47 times which I assume means that 47 threads were killed. Also memory.max_usage_in_bytes = 524259328 showing the 500M limit was hit. There definitely seems to be a memory bomb in liferea. I am now running it under gdb in the same shell. Please escalate this bug and relabel it as a confirmed memory leak in Liferea It would also be helpful if someone else could test it.
Liferea has been running stablely under GDB for 8 hours. This opens up the real probability that the memory - thread explosions in Liferea are caused by a race condition that GDB is masking.
