339785 – sys-kernel/gentoo-sources-2.6.34-r6: huge load while idle

Bug 339785 - sys-kernel/gentoo-sources-2.6.34-r6: huge load while idle

Summary: sys-kernel/gentoo-sources-2.6.34-r6: huge load while idle

Status:	RESOLVED UPSTREAM

Alias:	None

Product:	Gentoo Linux
Classification:	Unclassified
Component:	[OLD] Core system (show other bugs)
Hardware:	All Linux

Importance:	High critical (vote)
Assignee:	Gentoo Kernel Bug Wranglers and Kernel Maintainers

URL:
Whiteboard:
Keywords:

Depends on:
Blocks:

Reported:	2010-10-05 06:22 UTC by Marek Królikowski
Modified:	2011-04-13 05:27 UTC (History)
CC List:	3 users (show)

See Also:
Package list:
Runtime testing required:	---

Attachments
Kernel config file (config,85.30 KB, text/plain) 2011-01-03 08:04 UTC, Marek Królikowski	Details
jffnms - Before and after change kernel (i6-4d2f018a027da.png,17.66 KB, image/png) 2011-01-13 13:45 UTC, Marek Królikowski	Details
JFFNMS graph of Load Average on 2.6.32-hard-r33 (jffnms.loadaverage.on.2.6.32.png,18.90 KB, image/png) 2011-01-14 06:18 UTC, Marek Królikowski	Details
.config from 2.6.37 (config-2.6.37,86.53 KB, text/plain) 2011-02-02 06:05 UTC, Marek Królikowski	Details
Graph from jffnms - load average (jffnms.loadaverage.on.2.6.37.png,20.79 KB, text/plain) 2011-02-02 06:07 UTC, Marek Królikowski	Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Marek Królikowski 2010-10-05 06:22:44 UTC

After compiling sys-kernel/gentoo-sources-2.6.34-r6 with make oldconfig and reboot my server got huge load in idle:
load average: 1.97, 1.98, 1.98
But after i reboot server at old sys-kernel/gentoo-sources-2.6.32-r7 server got:
load average: 0.07, 0.04, 0.02


Reproducible: Always

Steps to Reproduce:
1. emerge gentoo-sources
2. reboot
3. after 10 minutes check load

Actual Results:  
load average: 1.97, 1.98, 1.98

Expected Results:  
load average: 0.07, 0.04, 0.02

Comment 1 Marek Królikowski 2010-10-05 06:36:17 UTC

ps aux, top, iotop other don`t show any process who get so many resources...

Comment 2 Michael Weber (RETIRED) gentoo-dev

2010-10-06 08:51:25 UTC

Hello,

this might be tricky to debug. Afaik iowait (processes waiting for disk/network operations, not consuming cpu cycles) is mentioned in this load value.

Do you recognize any lag or performance degrease?

Please comment a full `ps aux` output, to give us a look at the kernel processes.

Michael

Comment 3 Marek Królikowski 2010-10-06 09:17:21 UTC

Hello
Yes server is slow on kernel 2.6.34+... this is production machine so i do downgrade to 2.6.32...
This is what i got from 2.6.34+
WIZJA2 ~ # ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  2.6  0.0   1732   564 ?        Ss   11:10   0:05 init [3]
root         2  0.0  0.0      0     0 ?        S    11:10   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/0]
root         5  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/0]
root         6  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/1]
root         7  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/1]
root         8  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/1]
root         9  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/2]
root        10  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/2]
root        11  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/2]
root        12  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/3]
root        13  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/3]
root        14  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/3]
root        15  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/4]
root        16  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/4]
root        17  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/4]
root        18  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/5]
root        19  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/5]
root        20  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/5]
root        21  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/6]
root        22  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/6]
root        23  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/6]
root        24  0.0  0.0      0     0 ?        S    11:10   0:00 [migration/7]
root        25  0.0  0.0      0     0 ?        S    11:10   0:00 [ksoftirqd/7]
root        26  0.0  0.0      0     0 ?        S    11:10   0:00 [watchdog/7]
root        27  0.0  0.0      0     0 ?        S    11:10   0:00 [events/0]
root        28  0.0  0.0      0     0 ?        S    11:10   0:00 [events/1]
root        29  0.0  0.0      0     0 ?        S    11:10   0:00 [events/2]
root        30  0.0  0.0      0     0 ?        S    11:10   0:00 [events/3]
root        31  0.0  0.0      0     0 ?        S    11:10   0:00 [events/4]
root        32  0.0  0.0      0     0 ?        S    11:10   0:00 [events/5]
root        33  0.0  0.0      0     0 ?        S    11:10   0:00 [events/6]
root        34  0.0  0.0      0     0 ?        S    11:10   0:00 [events/7]
root        35  0.0  0.0      0     0 ?        S    11:10   0:00 [khelper]
root        40  0.0  0.0      0     0 ?        S    11:10   0:00 [async/mgr]
root       164  0.0  0.0      0     0 ?        S    11:10   0:00 [sync_supers]
root       166  0.0  0.0      0     0 ?        S    11:10   0:00 [bdi-default]
root       168  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/0]
root       169  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/1]
root       170  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/2]
root       171  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/3]
root       172  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/4]
root       173  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/5]
root       174  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/6]
root       175  0.0  0.0      0     0 ?        S    11:10   0:00 [kblockd/7]
root       177  0.0  0.0      0     0 ?        S    11:10   0:00 [kacpid]
root       178  0.0  0.0      0     0 ?        S    11:10   0:00 [kacpi_notify]
root       179  0.0  0.0      0     0 ?        S    11:10   0:00 [kacpi_hotplug]
root       302  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_aux]
root       303  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/0]
root       304  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/1]
root       305  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/2]
root       306  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/3]
root       307  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/4]
root       308  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/5]
root       309  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/6]
root       310  0.0  0.0      0     0 ?        S    11:10   0:00 [ata_sff/7]
root       312  0.0  0.0      0     0 ?        S    11:10   0:00 [kseriod]
root       344  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/0]
root       345  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/1]
root       346  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/2]
root       347  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/3]
root       348  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/4]
root       349  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/5]
root       350  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/6]
root       351  0.0  0.0      0     0 ?        S    11:10   0:00 [rpciod/7]
root       414  0.0  0.0      0     0 ?        S    11:10   0:00 [khungtaskd]
root       415  0.0  0.0      0     0 ?        S    11:10   0:00 [kswapd0]
root       416  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/0]
root       417  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/1]
root       418  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/2]
root       419  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/3]
root       420  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/4]
root       421  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/5]
root       422  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/6]
root       423  0.0  0.0      0     0 ?        S    11:10   0:00 [aio/7]
root       424  0.0  0.0      0     0 ?        S    11:10   0:00 [nfsiod]
root       425  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/0]
root       426  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/1]
root       427  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/2]
root       428  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/3]
root       429  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/4]
root       430  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/5]
root       431  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/6]
root       432  0.0  0.0      0     0 ?        S    11:10   0:00 [crypto/7]
root      1090  0.0  0.0      0     0 ?        S    11:10   0:00 [iscsi_eh]
root      1091  0.0  0.0      0     0 ?        S    11:10   0:00 [scsi_eh_0]
root      1111  0.0  0.0      0     0 ?        S    11:10   0:00 [scsi_eh_1]
root      1114  0.0  0.0      0     0 ?        S    11:10   0:00 [scsi_eh_2]
root      1119  0.0  0.0      0     0 ?        S    11:10   0:00 [scsi_eh_3]
root      1122  0.0  0.0      0     0 ?        S    11:10   0:00 [scsi_eh_4]
root      1218  0.0  0.0      0     0 ?        S    11:10   0:00 [kpsmoused]
root      2484  0.0  0.0      0     0 ?        S    11:10   0:00 [cciss_scan]
root      2666  0.0  0.0      0     0 ?        S    11:10   0:00 [khubd]
root      2901  0.0  0.0      0     0 ?        S    11:10   0:00 [usbhid_resumer]
root      3004  0.0  0.0      0     0 ?        S    11:10   0:00 [khpsbpkt]
root      3221  0.0  0.0      0     0 ?        S    11:10   0:00 [kstriped]
root      3493  0.0  0.0      0     0 ?        S    11:10   0:00 [ksnapd]
root      4064  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsIO]
root      4065  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4066  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4067  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4068  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4069  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4070  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4071  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4072  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsCommit]
root      4073  0.0  0.0      0     0 ?        S    11:10   0:00 [jfsSync]
root      4146  0.0  0.0      0     0 ?        S    11:10   0:00 [xfs_mru_cache]
root      4147  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/0]
root      4148  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/1]
root      4149  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/2]
root      4150  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/3]
root      4151  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/4]
root      4152  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/5]
root      4153  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/6]
root      4154  0.0  0.0      0     0 ?        S    11:10   0:00 [xfslogd/7]
root      4155  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/0]
root      4156  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/1]
root      4157  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/2]
root      4158  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/3]
root      4159  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/4]
root      4160  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/5]
root      4161  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/6]
root      4162  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsdatad/7]
root      4163  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/0]
root      4164  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/1]
root      4165  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/2]
root      4166  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/3]
root      4167  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/4]
root      4168  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/5]
root      4169  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/6]
root      4170  0.0  0.0      0     0 ?        S    11:10   0:00 [xfsconvertd/7]
root      4317  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/0]
root      4318  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/1]
root      4319  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/2]
root      4320  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/3]
root      4321  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/4]
root      4322  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/5]
root      4323  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/6]
root      4324  0.0  0.0      0     0 ?        S    11:10   0:00 [reiserfs/7]
root      4423  0.0  0.0   2116   828 ?        S<s  11:10   0:00 /sbin/udevd --daemon
root      5103  0.0  0.0      0     0 ?        S    11:11   0:00 [flush-8:0]
root      6348  0.0  0.0   1784   648 ?        Ss   11:11   0:00 /usr/sbin/syslogd -m 0 -r -i 0.0.0.0
root      6358  0.0  0.0   1736   408 ?        Ss   11:11   0:00 /usr/sbin/klogd -c 3 -2
named     7462  0.0  0.0  16016 13164 ?        Ss   11:11   0:00 /usr/sbin/named -u named -n 8 -t /chroot
bin       7576  0.0  0.0   1752   468 ?        Ss   11:11   0:00 /sbin/portmap
nobody    7639  0.0  0.0   1820   708 ?        Ss   11:11   0:00 /sbin/rpc.statd --no-notify
root      7700  0.0  0.0      0     0 ?        S    11:11   0:00 [lockd]
mysql     7761  1.7  0.1 135376 24272 ?        Ssl  11:11   0:02 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock
root      7838  0.0  0.0   4464   976 ?        Ss   11:11   0:00 /usr/sbin/sshd
root      7873  0.0  0.0   7468  2620 ?        Ss   11:11   0:00 sshd: root@pts/0
root      7900  0.0  0.0  28148 12784 ?        Ss   11:11   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7905  0.0  0.0  28148  9056 ?        S    11:11   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7906  0.0  0.0  28148  9056 ?        S    11:11   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7907  0.0  0.0  28148  9056 ?        S    11:11   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7908  0.0  0.0  28148  9056 ?        S    11:11   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7909  0.0  0.0  28148  9056 ?        S    11:11   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
root      7966  0.0  0.0   1932   372 ?        Ss   11:11   0:00 /usr/sbin/gpm -m /dev/psaux -t imps2
root      8027  0.0  0.0  67976  6312 ?        Ssl  11:11   0:00 /usr/sbin/nsrexecd -s backup01
root      8092  0.0  0.0   3848  1168 ?        Ss   11:11   0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid
nobody    8155  0.0  0.0   1956   548 ?        Ss   11:11   0:00 /usr/sbin/oidentd -u nobody -g nobody
root      8216  0.0  0.0   5424  2004 pts/0    Ss   11:11   0:00 -bash
root      8413  0.0  0.0   3240  1092 ?        Ss   11:11   0:00 //usr/lib/postfix/master
postfix   8425  0.0  0.0   7080  1776 ?        S    11:11   0:00 pickup -l -t fifo -u
postfix   8426  0.0  0.0   7120  1800 ?        S    11:11   0:00 qmgr -l -t fifo -u
nobody    8472  0.0  0.0   9096  1608 ?        Ss   11:11   0:00 proftpd: (accepting connections)
root      8534  0.0  0.0   4300   584 ?        Ss   11:11   0:00 /usr/bin/rsync --daemon
root      8704  0.0  0.0   9944  3748 ?        S    11:11   0:00 /usr/sbin/snmpd -p /var/run/snmpd.pid
root      8828  0.0  0.0   4172   700 ?        Ss   11:11   0:00 /usr/sbin/cron
root      8890  0.0  0.0   1768   604 ?        S    11:11   0:00 /usr/bin/redir --lport=65535 --laddr=82.160.232.132 --cport=3389 --caddr=172.23.1.146
root      8902  0.0  0.0   1780   676 tty1     Ss+  11:11   0:00 /sbin/agetty 38400 tty1 linux
root      8903  0.0  0.0   1780   676 tty2     Ss+  11:11   0:00 /sbin/agetty 38400 tty2 linux
root      8904  0.0  0.0   1780   676 tty3     Ss+  11:11   0:00 /sbin/agetty 38400 tty3 linux
root      8905  0.0  0.0   1780   676 tty4     Ss+  11:11   0:00 /sbin/agetty 38400 tty4 linux
root      8906  0.0  0.0   1780   680 tty5     Ss+  11:11   0:00 /sbin/agetty 38400 tty5 linux
root      8907  0.0  0.0   1780   676 tty6     Ss+  11:11   0:00 /sbin/agetty 38400 tty6 linux
root      9317  0.0  0.0   4564  1024 pts/0    R+   11:13   0:00 ps aux

WIZJA2 ~ # w
 11:15:49 up 5 min,  1 user,  load average: 2,11, 1,20, 0,51

WIZJA2 ~ # iostat -t 1
06.10.2010 11:16:01
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4,86    0,00    1,00    0,00    0,00   94,14

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0,00         0,00         0,00          0          0

Comment 4 Marek Królikowski 2010-10-06 09:19:28 UTC

Load is like i say:

WIZJA2 ~ # w
 11:16:35 up 6 min,  1 user,  load average: 2,18, 1,46, 0,66
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0     11:11    0.00s  0.05s  0.00s w

WIZJA2 ~ # w
 11:18:12 up 7 min,  1 user,  load average: 2,26, 1,60, 0,75
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0     11:11    0.00s  0.05s  0.00s w

All the time 2.00+

Comment 5 Marek Królikowski 2010-10-06 09:19:46 UTC

WIZJA2 ~ # uname -a
Linux WIZJA2 2.6.35-gentoo-r4 #1 SMP PREEMPT Mon Aug 30 07:47:08 CEST 2010 i686 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz GenuineIntel GNU/Linux

Comment 6 Marek Królikowski 2010-10-06 09:33:19 UTC

And this is what i got on 2.6.32:

WIZJA2 ~ # ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.5  0.0   1684   572 ?        Ss   11:23   0:01 init [3]
root         2  0.0  0.0      0     0 ?        S    11:23   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/0]
root         4  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/0]
root         6  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/1]
root         7  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/1]
root         8  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/1]
root         9  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/2]
root        10  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/2]
root        11  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/2]
root        12  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/3]
root        13  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/3]
root        14  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/3]
root        15  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/4]
root        16  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/4]
root        17  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/4]
root        18  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/5]
root        19  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/5]
root        20  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/5]
root        21  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/6]
root        22  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/6]
root        23  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/6]
root        24  0.0  0.0      0     0 ?        S    11:23   0:00 [migration/7]
root        25  0.0  0.0      0     0 ?        S    11:23   0:00 [ksoftirqd/7]
root        26  0.0  0.0      0     0 ?        S    11:23   0:00 [watchdog/7]
root        27  0.0  0.0      0     0 ?        S    11:23   0:00 [events/0]
root        28  0.0  0.0      0     0 ?        S    11:23   0:00 [events/1]
root        29  0.0  0.0      0     0 ?        S    11:23   0:00 [events/2]
root        30  0.0  0.0      0     0 ?        S    11:23   0:00 [events/3]
root        31  0.0  0.0      0     0 ?        S    11:23   0:00 [events/4]
root        32  0.0  0.0      0     0 ?        S    11:23   0:00 [events/5]
root        33  0.0  0.0      0     0 ?        S    11:23   0:00 [events/6]
root        34  0.0  0.0      0     0 ?        S    11:23   0:00 [events/7]
root        35  0.0  0.0      0     0 ?        S    11:23   0:00 [khelper]
root        40  0.0  0.0      0     0 ?        S    11:23   0:00 [async/mgr]
root       160  0.0  0.0      0     0 ?        S    11:23   0:00 [sync_supers]
root       162  0.0  0.0      0     0 ?        S    11:23   0:00 [bdi-default]
root       164  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/0]
root       165  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/1]
root       166  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/2]
root       167  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/3]
root       168  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/4]
root       169  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/5]
root       170  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/6]
root       171  0.0  0.0      0     0 ?        S    11:23   0:00 [kblockd/7]
root       172  0.0  0.0      0     0 ?        S    11:23   0:00 [kacpid]
root       173  0.0  0.0      0     0 ?        S    11:23   0:00 [kacpi_notify]
root       174  0.0  0.0      0     0 ?        S    11:23   0:00 [kacpi_hotplug]
root       298  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/0]
root       299  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/1]
root       300  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/2]
root       301  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/3]
root       302  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/4]
root       303  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/5]
root       304  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/6]
root       305  0.0  0.0      0     0 ?        S    11:23   0:00 [ata/7]
root       306  0.0  0.0      0     0 ?        S    11:23   0:00 [ata_aux]
root       308  0.0  0.0      0     0 ?        S    11:23   0:00 [kseriod]
root       348  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/0]
root       349  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/1]
root       350  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/2]
root       351  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/3]
root       352  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/4]
root       353  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/5]
root       354  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/6]
root       355  0.0  0.0      0     0 ?        S    11:23   0:00 [rpciod/7]
root       418  0.0  0.0      0     0 ?        S    11:23   0:00 [khungtaskd]
root       419  0.0  0.0      0     0 ?        S    11:23   0:00 [kswapd0]
root       420  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/0]
root       421  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/1]
root       422  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/2]
root       423  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/3]
root       424  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/4]
root       425  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/5]
root       426  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/6]
root       427  0.0  0.0      0     0 ?        S    11:23   0:00 [aio/7]
root       428  0.0  0.0      0     0 ?        S    11:23   0:00 [nfsiod]
root       429  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/0]
root       430  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/1]
root       431  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/2]
root       432  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/3]
root       433  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/4]
root       434  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/5]
root       435  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/6]
root       436  0.0  0.0      0     0 ?        S    11:23   0:00 [crypto/7]
root      1095  0.0  0.0      0     0 ?        S    11:23   0:00 [iscsi_eh]
root      1096  0.0  0.0      0     0 ?        S    11:23   0:00 [scsi_eh_0]
root      1110  0.0  0.0      0     0 ?        S    11:23   0:00 [scsi_eh_1]
root      1113  0.0  0.0      0     0 ?        S    11:23   0:00 [scsi_eh_2]
root      1117  0.0  0.0      0     0 ?        S    11:23   0:00 [scsi_eh_3]
root      1121  0.0  0.0      0     0 ?        S    11:23   0:00 [scsi_eh_4]
root      1204  0.0  0.0      0     0 ?        S    11:23   0:00 [kpsmoused]
root      2495  0.0  0.0      0     0 ?        S    11:23   0:00 [cciss_scan]
root      2673  0.0  0.0      0     0 ?        S    11:23   0:00 [ksuspend_usbd]
root      2678  0.0  0.0      0     0 ?        S    11:23   0:00 [khubd]
root      2913  0.0  0.0      0     0 ?        S    11:23   0:00 [usbhid_resumer]
root      3016  0.0  0.0      0     0 ?        S    11:23   0:00 [khpsbpkt]
root      3233  0.0  0.0      0     0 ?        S    11:23   0:00 [kstriped]
root      3505  0.0  0.0      0     0 ?        S    11:23   0:00 [ksnapd]
root      4076  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsIO]
root      4077  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4078  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4079  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4080  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4081  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4082  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4083  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4084  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsCommit]
root      4085  0.0  0.0      0     0 ?        S    11:23   0:00 [jfsSync]
root      4158  0.0  0.0      0     0 ?        S    11:23   0:00 [xfs_mru_cache]
root      4159  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/0]
root      4160  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/1]
root      4161  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/2]
root      4162  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/3]
root      4163  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/4]
root      4164  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/5]
root      4165  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/6]
root      4166  0.0  0.0      0     0 ?        S    11:23   0:00 [xfslogd/7]
root      4167  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/0]
root      4168  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/1]
root      4169  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/2]
root      4170  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/3]
root      4171  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/4]
root      4172  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/5]
root      4173  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/6]
root      4174  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsdatad/7]
root      4175  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/0]
root      4176  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/1]
root      4177  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/2]
root      4178  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/3]
root      4179  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/4]
root      4180  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/5]
root      4181  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/6]
root      4182  0.0  0.0      0     0 ?        S    11:23   0:00 [xfsconvertd/7]
root      4347  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/0]
root      4348  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/1]
root      4349  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/2]
root      4350  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/3]
root      4351  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/4]
root      4352  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/5]
root      4353  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/6]
root      4354  0.0  0.0      0     0 ?        S    11:23   0:00 [reiserfs/7]
root      4452  0.0  0.0   2344  1092 ?        S<s  11:23   0:00 /sbin/udevd --daemon
root      5490  0.0  0.0      0     0 ?        S    11:23   0:00 [flush-8:0]
root      6364  0.0  0.0   1732   644 ?        Ss   11:23   0:00 /usr/sbin/syslogd -m 0 -r -i 0.0.0.0
root      6374  0.0  0.0   1684   428 ?        Ss   11:23   0:00 /usr/sbin/klogd -c 3 -2
named     7478  0.0  0.0  15700 12936 ?        Ss   11:23   0:00 /usr/sbin/named -u named -n 8 -t /chroot
bin       7592  0.0  0.0   1704   464 ?        Ss   11:23   0:00 /sbin/portmap
nobody    7655  0.0  0.0   1768   720 ?        Ss   11:23   0:00 /sbin/rpc.statd --no-notify
root      7716  0.0  0.0      0     0 ?        S    11:23   0:00 [lockd]
mysql     7777  1.9  0.1 137840 25704 ?        Ssl  11:23   0:03 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock
root      7854  0.0  0.0   4412  1004 ?        Ss   11:23   0:00 /usr/sbin/sshd
root      7914  0.0  0.0  28100 12888 ?        Ss   11:23   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7918  0.0  0.0  28100  9112 ?        S    11:23   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7919  0.0  0.0  28100  9112 ?        S    11:23   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7920  0.0  0.0  28100  9112 ?        S    11:23   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7921  0.0  0.0  28100  9112 ?        S    11:23   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    7922  0.0  0.0  28100  9112 ?        S    11:23   0:00 /usr/sbin/apache2 -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D PERL -D LDAP -D AUTHNZ_LDAP -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
root      7979  0.0  0.0   1880   472 ?        Ss   11:23   0:00 /usr/sbin/gpm -m /dev/psaux -t imps2
root      8040  0.0  0.0  67972  6696 ?        Ssl  11:23   0:00 /usr/sbin/nsrexecd -s backup01
root      8105  0.0  0.0   3800  1176 ?        Ss   11:23   0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid
nobody    8168  0.0  0.0   1908   544 ?        Ss   11:23   0:00 /usr/sbin/oidentd -u nobody -g nobody
root      8418  0.0  0.0   3192  1136 ?        Ss   11:23   0:00 //usr/lib/postfix/master
postfix   8435  0.0  0.0   7032  1816 ?        S    11:23   0:00 pickup -l -t fifo -u
postfix   8436  0.0  0.0   7072  1880 ?        S    11:23   0:00 qmgr -l -t fifo -u
nobody    8477  0.0  0.0   9092  1660 ?        Ss   11:23   0:00 proftpd: (accepting connections)
root      8538  0.0  0.0   4252   608 ?        Ss   11:23   0:00 /usr/bin/rsync --daemon
root      8689  0.0  0.0  10068  3960 ?        S    11:24   0:00 /usr/sbin/snmpd -p /var/run/snmpd.pid
root      8813  0.0  0.0   4124   724 ?        Ss   11:24   0:00 /usr/sbin/cron
root      8875  0.0  0.0   1716   604 ?        S    11:24   0:00 /usr/bin/redir --lport=65535 --laddr=82.160.232.132 --cport=3389 --caddr=172.23.1.146
root      8887  0.0  0.0   1732   672 tty1     Ss+  11:24   0:00 /sbin/agetty 38400 tty1 linux
root      8888  0.0  0.0   1732   672 tty2     Ss+  11:24   0:00 /sbin/agetty 38400 tty2 linux
root      8889  0.0  0.0   1732   672 tty3     Ss+  11:24   0:00 /sbin/agetty 38400 tty3 linux
root      8890  0.0  0.0   1732   672 tty4     Ss+  11:24   0:00 /sbin/agetty 38400 tty4 linux
root      8891  0.0  0.0   1732   672 tty5     Ss+  11:24   0:00 /sbin/agetty 38400 tty5 linux
root      8892  0.0  0.0   1732   672 tty6     Ss+  11:24   0:00 /sbin/agetty 38400 tty6 linux
root      8895  0.0  0.0   7416  2640 ?        Ss   11:24   0:00 sshd: root@pts/0
root      8898  0.0  0.0   5376  2044 pts/0    Ss   11:24   0:00 -bash
root      9079  0.0  0.0   4480   980 ?        S    11:25   0:00 /usr/sbin/cron
root      9585  0.0  0.0   4516  1052 pts/0    R+   11:26   0:00 ps aux



WIZJA2 ~ # w
 11:31:52 up 8 min,  1 user,  load average: 0,02, 0,06, 0,08
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0     11:24    0.00s  0.02s  0.00s w

WIZJA2 ~ # iostat -t 1
06.10.2010 11:32:35
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,12    0,00    0,00   99,88

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4,00        16,00        88,00         16         88

Comment 7 Wormo (RETIRED) gentoo-dev

2010-11-05 01:56:38 UTC

The only recent mainline kernel bugs I could find about load average were on a regression between 2.6.34 and 2.6.35, which is obviously not the case here...

Please post output of 'emerge --info' and attach your kernel .config for good measure -- any clues about what might be unusual about I/O on this system would be good (e.g. using jfs/xfs/reiserfs instead of just ext[34]? exporting NFS partitions? using encrypted partitions?)

Comment 8 Panagiotis Christopoulos (RETIRED) gentoo-dev

2010-11-20 14:58:31 UTC

Any updates here? I resolve this bug as needinfo. Please reopen when you have submitted the information wormo wrote about.

Comment 9 Marek Królikowski 2011-01-03 08:04:34 UTC

Created attachment 258714 [details]
Kernel config file

Comment 10 Marek Królikowski 2011-01-03 08:10:53 UTC

WIZJA2 ~ # emerge --info
Portage 2.1.9.25 (default/linux/x86/10.0, gcc-4.4.4, glibc-2.11.2-r3, 2.6.36-hardened-r6 i686)
=================================================================
System uname: Linux-2.6.36-hardened-r6-i686-Intel-R-_Xeon-R-_CPU_X5570_@_2.93GHz-with-gentoo-1.12.14
Timestamp of tree: Mon, 03 Jan 2011 07:15:02 +0000
app-shells/bash:     4.1_p7
dev-lang/python:     2.6.6-r1, 3.1.2-r4
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 1.12.14-r1
sys-apps/sandbox:    2.4
sys-devel/autoconf:  2.65-r1
sys-devel/automake:  1.9.6-r2, 1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.3.4, 4.4.4-r2
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.10
sys-devel/make:      3.81-r2
virtual/os-headers:  2.6.30-r1 (sys-kernel/linux-headers)
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="*"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-Os -march=prescott -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-Os -march=prescott -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
GENTOO_MIRRORS="http://mirror.switch.ch/ftp/mirror/gentoo/ http://gentoo.po.opole.pl/ http://gentoo.zie.pg.gda.pl/ http://gentoo.prz.rzeszow.pl/"
LANG="pl_PL.utf8"
LC_ALL="pl_PL.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="pl"
MAKEOPTS="-j9"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="acl apache2 autofs bash-completion berkdb bidi bzip2 cli cracklib crypt ctype curl cxx dedicated devfs dri fontconfig foomaticdb fortran gd gdbm gif gpm hash http iconv imap ipv6 jadetex jpeg ldap ldap-sasl md5sum modules mudflap mysql ncurses nls no-htdocs nptl nptlonly openmp pam pcre pdo perl perlsuid php png pppd priority python readline ruby samba sasl session simplexml slang smtp snmp sockets sse ssl svg symlink sysfs tcpd truetype unicode vda wddx x86 xml xml2 xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="pl" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Partition:
Local:
/dev/sda1 on /boot type reiserfs (rw,noatime)
/dev/sda2 on / type reiserfs (rw,noatime)
NFS:
xxx.xxx.xxx.xxx:/backup2/wizja2 on /mnt/nfs type nfs (rw,addr=xxx.xxx.xxx.xxx,nfsvers=3,proto=udp,mountproto=udp,mountport=1048)

But i umount nfs partition and nothing help.

The most interesting i find what take so much load average:
games-simulation/openttd-1.0.5
When i run openttd 4 instances of openttd my load come to 4.00 - looks like every instance of openttd take 1.00 load.
But when i back to 2.6.32-hardened-r22 there is no problem load is 0.00 max 0.05
Mayby there is a problem with Openttd on new kernels??

Thanks

Comment 11 Marek Królikowski 2011-01-13 13:45:36 UTC

Created attachment 259700 [details]
jffnms - Before and after change kernel

Comment 12 Marek Królikowski 2011-01-13 13:46:19 UTC

Hello i upload from jffnms a graph how look load average before and after upgrade kernel.
Thanks

Comment 13 Marek Królikowski 2011-01-13 14:04:52 UTC

I now compile 2.6.32-hardened-r31 - newlatest stable 2.6.32 version.
Tommorow i show graph from jffnms and we will see what happened.
Thanks

Comment 14 George Kadianakis (RETIRED) gentoo-dev

2011-01-13 18:59:50 UTC

Anything interesting in dmesg, by the way?

Comment 15 Marek Królikowski 2011-01-14 06:18:10 UTC

Hello
Yesterday like i say i compile new kernel 2.6.32-hardened-r31
After whole night load average is less than 0.10:
WIZJA2 ~ # uname -a
Linux WIZJA2 2.6.32-hardened-r31 #1 SMP PREEMPT Thu Jan 13 15:05:01 CET 2011 i686 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz GenuineIntel GNU/Linux
In attachment i post next graph from jffnms from whole night.

Comment 16 Marek Królikowski 2011-01-14 06:18:48 UTC

Created attachment 259765 [details]
JFFNMS graph of Load Average on 2.6.32-hard-r33

Comment 17 Marek Królikowski 2011-01-14 06:28:58 UTC

(In reply to comment #14)
> Anything interesting in dmesg, by the way?
> 

I check whole dmesg and find only:
Marking TSC unstable due to check_tsc_sync_source failed
svc: failed to register lockdv1 RPC service (errno 97).

nothing else report problems/errrors.

Comment 18 Marek Królikowski 2011-01-27 08:06:17 UTC

So any idea what is wrong?
This is IBM xSeries 3650 M2 / mt-m 7947-92G

WIZJA2 ~ # lspci
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:10.0 PIC: Intel Corporation 5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13)
00:10.1 PIC: Intel Corporation 5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13)
00:11.0 PIC: Intel Corporation 5520/5500 Physical and Link Layer Registers Port 1 (rev 13)
00:11.1 PIC: Intel Corporation 5520/5500 Routing & Protocol Layer Register Port 1 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev 13)
00:15.0 PIC: Intel Corporation 5520/5500/X58 Trusted Execution Technology Registers (rev 13)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 1
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 5
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller
01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)
06:00.0 PCI bridge: Vitesse Semiconductor VSC452 [SuperBMC] (rev 01)
07:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200EV
0b:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
0b:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)


THanks

Comment 19 Mike Pagano gentoo-dev

2011-02-01 15:15:22 UTC

Can you test with a 2.6.37 kernel, please.

Comment 20 Marek Królikowski 2011-02-01 16:53:49 UTC

Hello
This same effect:
WIZJA2 ~ # uname -a
Linux WIZJA2 2.6.37-hardened-r1 #1 SMP PREEMPT Tue Feb 1 16:51:03 CET 2011 i686 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz GenuineIntel GNU/Linux
WIZJA2 ~ # w
 17:53:17 up 14 min,  2 users,  load average: 3,68, 3,57, 2,16
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/2     17:43    0.00s 13.58s  0.00s w

Comment 21 Marek Królikowski 2011-02-02 06:05:00 UTC

Created attachment 261309 [details]
.config from 2.6.37

Comment 22 Marek Królikowski 2011-02-02 06:07:39 UTC

Created attachment 261310 [details]
Graph from jffnms - load average

After 13 hours working:
WIZJA2 ~ # uname -a
Linux WIZJA2 2.6.37-hardened-r1 #1 SMP PREEMPT Tue Feb 1 16:51:03 CET 2011 i686 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz GenuineIntel GNU/Linux
WIZJA2 ~ # w
 07:05:31 up 13:26,  1 user,  load average: 3,39, 3,69, 3,70
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0     07:02    0.00s  0.02s  0.00s w

Comment 23 Mike Pagano gentoo-dev

2011-03-11 16:12:02 UTC

Does this happen with later kernels? I will not be releasing any new .34 kernel.

Comment 24 Marek Królikowski 2011-03-11 16:32:25 UTC

Hello
All kernel after 2.6.32 got this problem.
I check:
2.6.34-r6
2.6.35-gentoo-r4
2.6.36-hardened-r6
2.6.37-hardened-r1
and this same effect - strange load average on idle.

The most funny i check:
2.6.32-hardened-r22
2.6.32-hardened-r31
2.6.32-hardened-r34
and there is no problem that`s why i do
cat /etc/portage/package.mask
>sys-kernel/hardened-sources-2.6.33

and all working good..

Thanks

Comment 25 Mike Pagano gentoo-dev

2011-03-21 18:49:24 UTC

Pleaswe report upstream at http://bugzilla.kernel.org and post the url back here

Comment 26 Paul Tobias 2011-03-28 13:55:01 UTC

Can you please test if this command fixes the problem:
echo 500 > /proc/sys/vm/dirty_writeback_centisecs
Maybe it's the same bug as http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=594923

Comment 27 Marek Królikowski 2011-03-29 05:13:40 UTC

i check on 2.6.37-hardened-r1 :
WIZJA2 ~ # cat /proc/sys/vm/dirty_writeback_centisecs
500
WIZJA2 ~ # uname -a
Linux WIZJA2 2.6.37-hardened-r1 #1 SMP PREEMPT Tue Feb 1 16:51:03 CET 2011 i686 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz GenuineIntel GNU/Linux
WIZJA2 ~ # w
 07:05:21 up 9 min,  1 user,  load average: 3,61, 3,12, 1,61
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0     06:57    0.00s  9.76s  0.00s w

Not working still big load :(

After back to 2.6.32 working good:
WIZJA2 ~ # cat /proc/sys/vm/dirty_writeback_centisecs
500
WIZJA2 ~ # uname -a
Linux WIZJA2 2.6.32-hardened-r34 #1 SMP PREEMPT Sun Feb 6 00:33:04 CET 2011 i686 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz GenuineIntel GNU/Linux
WIZJA2 ~ # w
 07:12:22 up 2 min,  1 user,  load average: 0,19, 0,14, 0,06
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0     07:12    0.00s  0.03s  0.00s w

Comment 28 Mike Pagano gentoo-dev

2011-04-12 23:27:59 UTC

Please submit upstream as requested and post the url back here.

Comment 29 Marek Królikowski 2011-04-13 05:03:53 UTC

(In reply to comment #28)
> Please submit upstream as requested and post the url back here.

Hmmm who change that bug as resolved?
Change /proc/sys/vm/dirty_writeback_centisecs to 500 give nothing this same problem with huge load....

Comment 30 Marek Królikowski 2011-04-13 05:27:54 UTC

(In reply to comment #29)
> (In reply to comment #28)
> > Please submit upstream as requested and post the url back here.
> 
> Hmmm who change that bug as resolved?
> Change /proc/sys/vm/dirty_writeback_centisecs to 500 give nothing this same
> problem with huge load....

OK i open bug on kernel.org webpage 
https://bugzilla.kernel.org/show_bug.cgi?id=33192

Thanks for help - i hope someone find what is wrong..