I'm running sys-kernel/2.6.30-gentoo. I have a quad core (Intel Q9450) and four CPU intensive processes running. Three of them are nice +5, while one is +0. The +0 has a CPU to itself, while the three +5's are sharing only two CPUs. This leaves one CPU 100% idle. Something's seriously wrong here. Reproducible: Always Steps to Reproduce: 1. Run any four CPU intensive processes (say, an infinite loop) on a quad core 2. Renice three of them to +5. 3. Run top and see how the load is distributed Actual Results: One CPU is idle Expected Results: Regardless of nice level, none should be idle. Portage 2.1.6.13 (default/linux/amd64/2008.0, gcc-4.3.3, glibc-2.10.1-r0, 2.6.30-gentoo x86_64) ================================================================= System uname: Linux-2.6.30-gentoo-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q9450_@_2.66GHz-with-gentoo-2.0.1 Timestamp of tree: Fri, 24 Jul 2009 02:30:01 +0000 app-shells/bash: 4.0_p24 dev-java/java-config: 2.1.8-r1 dev-lang/python: 2.6.2-r1 dev-util/cmake: 2.6.4-r1 sys-apps/baselayout: 2.0.1 sys-apps/openrc: 0.4.3-r3 sys-apps/sandbox: 2.0 sys-devel/autoconf: 2.13, 2.63-r1 sys-devel/automake: 1.5, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.2, 1.11 sys-devel/binutils: 2.19.1-r1 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.6a virtual/os-headers: 2.6.30 ACCEPT_KEYWORDS="amd64 ~amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=core2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-O2 -march=core2 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--jobs=4" FEATURES="distlocks fixpackages parallel-fetch protect-owned sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://gentoo.netnitco.net http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ ftp://mirror.datapipe.net/gentoo ftp://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://gentoo.mirrors.easynews.com/linux/gentoo/ ftp://ftp.free.fr/mirrors/ftp.gentoo.org/ ftp://gentoo.imj.fr/pub/gentoo/ ftp://distro.ibiblio.org/pub/linux/distributions/gentoo/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1" LINGUAS="en en_US" MAKEOPTS="--jobs=5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi alsa amd64 apache2 apm bash-completion berkdb bidi bzip2 cairo cdda cdio cdr cli cracklib crypt ctype cups dbus dri dts dvd dvdr encode fbcon ffmpeg filter flac fortran freetype gcj gd gdbm gnutls gpm hal httpd iconv ipv6 isdnlog jadetex java jpeg jpeg2k kde kde4 kerberos lapack ldap live lm_sensors mad matroska midi mjpeg mmx mng mp3 mpeg mudflap multilib mysql mysqli ncurses nls nptl nptlonly ogg oggvorbis opengl openmp openssl pam pcre perl php png pppd python qt3support qt4 quicktime readline reflection ruby samba session spl sse sse2 sse3 ssl stream svg sysfs tcpd theora threads tiff tk truetype unicode utempter vcd vlm vorbis webkit wxwindows x264 xorg xv xvid zeroconf zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US" USERLAND="GNU" VIDEO_CARDS="radeon radeonhd" Unset: CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
The 2.6.30 release did include changes to the CFQ process scheduler. Have you tried this test with older kernels, to see if this is a regression in the scheduler?
Please attach your kernel's .config file. Did you do the tests Wormo asked for?
Sorry I haven't gotten around to running the test. I've been working towards deadlines and running simulations that take days to run. I'll attach my config. Also, I spoke with someone on LKML and got a plausible explanation. I have four cores, arranged on two dies, so pairs shade L2 cache. It appears the scheduler is trying to be "smart" by giving the nice=0 process an L2 cache all to itself, while making the lower priority processes share. But that is just conjecture. The developers ignored the discussion, as usual. :)
Created attachment 199945 [details] My kernel config
(In reply to comment #3) > Sorry I haven't gotten around to running the test. I've been working towards > deadlines and running simulations that take days to run. I'll attach my > config. > > Also, I spoke with someone on LKML and got a plausible explanation. I have > four cores, arranged on two dies, so pairs shade L2 cache. It appears the > scheduler is trying to be "smart" by giving the nice=0 process an L2 cache all > to itself, while making the lower priority processes share. Sounds very plausible to me. Having processes jump around from core to core is very costly. Please reopen this bug report when more information has been added (even if it's only the [not a, the] matching bug on http://bugzilla.kernel.org/).
Something similar here Here is a dual quad core intel i7 system. Kernel is also 2.6.30 (-gentoo-r5) If a java application runs with nice -n 19 and 8 threads every thread is happily running on each core on all cpus. But if another long running process with just one thread is started with no nice level suddenly the java application begins to run only on 4 cores on one cpu and the other process on one core on the other cpu but three cores don't do nothing. This is not happening if the other process is also started with nice level 19, then all 8 cores are used. Short summary Dual quad core system Process A with 8 threads and nice level 19 plus Process B with 1 thread and nice level 0 then 4 cores on cpu 1 running process A 1 core on cpu 2 running process B 3 cores on cpu 2 are idle
