There is a problem with squid-2.5.12 and maybe prior versions. In the following example a snip of the wget log of a cache hit, with wget proxy set on 127.0.0.1 shows clearly there are some problems when fetching objects from cache. The problem is present with or without the zero-penalty-hit flag. --15:55:49-- http://www.kernel.org/pub/linux/kernel/v2.6/patch-2.6.15.gz => `patch-2.6.15.gz.7' Connecting to 127.0.0.1:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 7,648,543 (7.3M) [application/x-gzip] 0K .......... .......... .......... .......... .......... 0% 262K 0:28 50K .......... .......... .......... .......... .......... 1% 260K 0:28 100K .......... .......... .......... .......... .......... 2% 272K 0:26 ----------------------------------- snip ------------------------------------ 1450K .......... .......... .......... .......... .......... 20% 258K 0:23 1500K .......... .......... .......... .......... .......... 20% 263K 0:22 1550K .......... .......... .......... .......... .......... 21% 240K 0:24 1600K .......... .......... .......... .......... .......... 22% 1.06M 0:05 1650K .......... .......... .......... .......... .......... 22% 9.77M 1700K .......... .......... .......... .......... .......... 23% 12.2M 1750K .......... .......... .......... .......... .......... 24% 9.77M 1800K .......... .......... .......... .......... .......... 24% 12.2M 1850K .......... .......... .......... .......... .......... 25% 9.77M ----------------------------------- snip ------------------------------------ 7400K .......... .......... .......... .......... .......... 99% 12.2M 7450K .......... ......... 100% 9.41M Here is my squid.conf: http_port 3128 hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY cache_mem 400 MB maximum_object_size 16 MB maximum_object_size_in_memory 100 KB ipcache_size 10240 ipcache_low 90 ipcache_high 95 fqdncache_size 1024 cache_replacement_policy heap LFUDA memory_replacement_policy heap LRU cache_dir diskd /var/cache/squid/ 1000 16 256 Q1=64 Q2=72 emulate_httpd_log on ftp_passive on diskd_program /usr/lib/squid/diskd auth_param basic children 5 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours auth_param basic casesensitive off refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 20% 4320 acl snmpcommunity snmp_community public acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 563 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 563 # https, snews acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT acl localnet src 192.168.1.0/24 http_access allow localnet http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost http_access deny all http_reply_access allow all icp_access allow all httpd_accel_host virtual httpd_accel_port 80 httpd_accel_with_proxy on httpd_accel_uses_host_header on forwarded_for off snmp_port 3401 snmp_access allow snmpcommunity localhost snmp_access deny all coredump_dir /var/cache/squid pipeline_prefetch on The machine is a P4 2Ghz, 1GB Memory.
[emerge info] Portage 2.0.53 (default-linux/x86/2005.0, gcc-3.3.6, glibc-2.3.5-r2, 2.6.14-gentoo-r5 i686) ================================================================= System uname: 2.6.14-gentoo-r5 i686 Intel(R) Pentium(R) 4 CPU 2.00GHz Gentoo Base System version 1.6.14 dev-lang/python: 2.3.5-r2, 2.4.2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=pentium4 -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=pentium4 -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://ftp.roedu.net/pub/mirrors/gentoo.org/ http://ftp.lug.ro/gentoo/" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X acl alsa apm arts avi berkdb bitmap-fonts bzip2 crypt cups curl eds emboss encode expat foomaticdb fortran gd gdbm gif gpm gstreamer idn imlib ipv6 jpeg lcms libg++ libwww mad mhash mikmod mng motif mp3 mpeg mysql ncurses nls ogg oggvorbis opengl oss pam pcre pdflib perl png postgres python qt quicktime readline samba sdl snmp spell ssl tcpd tiff truetype truetype-fonts type1-fonts udev vorbis xml xml2 xv zero-penalty-hit zlib userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS, MAKEOPTS, PORTDIR_OVERLAY
bug confirmed. on my machine the speed is about 250kB/s for objects served from cache. I think this limit is due to timer frequency selected in kernel (250 HZ on my host). Please compile your kernel with CONFIG_1000_HZ and repeat the test. If I'm right, the speed should be now about 1MB/s.
(In reply to comment #2) > bug confirmed. > on my machine the speed is about 250kB/s for objects served from cache. > I think this limit is due to timer frequency selected in kernel (250 HZ on my > host). > Please compile your kernel with CONFIG_1000_HZ and repeat the test. If I'm > right, the speed should be now about 1MB/s. Kernel compilation is under way. Meanwhile i would like to point that on another system the speed goes beyond 5-6 MBytes/s, even if the systems resources are lower than here. The system is not running gentoo though :\
(In reply to comment #3) > (In reply to comment #2) > > bug confirmed. > > on my machine the speed is about 250kB/s for objects served from cache. > > I think this limit is due to timer frequency selected in kernel (250 HZ on my > > host). > > Please compile your kernel with CONFIG_1000_HZ and repeat the test. If I'm > > right, the speed should be now about 1MB/s. > > Kernel compilation is under way. Meanwhile i would like to point that on > another system the speed goes beyond 5-6 MBytes/s, even if the systems > resources are lower than here. The system is not running gentoo though :\ > Kernel compiled and booted. Cleaned squid cache. Same effect, the speed varies from 300 to 400 Kbytes/s. After 1 to 5 MBytes downloaded, the speed goes to straight to 9-12 Mbytes/s, as in the example. Any other ideas ?!
2 IDE drives are in RAID1. hdparm -tT /dev/md2 /dev/md2: Timing cached reads: 872 MB in 2.01 seconds = 434.33 MB/sec Timing buffered disk reads: 128 MB in 3.04 seconds = 42.07 MB/sec the timings are good. What could be the problem ?
1) what version of squid is installed on the other system? 2) try to disable related configure options (e.g. --enable-async-io)
(In reply to comment #6) > 1) what version of squid is installed on the other system? > 2) try to disable related configure options (e.g. --enable-async-io) > I did a few tests. System 1: In the bug report. Compiled squid 2.5.10 from tarball. Same thing. System 2: Non-gentoo. Runing squid 2.5.10 5-6 Mbytes / s. System 3: Running very good is a gentoo this time. [emerge info] Portage 2.0.53 (default-linux/x86/2005.1, gcc-3.3.6, glibc-2.3.5-r2, 2.6.14-gentoo-r5 i686) ================================================================= System uname: 2.6.14-gentoo-r5 i686 AMD Athlon(tm) XP 2000+ Gentoo Base System version 1.6.14 dev-lang/python: 2.3.5-r2, 2.4.2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-Os -march=athlon-xp -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-Os -march=athlon-xp -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X alsa apm arts avi berkdb bitmap-fonts bzip2 crypt cups eds emboss encode expat foomaticdb fortran gdbm gif gnome gpm gstreamer gtk gtk2 imlib ipv6 jpeg kde libg++ libwww mad mhash mikmod motif mp3 mpeg ncurses nls ogg oggvorbis opengl oss pam pdflib perl png python qt quicktime readline sdl spell ssl tcpd truetype truetype-fonts type1-fonts udev vorbis xml2 xmms xv zero-penalty-hit zlib userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS, MAKEOPTS, PORTDIR_OVERLAY
Hello I'm back. After a evening spent on recompiling various kernel and squid options, googleing around i found http://www.mail-archive.com/squid-users@squid-cache.org/msg34923.html. The problem is diskd. As explained in the thread diskd is not good when dealing with little cache activity. So, the only thing to do in this case is to add a line in squid.conf to suggesting not to use diskd unless the cache server handles a large number of clients, otherwise the cache throughput is limited by design.
I experimented a bit with different cache_dir types and it seems aufs is better for my needs (large cache dir, few hits) with about 20-35 MB/s. ufs is also pretty close at 10-20MB/s. Do you have a warning message to propose?
# The diskd store type: # # "diskd" uses the same storage format as "ufs", utilizing a # separate process to avoid blocking the main Squid process on # disk-I/O. # # WARNING: "diskd" should be used only if the traffic trough squid # is sufficient to keep squid resonably busy, otherwise # the cache throughput will be low when serving big objects.
fixed in 2.5.12-r1. in addition, this version contains all current upstream patches.
