It looks like squid isn't caching downloads that are a lot larger than maximum_object_size_in_memory even if maximum_object_size is set to be very large. For example with the default maximum_object_size_in_memory value but maximum_object_size set to 32 MB, downloading http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/stage3-amd64-20140821.tar.bz2.CONTENTS always results in TCP_MISS. After setting maximum_object_size_in_memory to 5 MB (Or however big the file is + 1 MB), I get TCP_MEM_HIT and then after commenting out maximum_object_size_in_memory I get TCP_HIT. 3.3.12 is better than 3.3.8 though as with 3.3.8 it didn't even cache downloads of index.html (Tested on multiple servers with similar CFLAGS and hardened kernel). NOTE: This might be because of my CFLAGS or something on hardened kernels, so I'm posting here to start with instead of the Squid forums. Reproducible: Always Steps to Reproduce: 1. export http_proxy="http://localhost:3128" 2. wget -N -c "10 MB file" 3. rm "10 MB file" 4. tail /var/log/squid/access.log 5. Repeat a few times to see if squid is caching the download Actual Results: /var/log/squid/access.log has TCP_MISS instead of TCP_MEM_HIT or TCP_HIT squid.conf: acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localnet http_access allow localhost http_access deny all icp_access allow all http_port 3128 http_port 3130 tproxy http_port 3131 intercept memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs /var/cache/squid 2048 16 256 maximum_object_size 32 MB coredump_dir /var/cache/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 reload-into-ims refresh-ims via off shutdown_lifetime 2 seconds forwarded_for delete emerge --info: Portage 2.2.8-r1 (hardened/linux/amd64, gcc-4.6.3, unavailable, 3.14.15-hardened x86_64) ================================================================= System uname: Linux-3.14.15-hardened-x86_64-AMD_FX-tm-4100_Quad-Core_Processor-with-gentoo-2.1 KiB Mem: 8118992 total, 1826448 free KiB Swap: 6289400 total, 6265628 free Timestamp of tree: Thu, 21 Aug 2014 13:15:01 +0000 ld GNU ld (GNU Binutils) 2.22 app-shells/bash: 4.2_p37 dev-java/java-config: 1.3.7-r1, 2.1.10 dev-lang/python: 2.4.4-r13, 2.5.2-r7, 2.6.8, 2.7.5-r3, 3.2.5-r3, 3.3.2-r2 dev-util/cmake: 2.8.6-r4 dev-util/pkgconfig: 0.27.1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.9.8.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13::<unknown repository>, 2.69 sys-devel/automake: 1.4_p6::<unknown repository>, 1.5::<unknown repository>, 1.6.3::<unknown repository>, 1.7.9-r1::<unknown repository>, 1.8.5-r3::<unknown repository>, 1.9.6-r2::<unknown repository>, 1.10.1, 1.11.6, 1.12.6, 1.13.4 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 3.4.6-r2::<unknown repository>, 4.1.2::<unknown repository>, 4.3.6-r1, 4.5.3-r2, 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.7 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo x-portage ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -mtune=bdver1 -fomit-frame-pointer -ftree-vectorize -fpredictive-commoning -fno-tree-vect-loop-version" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -mtune=bdver1 -fomit-frame-pointer -ftree-vectorize -fpredictive-commoning -fno-tree-vect-loop-version" DISTDIR="/usr/portage.local/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://mirror.internode.on.net/pub/gentoo http://distfiles.gentoo.org http://www..ibiblio.org/pub/Linux/distributions/gentoo" LANG="en_AU.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j1" PKGDIR="/usr/portage.local/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://mirror.internode.on.net/gentoo-portage" USE="3dnow 3dnowext acl acpi alsa amd64 apache2 berkdb bzip2 caps cjk cli cracklib crypt cups cxx dlloader dri fam gdbm hardened iconv ipv6 jpeg justify kerberos logrotate mmx mmxext mng modules multilib ncurses nls nptl openmp pam pax_kernel pcre png qt readline session sse sse2 ssl tcpd threads tiff unicode urandom vhosts xattr xinerama xtpax zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias cgid" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-4" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
On a non-hardened test system with empty squid cache, I can't seem to cache any files larger than 4194304 with squid-3.3.12 no matter how high I set maximum_object_size or maximum_object_size_in_memory to. Testing was done by downloading from a local Apache server with a file created with dd. Test squid.conf slightly modified from default: acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localnet http_access allow localhost http_access deny all http_port 3128 cache_dir aufs /var/cache/squid 2048 16 256 coredump_dir /var/cache/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 reload-into-ims refresh-ims maximum_object_size_in_memory 600 MB maximum_object_size 600 MB via off forwarded_for delete emerge --info: Portage 2.2.8-r1 (default/linux/amd64/13.0/desktop/kde, gcc-4.6.3, glibc-2.15-r3, 3.15.6-gentoo x86_64) ================================================================= System uname: Linux-3.15.6-gentoo-x86_64-Intel-R-_Core-TM-_i7-4700HQ_CPU_@_2.40GHz-with-gentoo-2.1 KiB Mem: 16317388 total, 712736 free KiB Swap: 16777212 total, 16380240 free Timestamp of tree: Sun, 24 Aug 2014 13:15:01 +0000 ld GNU ld (GNU Binutils) 2.22 app-shells/bash: 4.2_p37 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.6.8, 2.7.5-r2, 3.1.2-r3, 3.2.5-r2, 3.3.2-r2 dev-util/cmake: 2.8.11.2 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1, 1.12.6, 1.13.4 sys-devel/binutils: 2.22-r1 sys-devel/gcc: 4.4.5, 4.5.3-r1, 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo bumblebee x-portage ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=corei7-avx -fomit-frame-pointer -mmmx -msse3 -msse4.1 -ftree-vectorize -fpredictive-commoning -fno-tree-vect-loop-version -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/maven-bin-2.2/conf /usr/share/maven-bin-3.0/conf /usr/share/themes/oxygen-gtk/gtk-2.0 /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=corei7-avx -fomit-frame-pointer -mmmx -msse3 -msse4.1 -ftree-vectorize -fpredictive-commoning -fno-tree-vect-loop-version -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://mirror.internode.on.net/pub/gentoo http://mirror.3fl.net.au/pub/gentoo ftp://gentoo.mirrors.pair.com/ ftp://mirrors.tds.net/gentoo http://mirrors.tds.net/gentoo" LANG="en_AU.utf8" LDFLAGS="-Wl,--as-needed" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/bumblebee /usr/local/portage" SYNC="rsync://rsync/gentoo-portage" USE="X a52 aac acl acpi alsa amd64 apache2 arts avahi berkdb bluetooth branding bzip2 cairo caps cdda cdr cjk cli consolekit cracklib crypt cups cxx dbus declarative divx4linux dlloader dri dts dv dvd dvdr emboss encode exif faac fam firefox flac fortran gdbm gif gnome gnutls gpm gstreamer gtk guile hal howl htmlhandbook iconv ieee1394 ipv6 jikes joystick jpeg jpeg2k kde kdeenablefinal kdehiddenvisibility kipi lcms ldap libnotify logrotate mad matroska mmx mng modules mp3 mp4 mpeg multilib nas ncurses net nls nptl odbc ogg openexr opengl openmp pam pango pcre pdf phonon physfs plasma png policykit povray ppds pulseaudio qt3support qt4 readline samba scanner sdl semantic-desktop session slp speex spell sse sse2 sse3 ssl ssse3 startup-notification svg tcpd tetex theora threads tiff truetype udev udisks unicode upower usb vorbis wmf wxwidgets x264 xattr xcb xcomposite xinerama xml xml2 xscreensaver xv xvid zeroconf zlib" ABI_X86="32 64" ALSA_CARDS="intel8x0 intel8x0m via82xx usb-audio hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" DVB_CARDS="usb-a800" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev keyboard mouse synaptics linuxinput ps2mouse joystick wacom" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_GB en_US en_AU zh_CN" LIRC_DEVICES="devinput mceusb mceusb2 atilibusb atiusb" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" QEMU_SOFTMMU_TARGETS="i386 x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev nv v4l vesa nvidia nouveau intel radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
I don't have the 4MB cache problem on the server that runs hardened (The first config posted) though even with a fresh cache folder. On that system if I set maximum_object_size_in_memory 500 MB, I can download a 200MB file and it is stored to the cache.
I believe you need to set maximum_object_size before your cache_dir line, can you see if that helps?
Okay that's fixed the problem thanks. This means that squid.conf.documented is incorrect as it has maximum_object_size below cache_dir and even though 3.1.8 doesn't have a problem with that, 3.2 series and 3.3 series do have a problem. What I didn't mention is the first config posted is based on squid.conf.documented and then I removed all the comments with "grep -v '^#' /etc/squid/squid.conf | grep '[a-zA-Z0-9]'" . Would this be classed as a bug in squid.conf.documented or in the Squid source code?
please retry with net-proxy/squid-3.5.28
