Squashfs image of the current portage tree (tested last week and today) created with squashfs-tools-4.0 reveals IO error. The same not happen with versione 4.2. --- kernel info ---------------------------------------------------------- Linux baltasar 2.6.39-gentoo-r3 #4 SMP PREEMPT Tue Dec 13 03:32:41 CET 2011 x86_64 Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz GenuineIntel GNU/Linux CONFIG_SQUASHFS=m CONFIG_SQUASHFS_XATTR=y CONFIG_SQUASHFS_LZO=y CONFIG_SQUASHFS_XZ=y # CONFIG_SQUASHFS_EMBEDDED is not set CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 ---------------------------------------------------------------------------- Before upgrading to 4.2 I've recompiled kernel building the squashfs as a module and rebuilding the tools package to ensure a clean environment. After a reboot I tried again to build an image with version 4.0 ------------------ Step to reproduce: 1. # mksquashfs /root/wrk/portage /var/portage/portage-current.sqfs ## image mounted on /usr/portage 2. # ls -lR /usr/portage | grep -i "error" ls: impossibile accedere a ./metadata/cache/kde-base/kdech-4.6.5Wu: Errore di input/output ls: impossibile accedere a ./metadata/cache/net-analyzer/cry: Errore di input/output
Created attachment 296307 [details] emerge --info
I don't have that problem. Generated with: #!/bin/bash DESTDIR="/data/www/j-schmitz.net/files/htdocs/squashed-portage/" NUMCPU=$(echo "$(cat /proc/cpuinfo | grep processor | wc -l) * 2" | bc) if [[ $UID != "0" ]]; then ewarn " you must be root" exit fi DATE=$(date "+%Y%m%d%H%M") LOCAL="$(mktemp -d)" TMPDIR="$(mktemp -d)" SQFSOPTS="${SQFSOPTS} -noappend -no-sparse -no-recovery -noI -no-xattrs" SQFSOPTS="${SQFSOPTS} -force-uid 250 -force-gid 250" SQFSOPTS="${SQFSOPTS} -processors $NUMCPU" SQFSOPTS="${SQFSOPTS} -e ${TMPDIR}/distfiles/* ${TMPDIR}/local" #mount -o bind ${DIST} /usr/portage/distfiles #mount -o bind ${PACK} /usr/portage/packages #mount -o bind ${LOCAL} /usr/portage/local mount -o bind /usr/portage/ ${TMPDIR} mount -o bind ${LOCAL} ${TMPDIR}/local rm -f "${DESTDIR}"/portage* echo "mksquashfs ${TMPDIR} ${DESTDIR}/portage.sqfs -comp gzip ${SQFSOPTS}" mksquashfs ${TMPDIR} "${DESTDIR}"/portage.sqfs -comp gzip ${SQFSOPTS} cp "${DESTDIR}"/portage.sqfs "${DESTDIR}"/portage.gzip.sqfs echo "mksquashfs ${TMPDIR} ${DESTDIR}/portage.sqfs -comp xz -Xbcj x86 -Xdict-size 100% ${SQFSOPTS}" mksquashfs ${TMPDIR} "${DESTDIR}"/portage.xz.sqfs -comp xz -Xbcj x86 -Xdict-size 100% ${SQFSOPTS} pushd "${DESTDIR}" >/dev/null md5sum portage.sqfs > portage.md5 md5sum portage.gzip.sqfs > portage.gzip.md5 md5sum portage.xz.sqfs > portage.xz.md5 popd >/dev/null umount ${LOCAL} ${TMPDIR} && rm -rf ${TMPDIR} ${LOCAL} chmod 664 "${DESTDIR}"/portage.sqfs "${DESTDIR}"/portage.md5
resulting in http://files.j-schmitz.net/squashed-portage/portage.gzip.sqfs http://files.j-schmitz.net/squashed-portage/portage.xz.sqfs [I] sys-fs/squashfs-tools Available versions: (3.0) 3.2_p2 (3.1) 3.4 (0) 4.0 (~)4.0-r1 (~)4.1 (~)4.1-r1 (~)4.2 {+gzip +lzma lzo xattr} Installed versions: 4.2(03:15:09 PM 11/11/2011)(gzip lzma -lzo -xattr) Homepage: http://squashfs.sourceforge.net Description: Tool for creating compressed filesystem type squashfs Found 2 matches.
I didn't try with the option "-noappend -no-sparse -no-recovery -noI -no-xattrs" as in your script. I'll check your portage.gzip.sqfs if it's something related with my kernel (even if it's a "simple" gentoo-sources-2.6.39-gentoo-r3), can you please check without additional options? It sounds strange for me too, squashfs has always done his job ---------- # qlop -l squashfs-tools Tue Nov 17 00:21:17 2009 >>> sys-fs/squashfs-tools-4.0 Fri Oct 29 03:35:42 2010 >>> sys-fs/squashfs-tools-4.0 Sun Dec 11 20:17:23 2011 >>> sys-fs/squashfs-tools-4.0 Sun Dec 18 20:59:08 2011 >>> sys-fs/squashfs-tools-4.2 ---------- My server daily sync and recreate the squashed portage, and something went wrong, looking at my logfiles, between 5-11 december. I didn't upgrade any packages in that interval, no kernel installation or rebooting. So I thought something went wrong while syncing, but trying again with a clean portage (tar.bz2 downloaded) didn't fix, and now that upgrading to 4.2 solve the problem proves (if I check properly all my logs) that something not work in squashfs-tools-4.0 itself or in conjunction with other things/circumstances. Hope to help, and tnx in advance to justin for your effort.
Could you please provide a version of your squashed portage, so I can test that here?
Sure, but I've upload limit-size to 1MB... :p Let me know if we can override this to post the file here as reference, or if you want I can upload in some "big-transfer-whathever-cloud"...
if you could, place it somewhere in the cloud.
Uploading now... http://dl.dropbox.com/u/2832401/portage-current.sqfs.4.0
(In reply to comment #8) > Uploading now... > > http://dl.dropbox.com/u/2832401/portage-current.sqfs.4.0 I can reproduce this error. for reference I place the image on my devspace http://dev.gentoo.org/~jlec/paste/portage-current.sqfs.4.0 From dmesg SQUASHFS error: Unable to read directory block [345c939:10] SQUASHFS error: Unable to read directory block [345c939:10] SQUASHFS error: Unable to read directory block [34b8181:d] SQUASHFS error: Unable to read directory block [34b8181:d] SQUASHFS error: Unable to read directory block [34bacc1:3] SQUASHFS error: Unable to read directory block [34bacc1:3] SQUASHFS error: Unable to read directory block [34c7230:4] SQUASHFS error: Unable to read directory block [34c7230:4] SQUASHFS error: Unable to read directory block [34cadf8:1] SQUASHFS error: Unable to read directory block [34cadf8:1] SQUASHFS error: Unable to read directory block [34ccb98:3] SQUASHFS error: Unable to read directory block [34ccb98:3] SQUASHFS error: Unable to read directory block [34d26a6:16] SQUASHFS error: Unable to read directory block [34d26a6:16] SQUASHFS error: Unable to read directory block [34d5103:5] SQUASHFS error: Unable to read directory block [34d5103:5] SQUASHFS error: Unable to read directory block [34d6fd2:8] SQUASHFS error: Unable to read directory block [34d6fd2:8] SQUASHFS error: Unable to read directory block [34da39d:18] SQUASHFS error: Unable to read directory block [34da39d:18] SQUASHFS error: Unable to read directory block [34e0e27:13] SQUASHFS error: Unable to read directory block [34e012d:d67] SQUASHFS error: Unable to read directory block [34e0e27:13] SQUASHFS error: Unable to read directory block [34e5bcc:1] SQUASHFS error: Unable to read directory block [34e5bcc:1] SQUASHFS error: Unable to read directory block [34e78da:6] SQUASHFS error: Unable to read directory block [34e78da:6] SQUASHFS error: Unable to read directory block [34e92d8:c] SQUASHFS error: Unable to read directory block [34e92d8:c] SQUASHFS error: Unable to read directory block [34ec061:13] SQUASHFS error: Unable to read directory block [34eb1bd:19e5] SQUASHFS error: Unable to read directory block [34ec061:13] SQUASHFS error: Unable to read directory block [34f07a1:1] SQUASHFS error: Unable to read directory block [34f07a1:1] SQUASHFS error: Unable to read directory block [34fa431:2] SQUASHFS error: Unable to read directory block [34fa431:2] SQUASHFS error: Unable to read directory block [3507f98:6] SQUASHFS error: Unable to read directory block [3507f98:6]
Ok, at least seems it's not a kernel issue... So in summary (on my system): - version 4.0 produce "malformed" squashed portage tree - version 4.2 seems not affected I explicit "portage tree" because the system synced daily portage trees between 5-11 December (can't be more precise giving my log files), and I haven't done any upgrades in that interval, so I think it's related in some way to the "input" of mksquashfs. A difference Justin, with your configuration, are the use flags: sys-fs/squashfs-tools-4.0: +gzip +lzma +lzo xattr >> errors sys-fs/squashfs-tools-4.0: +gzip +lzma +lzo +xattr >> errors sys-fs/squashfs-tools-4.2: +gzip +lzma +lzo +xattr >> ok Didn't tell you before because seems unrelated creating images with the default gzip compression algoritm, but... who knows... If I've time I can check if it's an input-related problem, testing with different portage-trees snapshots, but maybe a detailed 4.2 changelog could tell the fix of the "in rare case" bug that seem I've hit. Let me know anyway if you think that this deserve deep analysis or if it's better stabilizing 4.2 and "forgive"... ;-)
Have you looked at bug #366607 yet?
Not since now, well.. seems not too much promising for stabilization, even if gzip maybe could be set as a mandatory dependency for that release.
Created attachment 298635 [details, diff] this is the diff between the working (1.7) and broken (1.9) ebuild to the maintainer: please NEVER edit ebuild in place but create a copy and a new revision. otherwise people will waste so much time figuring out what went wrong instead of just using the previous revision. thanks!
re-emerging squashfs-tools-4.0 with that old 1.7 ebuild doesnt fix the problem. compilers and flags were also the same between builds. i only switched linux version recently, dont know if that would affect mksquashfs build. fyi, there is no need to mount that squashfs file: # unsquashfs -l blah.squashfs > /dev/null gives an segfault after some time for the broken images. if there is interest to investigate further, please tell me. otherwise squashfs-tools-4.2 works for me and hopefully it gets stabilized soon.
Um, so with all the other bugs out of the way, does version 4.2 fix this?
(In reply to comment #15) > Um, so with all the other bugs out of the way, does version 4.2 fix this? I've also seen this behavior. tools-4.0 creates invalid filesystems that will cause unsquashfs to crash (with a buffer overflow caught by gcc's fortify) and my 3.2.1 kernel to BUG and die. Currently stable tools-4.2 appears to create valid filesystems that exhibit none of the above behavior. I'd vote for closing this.
So after we have 4.2 stable, lets remove all other versions.
(In reply to Justin Lecher from comment #17) > So after we have 4.2 stable, lets remove all other versions. Well, not all of them. People like having the 3.* SLOTs. Assumed fixed.
