Gentoo/FreeBSD has two sed. One is sed of BSD. Another is gsed of GNU. As for these, operation is different. As a result, catalyst cannot appropriately substitute and causes this problem. Catalyst displayed the error of sed when BSD's sed was used, and build was added to the USE flag. The problem was not in any GNU sed. # cat /var/tmp/catalyst/tmp/default/stage3-amd64-bsdsed/etc/make.conf USE="mmx sse sse2" USE="${USE} build" # cat /var/tmp/catalyst/tmp/default/stage3-amd64-gnused/etc/make.conf USE="mmx sse sse2" Reproducible: Always
Created attachment 269931 [details] patch for catalyst-2.0.6.915 FYI, When overlay is specified, PORTDIR_OVERLAY remains in make.conf of the stage1 file. This code removes it. +++ catalyst-2.0.6.915/targets/stage1/stage1-preclean-chroot.sh @@ -4,6 +4,8 @@ source /tmp/chroot-functions.sh +sed_selector +${SED} -i '/^PORTDIR_OVERLAY/d' /etc/make.conf # cat /var/tmp/catalyst/tmp/default/stage1-amd64-fbsd-20110413/tmp/stage1root/etc/make.conf PORTDIR_OVERLAY="/usr/local/portage"
Created attachment 269933 [details] catalyst-stage1.log # catalyst -C target=stage1 version_stamp=fbsd-20110413 profile=default/bsd/fbsd/amd64/8.0 snapshot=fbsd-amd64-exp source_subpath=default/stage3-amd64-fbsd-8-exp subarch=amd64 rel_type=default portage_confdir=/mnt/stage3/unstable/usr/local/ready/portage/etc/portage portage_overlay=/mnt/stage3/unstable/usr/local/ready/portage chost=x86_64-gentoo-freebsd8.0 &> ~/catalyst-stage1.log You will be able to find two or more following messages. sed: 1: "/etc/make.conf": invalid command code m
Created attachment 269935 [details] catalyst-stage1-patched.log It is a log when the same work is executed with catalyst that applies patch.
Created attachment 305721 [details, diff] does not remove the libfl.a and libfl_pic.a from stage1. /usr/lib/libfl.a or /usr/lib/libfl_pic.a is not found, if you fail to create the stage3, please try this patch.
added bsd team.
Created attachment 305725 [details, diff] patch for modules/generic_stage_target.py Please try this patch tar to not operate properly when the message will be displayed below. tar: Pathname can't be converted from UTF-8 to current locale. tar: Pathname in pax header can't be converted to current locale. tar: Linkname in pax header can't be converted to current locale. tar: Error exit delayed from previous errors. If you fail to create stage*tar.bz2 files, please try the following steps. 1. export LANG=en_US.UTF-8 2. catalyst -f stage1.spec 3. catalyst -f stage2.spec 4. catalyst -f stage3.spec issues related to tar will occur in FreeBSD 9.0 maybe.
Created attachment 314077 [details] New problem, in preclean sequence Important thing is following in the error message. OSError: [Errno 2] No such file or directory: '/dev/urandom' * gcc-config: Could not get portage CHOST! * gcc-config: You should verify that CHOST is set in one of these places: * gcc-config: - //etc/portage/make.conf * gcc-config: - active environment
Created attachment 314079 [details, diff] patch for catalyst-2.0.8.1-r1 The new patch to fix the problem to mount devfs on /tmp/stage1root/dev.
Created attachment 326448 [details, diff] patch for catalyst-2.0.11 patch update
Created attachment 330332 [details, diff] patch for catalyst-2.0.12.1 Changes * added GNU sed wrapper. * gsed use instead of sed on FreeBSD. * linux32 is exist only on Linux. Use only chroot on FreeBSD. * need to mount the devfs to "/tmp/stage1root/dev" on FreeBSD. see Comment #7 * need to change run_merge "-e system" for packages to hard code sed. * tar -I lbzip2 does not work on FreeBSD. change not to use it. * tmpfs also works on FreeBSD. bsd team, I had to push catalyst-2.0.12.1 on bsd-overlay. Please test and feedback. tmpfs is enabled by modifying /etc/catalyst/catalyst.conf. You will be able to complete the stages more speedily. FYI, # diff catalyst.conf.orig catalyst.conf 67c67 < options="autoresume kerncache pkgcache seedcache snapcache" --- > options="autoresume kerncache pkgcache seedcache snapcache tmpfs" 93c93 < # var_tmpfs_portage=16 --- > var_tmpfs_portage=2
Instead of adding so many checks and changes for BSD, perhaps it's time we create modules for linux and BSD, where we can add definitions for each. What do you think?
(In reply to comment #11) > Instead of adding so many checks and changes for BSD, perhaps it's time we > create modules for linux and BSD, where we can add definitions for each. > What do you think? sounds sensible, but I don't get where you want to create this. e.g. for the first hunk it would make sense (catalyst-2.0.12.1.orig/arch/x86.py) but then the lbzip2 usage could (and should wrt this bug) be made optional or configurable, the tmpfs changes reduce the differences between linux and fbsd, the sed vs gsed issue could also be made configurable to avoid a if/then/else about the hunk for emerge system (catalyst-2.0.12.1.orig/targets/stage3/stage3-chroot.sh), can it hurt to do this unconditionally ? this would reduce the difference. a fbsd module would then just set the linux32, lbzip2 and sed usage defaults that work on fbsd, right ?
Created attachment 357332 [details, diff] catalyst-2.0.14-fbsd.patch patch updated. @bsd team, I had to push catalyst-2.0.14 on bsd-overlay. I’ve tested this patch. # ls -1 /var/tmp/catalyst/builds/default/stage*0829.* /var/tmp/catalyst/builds/default/stage1-i686-fbsd-9.2-20130829.tar.bz2 /var/tmp/catalyst/builds/default/stage1-i686-fbsd-9.2-20130829.tar.bz2.CONTENTS /var/tmp/catalyst/builds/default/stage1-i686-fbsd-9.2-20130829.tar.bz2.DIGESTS /var/tmp/catalyst/builds/default/stage2-i686-fbsd-9.2-20130829.tar.bz2 /var/tmp/catalyst/builds/default/stage2-i686-fbsd-9.2-20130829.tar.bz2.CONTENTS /var/tmp/catalyst/builds/default/stage2-i686-fbsd-9.2-20130829.tar.bz2.DIGESTS /var/tmp/catalyst/builds/default/stage3-i686-fbsd-9.2-20130829.tar.bz2 /var/tmp/catalyst/builds/default/stage3-i686-fbsd-9.2-20130829.tar.bz2.CONTENTS /var/tmp/catalyst/builds/default/stage3-i686-fbsd-9.2-20130829.tar.bz2.DIGESTS
Created attachment 425186 [details, diff] catalyst-3.0_rc1-fbsd.patch Update the patch.
Created attachment 425188 [details, diff] patch for catalyst-3.0_rc1.ebuild FreeBSD of tar does not support --xattrs and -I options. Force set compression_mode and decompressor_search_order...
need to change CONTENTS_DEFINITIONS and CONTENTS_SEARCH_ORDER of dev-python/pydecomp for FreeBSD.
Created attachment 425196 [details, diff] catalyst-3.0_rc1-fbsd.patch Update comment. catalyst-3.0_rc1/targets/stage3/stage3-chroot.sh -# workaround fix bug 425530, 438112, 448256 +# Workaround of SED contained in sys-devel/libtool. bug #438112
i've fixed the setarch issue in git in a diff way: https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=ce31e7ab7b5b8968b7ff0c17bbe00d09921729b7 also the make.conf/sed-i issue in a diff way: https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=d8b2a96f1d96d1c64b10c3c9607fd8ebf2a983a5 for the sed() func in chroot-functions.sh, can't you do: [ -e /usr/bin/gsed ] && alias sed=gsed why do you need to mess with the python shebang ?
Nice, thank you Yuta I have a suggestion for sed +# GNU sed wrapper +sed(){ + if [ -e /usr/bin/gsed ] + then + /usr/bin/gsed "$@" + elif [ -e /bin/sed ] + then + /bin/sed "$@" + elif [ -e /usr/bin/sed ] + then + /usr/bin/sed "$@" + else + echo "ERROR: failed the selection of sed." + exit 1 + fi +} + instead of this and the if ... in several places in the python code, we should set the sed command to use in the config at the start of catalyst with a python version of the above. then use that config'd sed command everywhere. In the bash side you would use ${clst_sed} as it is automatically created by the existing code. in main.py _main() for sed in ('/usr/bin/gsed', '/bin/sed', '/usr/bin/sed'): if os.path.exists(sed): conf_values["sed"] = sed break and a small loop like the above is better than seemingly ever expanding if/elif/elif/... don't forget to add a default sed in defaults.py confdefaults Also, while you re-do the patch, can you make the /usr/bin/env python changes a separate patch/commit please. That is a separate mistake. hmm, resume.py shouldn't have a shebang it is only an installed lib, but the other 2 yes, need one.
@vapier, @dolsen Thanks for review. * python shebang I have to change python shebang is because it did not work on FreeBSD. I search the Bugzilla, I found bug 568974 of this issue. Install dev-lang/python-exec-2.3 and app-eselect/eselect-python-20160207, I was confirmed that issue has been fixed. Of course, drop from patch of catalyst. FYI, $ cat ~/test.py #!/usr/bin/python import os if os.uname()[0] == "Linux": print ("Linux") else: print ("non-Linux") $ ~/test.py /home/nigoro/test.py: line 2: import: command not found /home/nigoro/test.py: line 4: syntax error near unexpected token `[0]' /home/nigoro/test.py: line 4: `if os.uname()[0] == "Linux":' $ gsed -i 's:#!/usr/bin/python:#!/usr/bin/env python:g' ~/test.py $ ~/test.py non-Linux * sys-devel/libtool issue. I filed new bug 574550. Drop from patch of catalyst.
Created attachment 425294 [details, diff] catalyst-3.0_rc1 sed part. alias=gsed did not work properly... Apply this patch I was confirm that can create stage 1,2,3 on Gentoo/FreeBSD. Note, bash code was replaced with the following command. gsed -i 's:sed -:${clst_sed} -:g' targets/stage1/*.sh gsed -i 's:sed -:${clst_sed} -:g' targets/stage2/*.sh gsed -i 's:sed -:${clst_sed} -:g' targets/support/*.sh
Created attachment 425296 [details, diff] catalyst-3.0_rc1 tmpfs part. With the same options as Linux, tmpfs can be used on FreeBSD. Please add tmpfs support. https://www.freebsd.org/cgi/man.cgi?query=tmpfs&apropos=0&sektion=5&manpath=FreeBSD+10.2-RELEASE&arch=default&format=html
Comment on attachment 425188 [details, diff] patch for catalyst-3.0_rc1.ebuild why doesn't lbzip2 work on bsd ? shouldn't that get filed/fixed independently ? same goes for tar/xattr behavior. if tools are broken, we should fix the tools, not workaround them in catalyst.
Comment on attachment 425294 [details, diff] catalyst-3.0_rc1 sed part. >+ for sed in ('/usr/bin/gsed', '/bin/sed', '/usr/bin/sed'): >+ if os.path.exists(sed): >+ conf_values["sed"] = sed >+ break pretty sure that os.path.exists is incorrect. you want to check if the path exists inside the chroot, not outside of it.
(In reply to SpanKY from comment #23) > Comment on attachment 425188 [details, diff] [details, diff] > patch for catalyst-3.0_rc1.ebuild > > why doesn't lbzip2 work on bsd ? shouldn't that get filed/fixed > independently ? tar on Gentoo/FreeBSD is provided by app-arch/libarchive. It does not support -I lbzip2 option. I was confirm that can use other tools by using --use-compress-program instead of -I. I think lbzip2 can be used by changing dev-python/pydecomp. More info please see https://bugs.gentoo.org/show_bug.cgi?id=574422#c3 > same goes for tar/xattr behavior. if tools are broken, we should fix the > tools, not workaround them in catalyst. bsdtar, xattr seems to be supported by default. If you don't set --no-xattr, probably xattr is enabled. https://github.com/libarchive/libarchive/commit/48b288a03347e49f2f9501f040f626f916195de6 $ equery b /usr/bin/tar * Searching for /usr/bin/tar ... app-arch/libarchive-3.1.2-r3 (/usr/bin/tar -> bsdtar) app-arch/libarchive-3.1.2-r3 (/usr/bin/bsdtar) GNU tar is named gtar on Gentoo/FreeBSD. If remove app-arch/tar xattr from p.use.mask, support is not enabled. # emerge -pv app-arch/tar [ebuild R ] app-arch/tar-1.28-r1::gentoo USE="acl nls xattr -minimal (-selinux) -static" 0 KiB # gtar --xattrs -cpf /tmp/test.tar.bz2 test.txt gtar: XATTR support is not available $ equery f app-arch/tar | grep /bin/ /bin/gtar
ok, so tar is being handled in bug 574422, and tar/attr in bug 538654. that leaves two issues in this bug: sed & tmpfs. let's just focus on those here.
(In reply to SpanKY from comment #24) > Comment on attachment 425294 [details, diff] [details, diff] > catalyst-3.0_rc1 sed part. > > >+ for sed in ('/usr/bin/gsed', '/bin/sed', '/usr/bin/sed'): > >+ if os.path.exists(sed): > >+ conf_values["sed"] = sed > >+ break > > pretty sure that os.path.exists is incorrect. you want to check if the path > exists inside the chroot, not outside of it. I still don't have any good ideas. Using the previous sed() func in chroot-functions.sh ... ?
Yeah, If it needs to do the sed check and assignment within the chroot, it needs to be done within the bash side. The python side is pretty much all prep and then finalize and cleanup. The bash side handling the core building in the middle. We could do the equivalent of that python loop. we just need to determine where/when the best place is to do it. Too early and they won't be installed. But it does need to run before the first time a sed command is needed.
(In reply to Brian Dolbec from comment #28) don't you have the path to the chroot at that point ? so you can append /usr/bin/sed to it and then run os.path.exists.
In that case it would have to move from where it is now, into stagebase.py. Then be run at some point after the target is set and unpacked, but before preparing the environment for the chroot. That might be easier than figuring out the best place in the bash code to insert it. It was just that since your other change, sed is no longer used on the python side. Is why I thought of moving to the bash side. But thinking a bit more about it now, Keeping it in the python side, the loop options to check for could also be moved to the defaults where they can easily be changed. Which also leads to the possibility to make that configurable if other alternatives come along or are desired. Especially for other arches or embedded.
Created attachment 509212 [details, diff] catalyst-3.0.0 sed part. I've just updated attachment 425294 [details, diff] for catalyst-3.0.0.
Created attachment 509214 [details, diff] catalyst-3.0.0 tmpfs part. I've just updated attachment 425296 [details, diff] for catalyst-3.0.0.
Created attachment 509216 [details, diff] add env=self.env to call umount, patch for catalyst-3.0.0. umount exists in /sbin on Gentoo/FreeBSD. # which umount /sbin/umount 02 Dec 2017 20:27:18 JST: NOTICE : --- Running action sequence: unbind 02 Dec 2017 20:27:19 JST: ERROR : Exception running action sequence unbind Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/catalyst/base/stagebase.py", line 1438, in run getattr(self, x)() File "/usr/lib/python2.7/site-packages/catalyst/base/stagebase.py", line 993, in unbind cmd(['umount', target]) File "/usr/lib/python2.7/site-packages/catalyst/support.py", line 47, in cmd proc = Popen(args, env=env) File "/usr/lib/python2.7/subprocess.py", line 390, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory 02 Dec 2017 20:27:19 JST: NOTICE : Cleaning up... Running unbind() Traceback (most recent call last): File "/usr/lib/python-exec/python2.7/catalyst", line 37, in <module> main(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/catalyst/main.py", line 277, in main return _main(parser, opts) File "/usr/lib/python2.7/site-packages/catalyst/main.py", line 442, in _main success = build_target(addlargs) File "/usr/lib/python2.7/site-packages/catalyst/main.py", line 114, in build_target return target.run() File "/usr/lib/python2.7/site-packages/catalyst/base/stagebase.py", line 1450, in run self.unbind() File "/usr/lib/python2.7/site-packages/catalyst/base/stagebase.py", line 993, in unbind cmd(['umount', target]) File "/usr/lib/python2.7/site-packages/catalyst/support.py", line 47, in cmd proc = Popen(args, env=env) File "/usr/lib/python2.7/subprocess.py", line 390, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory
Created attachment 509218 [details, diff] disable snakeoil's namespace module on Gentoo/FreeBSD. Fix the following issue. 10 Dec 2017 19:35:14 JST: NOTICE : Loading configuration file: /etc/catalyst/catalyst.conf 10 Dec 2017 19:35:14 JST: NOTICE : Processing spec file: /tmp/mk_stages_amd64_11.1/stage1.spec Traceback (most recent call last): File "/usr/lib/python-exec/python2.7/catalyst", line 37, in <module> main(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/catalyst/main.py", line 277, in main return _main(parser, opts) File "/usr/lib/python2.7/site-packages/catalyst/main.py", line 431, in _main hostname='catalyst') File "/usr/lib/python2.7/site-packages/snakeoil/process/namespaces.py", line 327, in simple_unshare unshare(CLONE_NEWNS) File "/usr/lib/python2.7/site-packages/snakeoil/process/namespaces.py", line 65, in unshare if libc.unshare(ctypes.c_int(flags)) != 0: File "/usr/lib/python2.7/ctypes/__init__.py", line 379, in __getattr__ func = self.__getitem__(name) File "/usr/lib/python2.7/ctypes/__init__.py", line 384, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) AttributeError: Undefined symbol "unshare"
Created attachment 509224 [details, diff] patch for catalyst-3.0.0.ebuild The dependence issue for amd64-fbsd will be fixed. Thanks.
The sed patch looks pretty good. But why did you add the detection/setting in main.py where you did? +++ b/catalyst/main.py @@ -345,6 +345,12 @@ def _main(parser, opts): conf_values['compress_definitions'] = COMPRESS_DEFINITIONS # TODO add capability to config/spec new definitions + # detect GNU sed + for sed in ('/usr/bin/gsed', '/bin/sed', '/usr/bin/sed'): + if os.path.exists(sed): + conf_values["sed"] = sed + break + # Start checking that digests are valid now that hash_map is initialized if "digests" in conf_values: digests = set(conf_values['digests'].split()) It looks like this could go elsewhere, at least not in the middle of the compress/hash/decompress initialization.
All changes pushed to the master branch with a few modifications.
Released in catalyst-3.0.1
