net-misc/frr (tested on 10.1.1) needs to be built out-of-source for cross compiles to work. (See error message below) The details are also documented in https://docs.frrouting.org/projects/dev-guide/en/latest/cross-compiling.html#cross-compiling-frr-itself which does the "other" solution (build clippy manually), but FRR's build system does that automatically if it's invoked out-of-source (and if something else changes in cross-compile needs, it's likely this will be automatically handled then). Additionally [yes technically separate bug], FRR autoconf behavior about --sysconfdir, --localstatedir and --runstatedir was changed to be in line with "normal" autotools expectations. FRR's configure.ac currently has a workaround that makes the old options still work. Reproducible: Always Steps to Reproduce: 1. create crosscompilation environment with sys-devel/crossdev (actual cross target is irrelevant, can also be cross from glibc to musl, which is easier to test) 2. attempt to cross-emerge net-misc/frr Actual Results: Build fails: ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=powerpc64-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --datarootdir=/usr/share --disable-dependency-tracking --disable-silent-rules --disable-static --docdir=/usr/share/doc/frr-10.1.1 --htmldir=/usr/share/doc/frr-10.1.1/html --with-sysroot=/usr/powerpc64-linux-gnu LEX=flex --with-pkg-extra-version=-gentoo --enable-configfile-mask=0640 --enable-logfile-mask=0640 --libdir=/usr/lib/frr --sbindir=/usr/lib/frr --libexecdir=/usr/lib/frr --sysconfdir=/etc/frr --localstatedir=/run/frr --with-moduledir=/usr/lib/frr/modules --enable-user=frr --enable-group=frr --enable-vty-group=frr --enable-multipath=64 --disable-doc --disable-fpm --disable-grpc --enable-realms --disable-nhrpd --disable-rpki --disable-snmp configure: loading site script /usr/share/config.site configure: loading site script /usr/share/config.site.d/80crossdev.conf configure: loading site script /usr/share/crossdev/include/site/linux configure: loading site script /usr/share/crossdev/include/site/linux-gnu checking whether --sysconfdir option is FRR-specific... yes, ends in /frr - removing suffix configure: WARNING: Please remove /frr suffix from --sysconfdir="/etc/frr" (it should be /etc in 99% of cases) checking whether --localstatedir option is FRR-specific... yes, ends in /run/frr - removing suffix configure: WARNING: Please remove /run/frr suffix from --localstatedir=/run/frr (it should be /var in 99% of cases) configure: WARNING: ^ configure: WARNING: ^ configure: WARNING: ^ warnings regarding system path configuration were printed above configure: WARNING: ^ paths have been adjusted by temporary workarounds configure: WARNING: ^ please fix your ./configure invocation (remove /frr) so it will work without the workarounds configure: WARNING: ^ configure: WARNING: ^ checking build system type... x86_64-pc-linux-gnu checking host system type... powerpc64-unknown-linux-gnu configure: error: cross-compilation is only possible with builddir separate from srcdir or by building clippy separately and using the --with-clippy option. create a separate directory and run as .../path-to-frr/configure. Expected Results: Build succeeds. Patch attached in a second. I've omitted further details here since there's a patch.
Created attachment 917968 [details, diff] ebuild patch
Right, this is why we had clippy split out originally (bug 766959). clippy was removed because of: # Jaco Kroon <jaco@uls.co.za> (2024-11-15) # clippy exists purely for frr (even comes from the same sources). This # package split is no longer implemented from frr-9.1.2 onwards, so this # package is no longer required. dev-util/clippy but I'm not sure if this missed the point of why it was split, or what was happening there.
(In reply to Sam James from comment #2) > Right, this is why we had clippy split out originally (bug 766959). > > clippy was removed because of: > > # Jaco Kroon <jaco@uls.co.za> (2024-11-15) > # clippy exists purely for frr (even comes from the same sources). This > # package split is no longer implemented from frr-9.1.2 onwards, so this > # package is no longer required. > dev-util/clippy > > but I'm not sure if this missed the point of why it was split, or what was > happening there. Came from before my time, and didn't notice bug 766959 at the time, so yes, missed the point of the split. This patch implements option 2? Is there any reason to prefer the one over the other? Ie, is one more efficient than the other? This does look simpler to maintain, and I'm happy to push a PR, or review one if Alarig gets to this before me, for a set of -r1s for all in-tree versions based on this patch provided here.
(In reply to Sam James from comment #2) > Right, this is why we had clippy split out originally (bug 766959). Ooof. Speaking as the creator of clippy, it wasn't intended to be used outside FRR or even packaged separately. It is inherently tied to FRR's CLI functions (which due to libfrr not having any kind of stable API aren't used anywhere else) and is also not built as a "proper" Python package. (I made it an executable purely to avoid dealing with the hassles of making and then loading a regular Python module, without shenanigans like accidentally using an old version from the system or so.) (In reply to Jaco Kroon from comment #3) > This patch implements option 2? Not sure which is which, it implements the option where FRR's build system handles this all internally; the reason for requiring a separate build directory is that autoconf refuses to use an already-configured source directory as the source for another separate build directory. FRR just runs its own ./configure a second time, once for the cross target/host, and once for the build host, and builds & uses clippy in the build host tree. There's a chance that FRR might add further build-time tooling, in which case this approach will still work while the "DIY" build clippy separately won't cover that. > Is there any reason to prefer the one over the other? Ie, is one more > efficient than the other? It won't make any difference at all for the output, and unless you build FRR a lot but don't want to rebuilding clippy every time it won't be more efficient either. (It's a pretty small tool, the efficiency saving wouldn't even be from its build but rather from the saved extra configure run.) FRR's build system only does the double-configure-itself thing if it detects a cross build, so regular builds aren't slowed down by this either. In theory, if you have a weird Python installation, the 2nd run to set up the build for clippy might need extra build options that can't easily be supplied if FRR does this on its own. But we haven't had any issues reported with this on FRR upstream (there were some other cross-compile issues, but nothing where controlling the 2nd configure run would've helped.) > This does look simpler to maintain, and I'm happy to push a PR, or review > one if Alarig gets to this before me, for a set of -r1s for all in-tree > versions based on this patch provided here. You might want to double check my changes regarding the changes for the directory parameters, I really sandwiched 2 bugs/fixes into one here. (My excuse is that supplying a patch is un-lazy so it balances out :D)
Looking at the text in https://bugs.gentoo.org/766959 it says "I tried to do 2) via out-of-source.eclass but didn't manage to get the build working" - I have no idea what the issues back then were, I've tested my patch with an x86_64 -> powerpc64 cross build and it works. (It's possible something related was fixed upstream in the 4 years since...)
With the changes w.r.t. system folders included: plastiekpoot [08:54:18] ~ # /etc/init.d/frr start * /run/frr: creating directory * /run/frr: correcting owner * Starting FRR ... watchfrr failed to start, exited 1 % Can't open configuration file /etc/vtysh.conf due to 'No such file or directory'. Configuration file[/etc/frr.conf] processing failure: 11 Exiting: failed to connect to any daemons. [ ok ] So I'm going to ignore the changes to the arguments passed to ./configure because then the above works. PR to follow shortly. Would appreciate if someone in a position to cross-compile (I'm a little short on capacity) would just test the PR for cross-compile prior to it being merged, but in the non-cross case it looks good.
(In reply to Jaco Kroon from comment #6) > With the changes w.r.t. system folders included: > > plastiekpoot [08:54:18] ~ # /etc/init.d/frr start > * /run/frr: creating directory > * /run/frr: correcting owner > * Starting FRR ... > watchfrr failed to start, exited 1 > % Can't open configuration file /etc/vtysh.conf due to 'No such file or > directory'. > Configuration file[/etc/frr.conf] processing failure: 11 Hrm. I've double checked and I'm not seeing this behavior. Can you check if you somehow have a mismatched version of the init script or something around? > PR to follow shortly. Would appreciate if someone in a position to > cross-compile (I'm a little short on capacity) would just test the PR for > cross-compile prior to it being merged, but in the non-cross case it looks > good. I've done a cross-build for arm-linux-gnueabihf with your PR and it works as expected.
(In reply to David Lamparter from comment #7) > (In reply to Jaco Kroon from comment #6) > > With the changes w.r.t. system folders included: > > > > plastiekpoot [08:54:18] ~ # /etc/init.d/frr start > > * /run/frr: creating directory > > * /run/frr: correcting owner > > * Starting FRR ... > > watchfrr failed to start, exited 1 > > % Can't open configuration file /etc/vtysh.conf due to 'No such file or > > directory'. > > Configuration file[/etc/frr.conf] processing failure: 11 > > Hrm. I've double checked and I'm not seeing this behavior. Can you check if > you somehow have a mismatched version of the init script or something around? I did not. Unless you changed something in the init script on your local system that we're not aware of here? Regardless, the fact that it then searches for config files in /etc/ rather than /etc/frr/ is a problem. Could be a version issue. In which version was the ./configure behaviour changed? Could be that on older versions we need to keep the old behaviour and apply new options to the newer versions? > > PR to follow shortly. Would appreciate if someone in a position to > > cross-compile (I'm a little short on capacity) would just test the PR for > > cross-compile prior to it being merged, but in the non-cross case it looks > > good. > > I've done a cross-build for arm-linux-gnueabihf with your PR and it works as > expected. Thank you.
(In reply to Jaco Kroon from comment #8) > (In reply to David Lamparter from comment #7) > > (In reply to Jaco Kroon from comment #6) > > > With the changes w.r.t. system folders included: > > > > > > plastiekpoot [08:54:18] ~ # /etc/init.d/frr start > > > * /run/frr: creating directory > > > * /run/frr: correcting owner > > > * Starting FRR ... > > > watchfrr failed to start, exited 1 > > > % Can't open configuration file /etc/vtysh.conf due to 'No such file or > > > directory'. > > > Configuration file[/etc/frr.conf] processing failure: 11 > > > > Hrm. I've double checked and I'm not seeing this behavior. Can you check if > > you somehow have a mismatched version of the init script or something around? > > I did not. Unless you changed something in the init script on your local > system that we're not aware of here? > > Regardless, the fact that it then searches for config files in /etc/ rather > than /etc/frr/ is a problem. > > Could be a version issue. In which version was the ./configure behaviour > changed? Could be that on older versions we need to keep the old behaviour > and apply new options to the newer versions? Argh, yes, silly me, this is a >=10.0 change. I guess you've been testing on 9.1? Apologies, I should've been more clear here.
These are gentoo default: --sysconfdir=/etc --localstatedir=/var/lib Do we need an explicit --runstatedir option? I think we do (from ./configure): 1222 sysconfdir='${prefix}/etc' 1223 sharedstatedir='${prefix}/com' 1224 localstatedir='${prefix}/var' 1225 runstatedir='${localstatedir}/run' Doesn't look like FRR actually uses localstatedir other than for finding /run, so we just need --runstatedir. PR updated, please confirm everyone is happy.
(In reply to Jaco Kroon from comment #10) > These are gentoo default: > > --sysconfdir=/etc --localstatedir=/var/lib Huh, why is that /lib there - localstatedir in normal autoconf is just "PREFIX/var"? > Do we need an explicit --runstatedir option? I think we do (from > ./configure): > > 1222 sysconfdir='${prefix}/etc' > 1223 sharedstatedir='${prefix}/com' > 1224 localstatedir='${prefix}/var' > 1225 runstatedir='${localstatedir}/run' > > > Doesn't look like FRR actually uses localstatedir other than for finding > /run, so we just need --runstatedir. It does use both, and --localstatedir should not have a trailing /lib: $ git grep -Pn 'frr_(lib|run)statedir' configure.ac lib/config_paths.h.in configure.ac:82:frr_runstatedir="\${runstatedir}/frr" configure.ac:94:frr_libstatedir="\${localstatedir}/lib/frr" configure.ac:2766:AX_SUBST_EXPANDED([frr_runstatedir]) configure.ac:2767:AX_SUBST_EXPANDED([frr_libstatedir]) configure.ac:2941:local state file dir : ${e_frr_libstatedir} configure.ac:2942:run state file dir : ${e_frr_runstatedir} lib/config_paths.h.in:14:#define FRR_RUNSTATE_PATH "@e_frr_runstatedir@" lib/config_paths.h.in:15:#define FRR_LIBSTATE_PATH "@e_frr_libstatedir@" $ git grep -Pn 'define.*frr_runstatedir' isisd/isis_main.c:64:#define ISISD_COMPAT_STATE_NAME "%s/isid-restart.json", frr_runstatedir lib/config_paths.h.in:14:#define FRR_RUNSTATE_PATH "@e_frr_runstatedir@" lib/libfrr.h:25:#define ZAPI_SOCK_NAME "%s/zserv.api", frr_runstatedir lib/mgmt_defines.h:14:#define MGMTD_FE_SOCK_NAME "%s/mgmtd_fe.sock", frr_runstatedir lib/mgmt_defines.h:15:#define MGMTD_BE_SOCK_NAME "%s/mgmtd_be.sock", frr_runstatedir ospf6d/ospf6_auth_trailer.c:33:#define OSPF6D_COMPAT_AUTHSEQ_NAME "%s/ospf6d-at-seq-no.dat", frr_runstatedir ospf6d/ospf6_main.c:44:#define OSPF6D_COMPAT_STATE_NAME "%s/ospf6d-gr.json", frr_runstatedir ospfd/ospf_main.c:61:#define OSPFD_COMPAT_STATE_NAME "%s/ospfd-gr.json", frr_runstatedir ospfd/ospf_main.c:64:#define OSPFD_COMPAT2_INST_STATE_NAME(i) "%s/ospfd-%d.json", frr_runstatedir, i $ git grep -Pn 'define.*frr_libstatedir' isisd/isis_main.c:56:#define FABRICD_STATE_NAME "%s/fabricd.json", frr_libstatedir isisd/isis_main.c:57:#define ISISD_STATE_NAME "%s/isisd.json", frr_libstatedir lib/config_paths.h.in:15:#define FRR_LIBSTATE_PATH "@e_frr_libstatedir@" mgmtd/mgmt_ds.h:32:#define MGMTD_COMMIT_FILE_PATH(id) "%s/commit-%s.json", frr_libstatedir, id mgmtd/mgmt_ds.h:33:#define MGMTD_COMMIT_INDEX_FILE_PATH "%s/commit-index.dat", frr_libstatedir ospf6d/ospf6_main.c:43:#define OSPF6D_STATE_NAME "%s/ospf6d.json", frr_libstatedir ospfd/ospf_main.c:50:#define OSPFD_STATE_NAME "%s/ospfd.json", frr_libstatedir ospfd/ospf_main.c:51:#define OSPFD_INST_STATE_NAME(i) "%s/ospfd-%d.json", frr_libstatedir, i The change was made to fix previously hardcoded "/var/lib" (completely ignoring ./configure options) for all the persistent state files, see last grep above.
(In reply to David Lamparter from comment #11) > (In reply to Jaco Kroon from comment #10) > > These are gentoo default: > > > > --sysconfdir=/etc --localstatedir=/var/lib > > Huh, why is that /lib there - localstatedir in normal autoconf is just > "PREFIX/var"? That you're asking the wrong person, but I'll check in with someone that will probably know. > > Do we need an explicit --runstatedir option? I think we do (from > > ./configure): > > > > 1222 sysconfdir='${prefix}/etc' > > 1223 sharedstatedir='${prefix}/com' > > 1224 localstatedir='${prefix}/var' > > 1225 runstatedir='${localstatedir}/run' > > > > > > Doesn't look like FRR actually uses localstatedir other than for finding > > /run, so we just need --runstatedir. > > It does use both, and --localstatedir should not have a trailing /lib: Is this for specific daemons only? I could not find that any of my installs has /var/frr or /var/lib/frr anywhere ... From your greps, this seems to relate to specific functions is ospf{,6} (which I use) and isis (which I don't), as well as mgmt which I don't think is a routing daemon but rather the overall configuration management daemon. What am I missing? Kind regards, Jaco
(In reply to Jaco Kroon from comment #12) > Is this for specific daemons only? I could not find that any of my installs > has /var/frr or /var/lib/frr anywhere ... > > From your greps, this seems to relate to specific functions is ospf{,6} > (which I use) and isis (which I don't), as well as mgmt which I don't think > is a routing daemon but rather the overall configuration management daemon. > > What am I missing? It's quite possible they're simply not used in your configuration; they are only used for: - ospf6d: the authentication trailer sequence number, to prevent replay attacks; you won't see it unless you have ospf6 auth enabled - ospf6d & ospfd: when performing a graceful restart to keep track of the grace period timer; you won't see it unless you're doing GR - isisd & fabricd: for the overload bit timer The mgmtd ones are only active if the (experimental and incomplete) commit history feature is turned on, which I don't believe is the case for Gentoo (or anyone else hopefully, it's really work in progress.) Now that I think of it, I believe we also misplaced them in /var/run/frr in some cases, which is really not great especially for the authentication trailer sequence number... reboot and everyone else ignores your packets due to seq#... [not sure if that one was affected :'(]
(In reply to David Lamparter from comment #13) > (In reply to Jaco Kroon from comment #12) > > Is this for specific daemons only? I could not find that any of my installs > > has /var/frr or /var/lib/frr anywhere ... > > > > From your greps, this seems to relate to specific functions is ospf{,6} > > (which I use) and isis (which I don't), as well as mgmt which I don't think > > is a routing daemon but rather the overall configuration management daemon. > > > > What am I missing? > > It's quite possible they're simply not used in your configuration; they are > only used for: > > - ospf6d: the authentication trailer sequence number, to prevent replay > attacks; you won't see it unless you have ospf6 auth enabled > > - ospf6d & ospfd: when performing a graceful restart to keep track of the > grace period timer; you won't see it unless you're doing GR OK. So it probably also means we need keepdir for /var/frr or /var/lib/frr then? Yea, we rely on the the subnets being 100% under our control, so no auth there. Also, Mikrotik clutched out more consistently with auth enabled, that's now out of the equation so might re-look at it just as an extra layer, definitely required if we're going to do ospf* on untrusted VLANs, but there I prefer BGP anyway as it's likely a downstream entity anyway. > - isisd & fabricd: for the overload bit timer I don't use these. > The mgmtd ones are only active if the (experimental and incomplete) commit > history feature is turned on, which I don't believe is the case for Gentoo > (or anyone else hopefully, it's really work in progress.) Definitely not. > Now that I think of it, I believe we also misplaced them in /var/run/frr in > some cases, which is really not great especially for the authentication > trailer sequence number... reboot and everyone else ignores your packets due > to seq#... [not sure if that one was affected :'(] Ok so I rechecked the configure code on the matter again, localstatedir is used to find runstatedir, otherwise localstatedir is not used at all (as far as I can determine), so just passing --runstatedir=/run sorts that too. localstate if there was any persistent really aught to go (in my opinion) to /var/lib/frr, so leaving the Gentoo default there. Also did this to confirm: plastiekpoot [15:13:52] ~ # strings $(equery files frr) | grep ^/var | grep -v ^/var/tmp And could find a few references to /var/log/frr and /var/run/netns (which will translate to /run/netns after symlinks). Similar for /run does have a few hits. As far as I'm concerned the PR is ready.
