Binary packages built with quickpkg using all default settings will overwrite config files usually protected by emerge without any warning. The resulting file contains the message "# empty file because --include-config=n when `quickpkg` was used". It is hard to conceive of any circumstances where this would be useful. I have stopped using quickpkg and given this behaviour either add this as a strongly worded caveat to the Binary Packages guide (in addition to the existing caveat) or, preferably, deprecate the program entirely. Reproducible: Always Steps to Reproduce: 1.build a package on a binhost using quickpkg with the default settings 2.emerge the package on a remote machine 3.restore the config files from your backups before your system falls apart Actual Results: Config files are overwritten without warning. Expected Results: Config files should be protected and only changed using dispatch-conf or etc-update.
>or, preferably, deprecate the program entirely I could live without it, but it can be useful for testing (I use it now and then for convenience in temporary testing chroots where I don't care) or last resort recovery (like using it to create a binpkg from a stage3 to save a system, albeit that is less useful now that we have official binpkgs unless it's an exotic stage3 w/o binpkgs). Either way I agree it should be discouraged as much as possible. It always been just a hack for convenience, and lucky when it happens to work for what you need it for. Maybe quickpkg itself could discourage its own use with a big eye catchy warning when you call it so don't have to spread warnings at every corners of the wiki/docs.
(In reply to Robert Sharp from comment #0) > Binary packages built with quickpkg using all default settings will > overwrite config files usually protected by emerge without any warning. The > resulting file contains the message "# empty file because --include-config=n > when `quickpkg` was used". I just tested how quickpkg works on my machine. If I install a package, then use quickpkg to make a binary package, change one or two config files, then immediately reinstall that package from the binary package, it seems to leave "intact" any protected config files that I have changed, but overwrites configuration files that are still in the "default" state with that message. Robert Sharp: is this how it is working for you, just overwriting config files that have not been changed by the user? Thankfully, if this is indeed how it works, it would not seem to actually destroy personal config data... but it's still likely not what a user would expect. > [...] add this as a strongly worded caveat [...] Agreed that it would seem useful to make the docs clearer on how this works and what it can be useful for, and maybe favor "emerge --buildpkgonly" etc. (In reply to Ionen Wolkens from comment #1) > Either way I agree it should be discouraged as much as possible. It always > been just a hack for convenience, and lucky when it happens to work for what > you need it for. > > Maybe quickpkg itself could discourage its own use with a big eye catchy > warning when you call it so don't have to spread warnings at every corners > of the wiki/docs. If there could be any sort of consensus on this, I'd be happy to try to make things clearer in the wiki at least (there actually aren't that many places quickpkg is mentioned). Currently we seem to be putting forward quickpkg as equal to "--buildpkgonly" on the wiki, this doesn't seem great if quickpkg will only be useful for some specific use-cases and might seem confusing if you don't know why it is doing it's thing. If what I wrote above is true, it would seem that the only way to get a binary package with quickpkg that could be used and distributed as a "normal" package (as if installed "from source") would be to use "--include-unmodified-config y" _and_ to do this on an installation for which none of the config files have been edited.
(In reply to ris from comment #2) > (In reply to Robert Sharp from comment #0) > > Binary packages built with quickpkg using all default settings will > > overwrite config files usually protected by emerge without any warning. The > > resulting file contains the message "# empty file because --include-config=n > > when `quickpkg` was used". > > I just tested how quickpkg works on my machine. If I install a package, then > use quickpkg to make a binary package, change one or two config files, then > immediately reinstall that package from the binary package, it seems to > leave "intact" any protected config files that I have changed, but > overwrites configuration files that are still in the "default" state with > that message. > > Robert Sharp: is this how it is working for you, just overwriting config > files that have not been changed by the user? > I'll give you an example of a config file that it silently changed causing havok and bewilderment: /etc/sandbox.conf. I have never touched this file. The result of blatting it with quickpkg's handy message was strange emerge failures. I grep'd /etc for the message and saw this file was one affected. I restored it from my backups and everything was fixed. There are plenty of other default config files that are needed and I expect would be protected if I had used --buildpkg.
(In reply to Robert Sharp from comment #3) > I'll give you an example of a config file that it silently changed causing > havok and bewilderment: /etc/sandbox.conf. I have never touched this file. > The result of blatting it with quickpkg's handy message was strange emerge > failures. I grep'd /etc for the message and saw this file was one affected. > I restored it from my backups and everything was fixed. There are plenty of > other default config files that are needed and I expect would be protected > if I had used --buildpkg. "emerge --buildpkgonly" etc. are definitely the go-to commands, yes. quickpkg can only be used for certain specific scenarios. I'll update the documentation on the wiki some time and maybe make a pull request for the man page per my last comment (after waiting to see if no one is opposed to this as a "fix").
Based on the discussed here, I have looked into changing https://wiki.gentoo.org/wiki/Binary_package_guide which IMO is the worse offending article for recommending quickpkg. I have relegated quickpkg to become the final option for users, rather than how it was before where as before it was highlighted as the "try this first option." You can see my changes at: https://wiki.gentoo.org/index.php?title=Binary_package_guide&type=revision&diff=1314131&oldid=1302357 I do not think quickpkg should be removed as it can be a very useful tool so I believe this solves all issues. I'm not closing this bug down as I think it's a doc bug personally and further discussion is required before we can safely mark this as closed.
The main issue seems to be that quickpkg will exclude config files even when the user has not modified them in anyway. quickpkg already has an option to include unmodified config files (--include-unmodified-config=y), but it is disabled by default. Perhaps we should enable it by default instead?
Created a PR to change the default for include-unmodified-config.
My vague memory was that this option was explained to me thusly: quickpkg was originally badly designed, as it included all configs. --include-config was a valid option, preserving existing behavior, but insecure as it would spill secrets, and also basically unwise to use as you'd end up with configs that don't know they aren't actually defaults or otherwise be unable to retrieve the correct defaults. *not* using --include-config would, conversely, break all your configs unconditionally. Having gotten into this mess, it becomes actually impossible to know whether an installed package with "unmodified" configs is actually unmodified, or simply vandalized by quickpkg (due to having been created before the options were added)... ... which in turn means that --include-unmodified-config cannot know that your configs are in fact unmodified, so enabling it is "technically" unsafe by default. Just like when it was originally added, changing it now is technically unsafe because packages built with --include-config=y will be wrong if you change the default. I think the only way to actually fix this would be for packages to know the pristine unmodified config irrespective of whether the one in /etc has been modified :D so that it can always be restored or loaded into a quickpkg. But changing the default as your patch does is also, I suspect, going to cause more good than harm. People already have to go out of their way to enable --include-config and in theory shouldn't be spilling secrets without knowing they did something wrong.
--include-config and --include-unmodified-config were added 10 years ago, both with a default value of 'n'. In the common case where we are running quickpkg for a package generated by emerge, quickpkg can definitely tell if the config has been modified and no security risk is possible. In the uncommon case where we are running quickpkg to generate a package based on files from another quickpkg-based package generated over 10 years ago, I suppose we might leak in a user-modified config. I really don't think we should be setting the default based on an unlikely scenario resulting from running quickpkg multiple times for the same files.
(In reply to Mike Gilbert from comment #9) > I really don't think we should be setting the default based on an unlikely > scenario resulting from running quickpkg multiple times for the same files. This seems like a reasonably argument, but I think we could consider it due diligence to add a warning about this scenario to the quickpkg man page, and a small note in the quickpkg --help message for -include-unmodified-config that refers to the corresponding warning in the man page.
I found the caveat documented in the quickpkg man page description: > The downside is that the package will contain the files that exist on yourfilesystem even if they have modified since they were first installed So, it would be nice to somehow reference this information in the --include-unmodified-config documentation.
I mostly use quickpkg through the --quickpkg-direct=y option, as this aids tremendously in generating a fresh rootfs from an existing system, without rebuilding everything beforehand. In my scenario, I like the current default of not including config files: It loudly and proudly generates a broken system, missing critical files for it to function. This puts me in a spot where I have to make an explicit choice: 1) acknowledge that some but not all of the files in question will be copied, depending on the state of my system, or 2) acknowledge that I'm fully copying my current system state. I would warn against making 1 the default, as this jeopardizes the natural process of discovery: A user will not notice that some files may not be included in the binpkg, and while at one point their scripts will work, they'll edit a config file, forget about it, and then wonder why their new binpkgs don't work. I susppose the above could be considered a minor concern due to the already present warning messages, I just really prefer having consistently wrong behavior by default instead of may-bite-you-in-the-ass-later. Anyways, none of this solves the real issue, as even with the changed default, when a package is copied to a new machine, unmodified versions of files will still be replaced by the dummy files contained in the package, if they were modified in the system that created the package.
I don't really understand the objection. If I recall correctly, quickpkg can only ever do two things: - include a dummy file containing a #-style comment stating that the file is "empty because --include-config=n was used" - include the current machine version of the file The proposal is to do option 2 instead of option 1 in more cases -- i.e. any case where the file hasn't been modified after installation. (In reply to Esteve Varela Colominas from comment #12) > In my scenario, I like the current default of not including config files: It > loudly and proudly generates a broken system, missing critical files for it > to function. > This puts me in a spot where I have to make an explicit choice: 1) > acknowledge that some but not all of the files in question will be copied, > depending on the state of my system, or 2) acknowledge that I'm fully > copying my current system state. > > I would warn against making 1 the default, as this jeopardizes the natural > process of discovery: A user will not notice that some files may not be > included in the binpkg, and while at one point their scripts will work, > they'll edit a config file, forget about it, and then wonder why their new > binpkgs don't work. Either you are missing critical files or your system works. I don't understand why you think it is helpful to produce erroneous output ***all the time*** as a "teaching aid" for the "natural process of discovery". If the user edits a config file and forget about it and then their binpkgs don't work, then they will naturally discover this at the time the error happens. If they don't edit the config files, things will work and there is nothing to discover. You're trying to punish people for not doing anything wrong, because you want to teach them that situations which don't affect them are wrong. > I susppose the above could be considered a minor concern due to the already > present warning messages, I just really prefer having consistently wrong > behavior by default instead of may-bite-you-in-the-ass-later. This is total nonsense. The current behavior is *both* consistently wrong, *and* bites you later. We are going from "bites you later" to "bites you less often". Having wrong behavior is... wrong. We should refrain from having wrong behavior. Arguing that one should have wrong behavior because it allows you to act wise at users and explain how you don't like the functionality to begin with, is a silly reason to have wrong behavior. But yes, also the warning messages do make this less of a concern. We could go even further, and make it error out by default instead of making broken binpkgs at all. > Anyways, none of this solves the real issue, as even with the changed > default, when a package is copied to a new machine, unmodified versions of > files will still be replaced by the dummy files contained in the package, if > they were modified in the system that created the package. Fortunately nobody said it is solved, just made a lot harder to hit. Fully solving this problem cannot be done by quickpkg as there is no specification for how to store a binary package and mark some of its installed CONTENTS as differing from the installed files, nor save the unmodified version for quickpkg'ing. You'd need to define that somewhere higher up...
Hey, I didn't mean to piss anyone off, and I'm really sorry about that. I also really didn't mean to imply I'm out to "act wise" at users or that I don't like quickpkg at all. My concern is a genuine one, although it's completely fine with me if you dismiss it. I only meant to add a different perspective to the discussion. I tend to cover my arguments in too many words, so to sum it up: I think the new behavior is still wrong, and I believe making it default helps to obfuscate that behind system state, instead of making the user aware of and acknowledge this dependence on system state. (In reply to Eli Schwartz from comment #13) > If the user edits a config file and forget about it and then > their binpkgs don't work, then they will naturally discover this at the time > the error happens. It's really easy to not notice the broken configs. Some programs will silently ignore them and just have different behavior than their USE flags imply they would, and it's hard to track this down. Having this wrongness be consistent means the user will find out find out the *first* time they test it, instead of the behavior changing down the line. (In reply to Eli Schwartz from comment #13) > We could > go even further, and make it error out by default instead of making broken > binpkgs at all. Honestly, great idea! This would solve my concerns: It makes the behavior discoverable and obvious, and allows the user to make an informed decision on how to solve it.
