Kernal tarballs and patches have been available as .xz for a while now. These are smaller than .bz2 and take much less time to decompress. An alternative would be to provide a use-flag to pick between xz, gz, and bz2 tarballs. Reproducible: Always
This is interesting. My desktop take 16 seconds to decompress the tar.bz2 archive and only 5 to decompress the same archive in .tar.xz format.
(In reply to comment #1) > This is interesting. My desktop take 16 seconds to decompress the tar.bz2 > archive and only 5 to decompress the same archive in .tar.xz format. that a main feature of xz. Longer compression times but much shorter decompression times compared to bz2.
+ 09 Mar 2013; Tom Wijsman <TomWij@gentoo.org> + eclass/kernel-2.eclass, sys-kernel/*-sources-*.ebuild: + EAPI 5. Kernel sources and (gen)patches now use xz instead of bz2. + Bug #421721. All ebuilds that use the fetch / unpacking logic of kernel-2 have been tested.
Not all: Timestamp of tree: Sun, 10 Mar 2013 08:15:01 +0000 emerge -1vf sys-kernel/hardened-sources These are the packages that would be fetched, in order: Calculating dependencies... done! [ebuild NS ~] sys-kernel/hardened-sources-3.8.2-r1:3.8.2-r1 [3.8.2:3.8.2] USE="symlink -build -deblob" 0 kB Total: 1 package (1 in new slot), Size of downloads: 0 kB >>> Fetching (1 of 1) sys-kernel/hardened-sources-3.8.2-r1 !!! Fetched file: linux-3.8.tar.xz VERIFY FAILED! !!! Reason: Insufficient data for checksum verification !!! Got: !!! Expected: MD5 RMD160 SHA1 SHA256 SHA512 WHIRLPOOL * hardened-patches-3.8.2-2.extras.tar.bz2 SHA256 SHA512 WHIRLPOOL size ;-) ... [ ok ] !!! Fetched file: genpatches-3.8-3.base.tar.xz VERIFY FAILED! !!! Reason: Insufficient data for checksum verification !!! Got: !!! Expected: MD5 RMD160 SHA1 SHA256 SHA512 WHIRLPOOL * Fetch failed for 'sys-kernel/hardened-sources-3.8.2-r1' >>> Failed to emerge sys-kernel/hardened-sources-3.8.2-r1
fixed
Thanks for breaking all the out-of-tree ebuilds!
So, how am I supposed to keep using my .tar.bz2 sources now that tar.xz got hardcoded into kernel-2.eclass ?
All my ebuilds (those in the "sabayon-distro" overlay) are now broken because I don't use .tar.xz for my sources and .tar.xz is now hardcoded pretty much everywhere in kernel-2.eclass.
Please de-hardcode the tarball extension and move it to an eclass variable or I will request QA to revert the commit.
(In reply to comment #9) > Please de-hardcode the tarball extension and move it to an eclass variable > or I will request QA to revert the commit. that's bs (and job for the overlays maintainers to maintain compatible set of ebuilds and eclasses)
Sure, let's break everything and just don't care about communicating it to the rest of the world. This is the typical Gentoo developers' attitude and I am quite sick of it.
This is not just about fixing some code, this is about regenerating tarballs of all the kernel sources you see here: http://distfiles.sabayon.org/sys-kernel/ And I am sure other people are using kernel-2 eclass as I do.
http://www.gentoo.org/proj/en/qa/ 1. Project Description The Gentoo Quality Assurance Project aims to keep the portage tree in a consistent state. [...] Removing QA from CC. Feel free to re-add if there are inconsistencies in *the*portage*tree* caused by this.
I think adding variable for using in ebuilds is good idea. Fabio, may be you can provide patch for eclass?
Sorry to bug qa@ one last time. Can you clarify this quote from the devmanual: "When removing a function or changing the API of an eclass, make sure that it doesn't break any ebuilds in the tree, and post a notice to gentoo-dev at least 30 days in advance, preferably with a patch included." In particular the part saying ", and post a notice to gentoo-dev at least 30 days in advance, preferably with a patch included.". To my understanding, devs here should have posted a message to gentoo-dev, which I couldn't find.
Created attachment 342632 [details, diff] de-hardcode the tarball extension This however does not restore the original default behaviour, which should have been the case.
(In reply to comment #6) > Thanks for breaking all the out-of-tree ebuilds! If they depend on an eclass in the Portage tree, they need to adapt to it. Until now, I never heard about the existence of your kernel ebuilds; so, how can I not break them? (In reply to comment #7) > So, how am I supposed to keep using my .tar.bz2 sources now that tar.xz got > hardcoded into kernel-2.eclass ? Which sources exactly? If they are the upstream kernel sources, patches and our genpatches patches, I see no reason to use bz2, you can just use xz there. If they are your own sources, nothing changed in that light, you can still use them in that case. If not, please clearly state what and why; I can't reproduce this. (In reply to comment #8) > All my ebuilds (those in the "sabayon-distro" overlay) are now broken > because I don't use .tar.xz for my sources and .tar.xz is now hardcoded > pretty much everywhere in kernel-2.eclass. Switch to EAPI 5, remove Manifest, re-manifest; tell me those that still break. (In reply to comment #9) > Please de-hardcode the tarball extension and move it to an eclass variable > or I will request QA to revert the commit. Please state a clear reason why this needs to be hardcoded. Reverting the commit for an overlay that could just adapt doesn't look right; we're not talking about a single commit here, but as well as a full remanifest of the sources ebuilds and more... (In reply to comment #11) > Sure, let's break everything and just don't care about communicating it to > the rest of the world. > This is the typical Gentoo developers' attitude and I am quite sick of it. You can filter changes of the eclasses you use on the gentoo-commits ML, you can do the same for dependencies which your packages depend on; for both their is no explicit communication policy in general, unless your eclass is used on a much larger scale than the eclass that is concerned here. There is no easy way for me to reach every out-of-tree non Gentoo Developer maintainer; and no, I'm not going to use my free time to look them all up. (In reply to comment #12) > This is not just about fixing some code, this is about regenerating tarballs > of all the kernel sources you see here: > http://distfiles.sabayon.org/sys-kernel/ > And I am sure other people are using kernel-2 eclass as I do. The majority has already been done as part of the change in the Portage tree. Can you please give me a list of remaining tarballs that still need to be converted from bz2 to xz, uploaded to our development box and mirrored? (In reply to comment #15) > Sorry to bug qa@ one last time. > Can you clarify this quote from the devmanual: QA has nothing to do here for the reason stated in their earlier comment and even as listed in the quote you just gave (quoted below, capitalized), removing them from CC. > "When removing a function I did not remove a function. > or changing the API of an eclass, The API has not been changed. > make sure that it doesn't break any ebuilds in the tree, It did not break any ebuilds IN THE TREE, this has been carefully prepared. > and post a notice to gentoo-dev at least 30 days in advance, preferably with a patch included." To my understanding, devs here should have posted a message to gentoo-dev, which I couldn't find. This action is not required since those previous conditions were not met; eclass is barely used anyway. (In reply to comment #16) > Created attachment 342632 [details, diff] [details, diff] > de-hardcode the tarball extension > > This however does not restore the original default behaviour, which should > have been the case. Which problem is this trying to solve? Why not fork the eclass?
I am not willing to waste my free time trying to teach you how you, as developer and maybe software engineer, are supposed to deal with code. If you don't get it, I am sorry about you. I will bring this bug to the attention of the council during the next meeting to discuss about policy changes that will avoid these things to happen in future. You've once again proved that you were not interested in migrating to .tar.xz in a reliable way for eclass users. Next time, just don't do it. You are also not supposed to un-CC qa@ on their behalf, unless you are a member of qa@
(In reply to comment #18) > I am not willing to waste my free time trying to teach you how you, as developer and maybe software engineer, are supposed to deal with code. If you don't get it, I am sorry about you. You are not willing to state the problem you are trying to solve; there is nothing here for me to understand anyway, you haven't even tried to teach me anything. Ad hominem. > I will bring this bug to the attention of the council during the next > meeting to discuss about policy changes that will avoid these things to > happen in future. Which problem? Which policy changes? Which things? I'll try to be present. > You've once again proved that you were not interested in migrating to > .tar.xz in a reliable way for eclass users. Next time, just don't do it. This has been done under the permission of mpagano, which has been committing to this ebuild for the last four years. > You are also not supposed to un-CC qa@ on their behalf, unless you are a > member of qa@ Sorry, but we don't add random herds to bugs for no reason. As the wrangler of this bug, I see no purpose for them to be here; there isn't a problem they can address here at all. They have been informed by the un-CC; but well, I'll let them explain their irrelevance to this bug to you again.
The problem is crystal clear, you broke the ebuilds of all the out-of-tree kernel-2.eclass users, without proper communication and without documenting it, without caring about the consequences, wasting other developers time to try to figure out how to fix them. In my little case, you just broke all my 97 tarballs that I have to re-compress using xz for no absolute reason. This will take ~12 hours or even more on the hardware on where they are stored. The eclass change did not go through gentoo-dev ML review (it could have helped) and it shows. It could have been really easy to just expose the compression format through a simple eclass variable and make everybody happy. If you still don't get what's wrong in all this, I am really, really sorry about that. I am going to look for a reviewer of my patch and hopefully commit it to CVS.
(In reply to comment #20) > The problem is crystal clear, you broke the ebuilds of all the out-of-tree > kernel-2.eclass users, You are the only one complaining after more than a week, I doubt it... I have no access to yours and therefore can not prevent them from being broken. > without proper communication and without documenting it, without caring about the consequences, This has been properly documented in the commit message which is there for this purpose, they are there for a purpose; if you don't read them, then you get to live with the consequences. I have no other reasonable way to contact out-of-tree users... > wasting other developers time to try to figure out how to fix them. It is a logical consequence from reading the message that they need to be converted from bz2 to xz; if not clear to you, you could have just asked us. As you started with "Thanks for breaking ..." right away, it is clear that you haven't considered doing either of that. You have chosen to waste your time. > In my little case, you just broke all my 97 tarballs that I have to re-compress using xz for no absolute reason. The reason is stated in this bug, the bug has been referenced from the commit message; it just sounds like you didn't read it. I have done the same for tarballs in the Portage tree; if you use your own tarballs, you need to re-compress them yourself. You can't expect me to do that. > This will take ~12 hours or even more on the hardware on where they are stored. Once in a lifetime flood. You should realize that xz is a benefit for your hardware. > The eclass change did not go through gentoo-dev ML review (it could have > helped) and it shows. That mailing list does not target out-of-tree users; for changes inside the Portage tree, the ML was clearly an unnecessary step. > It could have been really easy to just expose the compression format through a simple eclass variable and make everybody happy. It could have been really easy to state the actual practical problem which makes such proposed changes make sense; if you don't state it, there has been no discussion about it and there is therefore no reason to add that change. Just committing a variable for a sole user that requests it but doesn't state what and why, isn't really an approach to collaborative development that benefits the majority; at least not from a Software Engineering point of view. That said, I'm not necessarily for or against adding it; it just misses context. > If you still don't get what's wrong in all this, I am really, really sorry > about that. Ad hominem. > I am going to look for a reviewer of my patch and hopefully commit it to CVS. Side stepping to get a patch committed that benefits nobody but you; there is no reason to change from xz as far as I know, if there is, please tell us so we can adapt this unknown unknown as well. Alongside that the Sabayon overlay has 1) reuploaded them as xz, 2) forked the eclass which you can change freely (sabayon-kernel.eclass) and 3) the ebuilds should work after the overlay's changes; so, once again, I don't see the actual practical problem which you are experiencing.
Ah, I found the problem you are experiencing: In the future, please do not complain if you are changing environment variables like KERNEL_BASE_URI which are not documented as those that you can change; because you are not using the eclass in a legit way, it doesn't surprise me that it broke for you. You're on your own.
(In reply to comment #21) > You are the only one complaining after more than a week, I doubt it... He is not the only one: I was using tar.bz2 genpatches from ~mpagano for an old kernel version in Liberté Linux. Since I am not compelled to recompress and host these files elsewhere, the build will remain broken until I migrate to a newer kernel version sometime in the future. I will save you the effort of writing another point-by-point rebuttal by readily admitting that yes, your lack of adherence to proper software engineering practices is solely my fault.
(In reply to comment #23) > I will save you the effort of writing another point-by-point rebuttal by > readily admitting that yes, /my/ lack of adherence to proper software > engineering practices is solely my fault. Corrected it for you... It's not my fault if you change undocumented variables and ignore commit logs.
Created attachment 345372 [details] Warn users that we don't support unsupported features. Sorry, we'll mention this more explicitly, see attached patch. We also can't de-hardcode the tarball extension using the attached patch since genpatches are only available with the xz extension and would therefore result in requesting files that are not present, the Gentoo mirrors are also not meant to mirror files for usage outside of the Portage tree. Even though there is no requirement to do so; any changes made by me to this eclass will frow now on be sent to both the Gentoo Developer and Gentoo Kernel mailing list, you are suggested to subscribe to them in order to avoid being unaware of future changes. Till this point the ChangeLog was the main place for this, you can see that at http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/eclass/kernel-2.eclass?view=log where you will see sys-devel/bc has been added as a dependency as well as changing "root" to "0" to be more conform with Linux standards. Above patch will be part of a series of patches and will be sent to the above mentioned mailing lists and committed at once to the tree in the near future.
17 Apr 2013; Tom Wijsman <TomWij@gentoo.org> kernel-2.eclass: Added a warning after the variables that modifying other variables in the eclass is not supported, there is a chance that we will not fix resulting bugs. Fixes bug #421721. Clarify the default DESCRIPTION and make it not use versions, a directory with ebuilds that inherit this eclass may contain multiple versions and we also don't want to give the impression that a new directory needs to made if that's not the case. Fixes bug #445110. Clarified which patch depths are used in the normal output and error output when applying patches. Fixes bug #436402. Made sure .tmp_gas_check is created inside the temp folder, it accidentally created temp.tmp_gas_check instead. Fixes bug #336732. Make UNIPATCH_DOCS work again, install 0000_README document when using genpatches. Fixes bug #301478.
