Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 421721 - sys-kernel/*-sources - Use xz instead of bz2 compressed kernel tarballs and patches
Summary: sys-kernel/*-sources - Use xz instead of bz2 compressed kernel tarballs and p...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: Normal major (vote)
Assignee: Gentoo Kernel Miscellaneous
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-06-18 09:08 UTC by Daniel Bumke
Modified: 2013-05-13 16:28 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
de-hardcode the tarball extension (k_tarball_ext.patch,8.04 KB, patch)
2013-03-19 11:57 UTC, Fabio Erculiani (RETIRED)
Details | Diff
Warn users that we don't support unsupported features. (file_421721.txt,652 bytes, text/plain)
2013-04-12 13:24 UTC, Tom Wijsman (TomWij) (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Bumke 2012-06-18 09:08:39 UTC
Kernal tarballs and patches have been available as .xz for a while now. These are smaller than .bz2 and take much less time to decompress.

An alternative would be to provide a use-flag to pick between xz, gz, and bz2 tarballs.

Reproducible: Always
Comment 1 Agostino Sarubbo gentoo-dev 2013-03-09 13:07:29 UTC
This is interesting. My desktop take 16 seconds to decompress the tar.bz2 archive and only 5 to decompress the same archive in .tar.xz format.
Comment 2 Justin Lecher (RETIRED) gentoo-dev 2013-03-09 13:11:51 UTC
(In reply to comment #1)
> This is interesting. My desktop take 16 seconds to decompress the tar.bz2
> archive and only 5 to decompress the same archive in .tar.xz format.

that a main feature of xz. Longer compression times but much shorter decompression times compared to bz2.
Comment 3 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-03-09 21:12:00 UTC
+  09 Mar 2013; Tom Wijsman <TomWij@gentoo.org>
+  eclass/kernel-2.eclass, sys-kernel/*-sources-*.ebuild:
+  EAPI 5. Kernel sources and (gen)patches now use xz instead of bz2.
+  Bug #421721.

All ebuilds that use the fetch / unpacking logic of kernel-2 have been tested.
Comment 4 Nikoli 2013-03-10 08:31:17 UTC
Not all:
Timestamp of tree: Sun, 10 Mar 2013 08:15:01 +0000

 emerge -1vf sys-kernel/hardened-sources

These are the packages that would be fetched, in order:

Calculating dependencies... done!
[ebuild  NS   ~] sys-kernel/hardened-sources-3.8.2-r1:3.8.2-r1 [3.8.2:3.8.2] USE="symlink -build -deblob" 0 kB

Total: 1 package (1 in new slot), Size of downloads: 0 kB


>>> Fetching (1 of 1) sys-kernel/hardened-sources-3.8.2-r1
!!! Fetched file: linux-3.8.tar.xz VERIFY FAILED!
!!! Reason: Insufficient data for checksum verification
!!! Got:      
!!! Expected: MD5 RMD160 SHA1 SHA256 SHA512 WHIRLPOOL
 * hardened-patches-3.8.2-2.extras.tar.bz2 SHA256 SHA512 WHIRLPOOL size ;-) ...                                                                                                             [ ok ]
!!! Fetched file: genpatches-3.8-3.base.tar.xz VERIFY FAILED!
!!! Reason: Insufficient data for checksum verification
!!! Got:      
!!! Expected: MD5 RMD160 SHA1 SHA256 SHA512 WHIRLPOOL
 * Fetch failed for 'sys-kernel/hardened-sources-3.8.2-r1'

>>> Failed to emerge sys-kernel/hardened-sources-3.8.2-r1
Comment 5 Matthew Thode ( prometheanfire ) archtester Gentoo Infrastructure gentoo-dev Security 2013-03-10 08:55:28 UTC
fixed
Comment 6 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 08:25:11 UTC
Thanks for breaking all the out-of-tree ebuilds!
Comment 7 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 08:26:13 UTC
So, how am I supposed to keep using my .tar.bz2 sources now that tar.xz got hardcoded into kernel-2.eclass ?
Comment 8 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 08:28:15 UTC
All my ebuilds (those in the "sabayon-distro" overlay) are now broken because I don't use .tar.xz for my sources and .tar.xz is now hardcoded pretty much everywhere in kernel-2.eclass.
Comment 9 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 08:37:49 UTC
Please de-hardcode the tarball extension and move it to an eclass variable or I will request QA to revert the commit.
Comment 10 Samuli Suominen gentoo-dev 2013-03-19 08:46:00 UTC
(In reply to comment #9)
> Please de-hardcode the tarball extension and move it to an eclass variable
> or I will request QA to revert the commit.

that's bs (and job for the overlays maintainers to maintain compatible set of ebuilds and eclasses)
Comment 11 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 08:48:40 UTC
Sure, let's break everything and just don't care about communicating it to the rest of the world.
This is the typical Gentoo developers' attitude and I am quite sick of it.
Comment 12 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 08:53:18 UTC
This is not just about fixing some code, this is about regenerating tarballs of all the kernel sources you see here: http://distfiles.sabayon.org/sys-kernel/
And I am sure other people are using kernel-2 eclass as I do.
Comment 13 Ulrich Müller gentoo-dev 2013-03-19 09:51:00 UTC
http://www.gentoo.org/proj/en/qa/

   1.  Project Description
   The Gentoo Quality Assurance Project aims to keep the portage tree
   in a consistent state. [...]

Removing QA from CC. Feel free to re-add if there are inconsistencies in *the*portage*tree* caused by this.
Comment 14 Nikoli 2013-03-19 11:32:47 UTC
I think adding variable for using in ebuilds is good idea. Fabio, may be you can provide patch for eclass?
Comment 15 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 11:35:51 UTC
Sorry to bug qa@ one last time.
Can you clarify this quote from the devmanual:

"When removing a function or changing the API of an eclass, make sure that it doesn't break any ebuilds in the tree, and post a notice to gentoo-dev at least 30 days in advance, preferably with a patch included."

In particular the part saying ", and post a notice to gentoo-dev at least 30 days in advance, preferably with a patch included.".
To my understanding, devs here should have posted a message to gentoo-dev, which I couldn't find.
Comment 16 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 11:57:32 UTC
Created attachment 342632 [details, diff]
de-hardcode the tarball extension

This however does not restore the original default behaviour, which should have been the case.
Comment 17 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-03-19 16:18:59 UTC
(In reply to comment #6)
> Thanks for breaking all the out-of-tree ebuilds!

If they depend on an eclass in the Portage tree, they need to adapt to it. Until now, I never heard about the existence of your kernel ebuilds; so, how can I not break them?

(In reply to comment #7)
> So, how am I supposed to keep using my .tar.bz2 sources now that tar.xz got
> hardcoded into kernel-2.eclass ?

Which sources exactly? If they are the upstream kernel sources, patches and our genpatches patches, I see no reason to use bz2, you can just use xz there. If they are your own sources, nothing changed in that light, you can still use them in that case. If not, please clearly state what and why; I can't reproduce this.

(In reply to comment #8)
> All my ebuilds (those in the "sabayon-distro" overlay) are now broken
> because I don't use .tar.xz for my sources and .tar.xz is now hardcoded
> pretty much everywhere in kernel-2.eclass.

Switch to EAPI 5, remove Manifest, re-manifest; tell me those that still break.

(In reply to comment #9)
> Please de-hardcode the tarball extension and move it to an eclass variable
> or I will request QA to revert the commit.

Please state a clear reason why this needs to be hardcoded.

Reverting the commit for an overlay that could just adapt doesn't look right; we're not talking about a single commit here, but as well as a full remanifest of the sources ebuilds and more...

(In reply to comment #11)
> Sure, let's break everything and just don't care about communicating it to
> the rest of the world.
> This is the typical Gentoo developers' attitude and I am quite sick of it.

You can filter changes of the eclasses you use on the gentoo-commits ML, you can do the same for dependencies which your packages depend on; for both their is no explicit communication policy in general, unless your eclass is used on a much larger scale than the eclass that is concerned here. There is no easy way for me to reach every out-of-tree non Gentoo Developer maintainer; and no, I'm not going to use my free time to look them all up.

(In reply to comment #12)
> This is not just about fixing some code, this is about regenerating tarballs
> of all the kernel sources you see here:
> http://distfiles.sabayon.org/sys-kernel/
> And I am sure other people are using kernel-2 eclass as I do.

The majority has already been done as part of the change in the Portage tree.

Can you please give me a list of remaining tarballs that still need to be converted from bz2 to xz, uploaded to our development box and mirrored?

(In reply to comment #15)
> Sorry to bug qa@ one last time.
> Can you clarify this quote from the devmanual:

QA has nothing to do here for the reason stated in their earlier comment and even as listed in the quote you just gave (quoted below, capitalized), removing them from CC.

> "When removing a function

I did not remove a function.

> or changing the API of an eclass,

The API has not been changed.

> make sure that it doesn't break any ebuilds in the tree,

It did not break any ebuilds IN THE TREE, this has been carefully prepared.

> and post a notice to gentoo-dev at least 30 days in advance, preferably with a patch included." To my understanding, devs here should have posted a message to gentoo-dev, which I couldn't find.

This action is not required since those previous conditions were not met; eclass is barely used anyway.

(In reply to comment #16)
> Created attachment 342632 [details, diff] [details, diff]
> de-hardcode the tarball extension
> 
> This however does not restore the original default behaviour, which should
> have been the case.

Which problem is this trying to solve? Why not fork the eclass?
Comment 18 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 16:53:14 UTC
I am not willing to waste my free time trying to teach you how you, as developer and maybe software engineer, are supposed to deal with code. If you don't get it, I am sorry about you.
I will bring this bug to the attention of the council during the next meeting to discuss about policy changes that will avoid these things to happen in future.

You've once again proved that you were not interested in migrating to .tar.xz in a reliable way for eclass users. Next time, just don't do it.

You are also not supposed to un-CC qa@ on their behalf, unless you are a member of qa@
Comment 19 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-03-19 17:10:44 UTC
(In reply to comment #18)
> I am not willing to waste my free time trying to teach you how you, as developer and maybe software engineer, are supposed to deal with code. If you don't get it, I am sorry about you.

You are not willing to state the problem you are trying to solve; there is nothing here for me to understand anyway, you haven't even tried to teach me anything. Ad hominem.

> I will bring this bug to the attention of the council during the next
> meeting to discuss about policy changes that will avoid these things to
> happen in future.

Which problem? Which policy changes? Which things? I'll try to be present.

> You've once again proved that you were not interested in migrating to
> .tar.xz in a reliable way for eclass users. Next time, just don't do it.

This has been done under the permission of mpagano, which has been committing to this ebuild for the last four years.

> You are also not supposed to un-CC qa@ on their behalf, unless you are a
> member of qa@

Sorry, but we don't add random herds to bugs for no reason. As the wrangler of this bug, I see no purpose for them to be here; there isn't a problem they can address here at all. They have been informed by the un-CC; but well, I'll let them explain their irrelevance to this bug to you again.
Comment 20 Fabio Erculiani (RETIRED) gentoo-dev 2013-03-19 17:20:48 UTC
The problem is crystal clear, you broke the ebuilds of all the out-of-tree kernel-2.eclass users, without proper communication and without documenting it, without caring about the consequences, wasting other developers time to try to figure out how to fix them. In my little case, you just broke all my 97 tarballs that I have to re-compress using xz for no absolute reason. This will take ~12 hours or even more on the hardware on where they are stored. The eclass change did not go through gentoo-dev ML review (it could have helped) and it shows. It could have been really easy to just expose the compression format through a simple eclass variable and make everybody happy.
If you still don't get what's wrong in all this, I am really, really sorry about that.

I am going to look for a reviewer of my patch and hopefully commit it to CVS.
Comment 21 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-03-19 17:54:35 UTC
(In reply to comment #20)
> The problem is crystal clear, you broke the ebuilds of all the out-of-tree
> kernel-2.eclass users,

You are the only one complaining after more than a week, I doubt it...

I have no access to yours and therefore can not prevent them from being broken.

> without proper communication and without documenting it, without caring about the consequences, 

This has been properly documented in the commit message which is there for this purpose, they are there for a purpose; if you don't read them, then you get to live with the consequences. I have no other reasonable way to contact out-of-tree users...

> wasting other developers time to try to figure out how to fix them.

It is a logical consequence from reading the message that they need to be converted from bz2 to xz; if not clear to you, you could have just asked us. As you started with "Thanks for breaking ..." right away, it is clear that you haven't considered doing either of that. You have chosen to waste your time.

> In my little case, you just broke all my 97 tarballs that I have to re-compress using xz for no absolute reason.

The reason is stated in this bug, the bug has been referenced from the commit message; it just sounds like you didn't read it. I have done the same for tarballs in the Portage tree; if you use your own tarballs, you need to re-compress them yourself. You can't expect me to do that.

> This will take ~12 hours or even more on the hardware on where they are stored.

Once in a lifetime flood.

You should realize that xz is a benefit for your hardware.

> The eclass change did not go through gentoo-dev ML review (it could have
> helped) and it shows.

That mailing list does not target out-of-tree users; for changes inside the Portage tree, the ML was clearly an unnecessary step.

> It could have been really easy to just expose the compression format through a simple eclass variable and make everybody happy.

It could have been really easy to state the actual practical problem which makes such proposed changes make sense; if you don't state it, there has been no discussion about it and there is therefore no reason to add that change.

Just committing a variable for a sole user that requests it but doesn't state what and why, isn't really an approach to collaborative development that benefits the majority; at least not from a Software Engineering point of view.

That said, I'm not necessarily for or against adding it; it just misses context.

> If you still don't get what's wrong in all this, I am really, really sorry
> about that.

Ad hominem.

> I am going to look for a reviewer of my patch and hopefully commit it to CVS.

Side stepping to get a patch committed that benefits nobody but you; there is no reason to change from xz as far as I know, if there is, please tell us so we can adapt this unknown unknown as well.

Alongside that the Sabayon overlay has 1) reuploaded them as xz, 2) forked the eclass which you can change freely (sabayon-kernel.eclass) and 3) the ebuilds should work after the overlay's changes; so, once again, I don't see the actual practical problem which you are experiencing.
Comment 22 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-03-20 07:24:46 UTC
Ah, I found the problem you are experiencing: In the future, please do not complain if you are changing environment variables like KERNEL_BASE_URI which are not documented as those that you can change; because you are not using the eclass in a legit way, it doesn't surprise me that it broke for you. You're on your own.
Comment 23 Maxim Kammerer 2013-04-12 10:44:12 UTC
(In reply to comment #21)
> You are the only one complaining after more than a week, I doubt it...

He is not the only one: I was using tar.bz2 genpatches from ~mpagano for an old kernel version in Liberté Linux. Since I am not compelled to recompress and host these files elsewhere, the build will remain broken until I migrate to a newer kernel version sometime in the future.

I will save you the effort of writing another point-by-point rebuttal by readily admitting that yes, your lack of adherence to proper software engineering practices is solely my fault.
Comment 24 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-04-12 11:09:05 UTC
(In reply to comment #23)
> I will save you the effort of writing another point-by-point rebuttal by
> readily admitting that yes, /my/ lack of adherence to proper software
> engineering practices is solely my fault.

Corrected it for you...

It's not my fault if you change undocumented variables and ignore commit logs.
Comment 25 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-04-12 13:24:16 UTC
Created attachment 345372 [details]
Warn users that we don't support unsupported features.

Sorry, we'll mention this more explicitly, see attached patch.

We also can't de-hardcode the tarball extension using the attached patch since genpatches are only available with the xz extension and would therefore result in requesting files that are not present, the Gentoo mirrors are also not meant to mirror files for usage outside of the Portage tree.

Even though there is no requirement to do so; any changes made by me to this eclass will frow now on be sent to both the Gentoo Developer and Gentoo Kernel mailing list, you are suggested to subscribe to them in order to avoid being unaware of future changes. Till this point the ChangeLog was the main place for this, you can see that at http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/eclass/kernel-2.eclass?view=log where you will see sys-devel/bc has been added as a dependency as well as changing "root" to "0" to be more conform with Linux standards.

Above patch will be part of a series of patches and will be sent to the above mentioned mailing lists and committed at once to the tree in the near future.
Comment 26 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2013-04-17 13:53:03 UTC
  17 Apr 2013; Tom Wijsman <TomWij@gentoo.org> kernel-2.eclass:
  Added a warning after the variables that modifying other variables in
  the eclass is not supported, there is a chance that we will not fix
  resulting bugs. Fixes bug #421721.
  Clarify the default DESCRIPTION and make it not use versions, a
  directory with ebuilds that inherit this eclass may contain multiple
  versions and we also don't want to give the impression that a new
  directory needs to made if that's not the case. Fixes bug #445110.
  Clarified which patch depths are used in the normal output and error
  output when applying patches. Fixes bug #436402.
  Made sure .tmp_gas_check is created inside the temp folder, it
  accidentally created temp.tmp_gas_check instead. Fixes bug #336732.
  Make UNIPATCH_DOCS work again, install 0000_README document when
  using genpatches. Fixes bug #301478.