I have a savedconfig with about a dozen entries. The log produces (acknowledging the savedconfig): "Building using saved configfile "/etc/portage/savedconfig/sys-kernel/linux-firmware" Yet the ebuild insists on copying/compressing all ~2800 files in the archive, creating ~575 symbolic links, and THEN removing all of the files not listed in the savedconfig. The emerge takes over 50 minutes to complete on this machine - a ridiculous amount of time to essentially extract about dozen files from an archive. Does some way exist to remove the unnecessary files BEFORE copying/compressing and linking? That would save a huge amount of time.
It looks like it's safe to move the deleting the unwanted firmware to src_prepare as far as I can tell. I think we should do that. I don't understand all the of savedconfig related calls though, so for now I'm leaving this to someone who does. that said the `./copy-firmware.sh` call at the start of src_install will behave exactly as mentioned in this bug, and that seems very expensive for anyone with savedconfig turned on.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5cc19234cf0cf7b82b5902b870648aedcd8420ae commit 5cc19234cf0cf7b82b5902b870648aedcd8420ae Author: Mike Pagano <mpagano@gentoo.org> AuthorDate: 2024-01-20 19:56:38 +0000 Commit: Mike Pagano <mpagano@gentoo.org> CommitDate: 2024-01-20 19:57:28 +0000 package.mask: Mask linux-firmware-20250115-r1 for testing Modifications to USE=savedconfig to only install files from there instead of deleting them afterwards. Bug: https://bugs.gentoo.org/913049 Signed-off-by: Mike Pagano <mpagano@gentoo.org> profiles/package.mask | 7 +++++++ 1 file changed, 7 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=df7daea6e021ab74a4ff7d0c662e428b5cf2b995 commit df7daea6e021ab74a4ff7d0c662e428b5cf2b995 Author: Mike Pagano <mpagano@gentoo.org> AuthorDate: 2024-01-20 20:00:08 +0000 Commit: Mike Pagano <mpagano@gentoo.org> CommitDate: 2024-01-20 20:01:25 +0000 sys-kernel/linux-firmware: Modifications to USE=savedconfig Modified to only install files from the config file instead of deleting them afterwards. Bug: https://bugs.gentoo.org/913049 Signed-off-by: Mike Pagano <mpagano@gentoo.org> .../files/linux-firmware-copy-firmware.patch | 59 +++ .../linux-firmware-20240115-r1.ebuild | 403 +++++++++++++++++++++ 2 files changed, 462 insertions(+)
It's masked for now, but if someone can test linux-firmware-20240115-r1 and let me know of any disasters that would be helpful.
Am I the only one who tested this ?
Looks like your commit made a massive difference. Nice work. Works great here.
(In reply to Matt Turner from comment #6) > Looks like your commit made a massive difference. Nice work. Works great > here. Thanks, Matt.
Works a treat on my headless systems that only need amd-ucode. It brings the merge time down by 6x! host01 /etc/portage # qlop sys-kernel/linux-firmware 2024-02-03T11:14:14 >>> sys-kernel/linux-firmware: 1′11″ 2024-02-03T11:18:40 >>> sys-kernel/linux-firmware: 1′01″ 2024-02-04T03:26:23 >>> sys-kernel/linux-firmware: 10s However, I noticed on the system where I use +libarchive for app-alternatives/cpio, app-arch/cpio is now being pulled in, whereas before it depended on app-alternatives/cpio: host02 /etc/portage # emerge -avu linux-firmware These are the packages that would be merged, in order: Calculating dependencies ... done! Dependency resolution took 1.03 s (backtrack: 0/20). [ebuild N ] app-arch/cpio-2.14::gentoo USE="-nls" 1486 KiB [ebuild U ~] sys-kernel/linux-firmware-20240115-r1::gentoo [20240115::gentoo] USE="initramfs redistributable savedconfig -compress-xz -compress-zstd -deduplicate (-unknown-license)" 0 KiB Total: 2 packages (1 upgrade, 1 new), Size of downloads: 1486 KiB Is there a reason for this dependency change?
(In reply to A. Wilcox (awilfox) from comment #8) > Works a treat on my headless systems that only need amd-ucode. It brings > the merge time down by 6x! > > host01 /etc/portage # qlop sys-kernel/linux-firmware > 2024-02-03T11:14:14 >>> sys-kernel/linux-firmware: 1′11″ > 2024-02-03T11:18:40 >>> sys-kernel/linux-firmware: 1′01″ > 2024-02-04T03:26:23 >>> sys-kernel/linux-firmware: 10s > > However, I noticed on the system where I use +libarchive for > app-alternatives/cpio, app-arch/cpio is now being pulled in, whereas before > it depended on app-alternatives/cpio: > > host02 /etc/portage # emerge -avu linux-firmware > > These are the packages that would be merged, in order: > > Calculating dependencies ... done! > Dependency resolution took 1.03 s (backtrack: 0/20). > > [ebuild N ] app-arch/cpio-2.14::gentoo USE="-nls" 1486 KiB > [ebuild U ~] sys-kernel/linux-firmware-20240115-r1::gentoo > [20240115::gentoo] USE="initramfs redistributable savedconfig -compress-xz > -compress-zstd -deduplicate (-unknown-license)" 0 KiB > > Total: 2 packages (1 upgrade, 1 new), Size of downloads: 1486 KiB > > Is there a reason for this dependency change? Thanks for the catch, fixing that now
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=35e9df627447ac1afff3715be904a8be6d6391ee commit 35e9df627447ac1afff3715be904a8be6d6391ee Author: Mike Pagano <mpagano@gentoo.org> AuthorDate: 2024-02-04 18:45:02 +0000 Commit: Mike Pagano <mpagano@gentoo.org> CommitDate: 2024-02-04 18:45:02 +0000 sys-kernel/linux-firmware: Fix cpio dependency Thanks to awifox for reporting Bug: https://bugs.gentoo.org/913049 Signed-off-by: Mike Pagano <mpagano@gentoo.org> sys-kernel/linux-firmware/linux-firmware-20240115-r1.ebuild | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
This update had the exact opposite effect for me when it comes to merge time. My savedconfig contains 853 entries built from grepping anything amd, radeon and rtl_nic from an already downloaded distfile. The past dozen merge times have been around a minute to a minute and a half. Now it's been running for half an hour with copy-firmware.sh at 100% CPU usage and seemingly stuck at "copying/compressing file amd-ucode/README" and only some LICENSE.* files left. I have only redistributable and savedconfig USE flags set.
(In reply to franklyfrank from comment #11) > This update had the exact opposite effect for me when it comes to merge > time. My savedconfig contains 853 entries built from grepping anything amd, > radeon and rtl_nic from an already downloaded distfile. The past dozen merge > times have been around a minute to a minute and a half. Now it's been > running for half an hour with copy-firmware.sh at 100% CPU usage and > seemingly stuck at "copying/compressing file amd-ucode/README" and only some > LICENSE.* files left. I have only redistributable and savedconfig USE flags > set. attach your saved config
Created attachment 884699 [details] linux-firmware-20240115-r1 savedconfig
(In reply to franklyfrank from comment #13) > Created attachment 884699 [details] > linux-firmware-20240115-r1 savedconfig It appears to be barfing on the LICENSE files # ./copy-firmware.sh -v --firmware-list LICENSE.amd-ucode amd/amd_sev_fam17h_model0xh.sbin --ignore-duplicates /tmp/portage/sys-kernel/linux-firmware-20240115-r1/image/lib/firmware ERROR: unknown command-line options: /tmp/portage/sys-kernel/linux-firmware-20240115-r1/image/lib/firmware # ./copy-firmware.sh -v --firmware-list amd/amd_sev_fam17h_model0xh.sbin --ignore-duplicates /tmp/portage/sys-kernel/linux-firmware-20240115-r1/image/lib/firmware copying/compressing file amd/amd_sev_fam17h_model0xh.sbin #
(In reply to Mike Pagano from comment #14) > (In reply to franklyfrank from comment #13) > > Created attachment 884699 [details] > > linux-firmware-20240115-r1 savedconfig > > It appears to be barfing on the LICENSE files > > > # ./copy-firmware.sh -v --firmware-list LICENSE.amd-ucode > amd/amd_sev_fam17h_model0xh.sbin --ignore-duplicates > /tmp/portage/sys-kernel/linux-firmware-20240115-r1/image/lib/firmware > > ERROR: unknown command-line options: > /tmp/portage/sys-kernel/linux-firmware-20240115-r1/image/lib/firmware > > # ./copy-firmware.sh -v --firmware-list amd/amd_sev_fam17h_model0xh.sbin > --ignore-duplicates > /tmp/portage/sys-kernel/linux-firmware-20240115-r1/image/lib/firmware > copying/compressing file amd/amd_sev_fam17h_model0xh.sbin > > # Ignore this, I can replicate your issue
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f12fcc927ba762bac9cfa8ccd7f699e61b262d3c commit f12fcc927ba762bac9cfa8ccd7f699e61b262d3c Author: Mike Pagano <mpagano@gentoo.org> AuthorDate: 2024-02-15 20:20:51 +0000 Commit: Mike Pagano <mpagano@gentoo.org> CommitDate: 2024-02-15 20:20:51 +0000 sys-kernel/linux-firmware: savedconfig update, r2 Bug: https://bugs.gentoo.org/913049 Signed-off-by: Mike Pagano <mpagano@gentoo.org> .../files/linux-firmware-copy-firmware-r2.patch | 60 +++ .../linux-firmware-20240115-r2.ebuild | 401 +++++++++++++++++++++ 2 files changed, 461 insertions(+)
(In reply to franklyfrank from comment #11) > This update had the exact opposite effect for me when it comes to merge > time. My savedconfig contains 853 entries built from grepping anything amd, > radeon and rtl_nic from an already downloaded distfile. The past dozen merge > times have been around a minute to a minute and a half. Now it's been > running for half an hour with copy-firmware.sh at 100% CPU usage and > seemingly stuck at "copying/compressing file amd-ucode/README" and only some > LICENSE.* files left. I have only redistributable and savedconfig USE flags > set. Can you test -r2, please? May take a little time to sync to the mirrors.
Yep, -r2 works fine now and finishes in line with previous versions.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f00488d02e9bb9d5c85dbdee80bd5b1a780a4100 commit f00488d02e9bb9d5c85dbdee80bd5b1a780a4100 Author: Mike Pagano <mpagano@gentoo.org> AuthorDate: 2024-02-15 23:18:55 +0000 Commit: Mike Pagano <mpagano@gentoo.org> CommitDate: 2024-02-15 23:18:55 +0000 sys-kernel/linux-firmware: Update savedconfig patch Bug: https://bugs.gentoo.org/913049 Closes: https://bugs.gentoo.org/924679 Signed-off-by: Mike Pagano <mpagano@gentoo.org> .../files/linux-firmware-copy-firmware-r3.patch | 59 +++ .../linux-firmware-20240115-r3.ebuild | 401 +++++++++++++++++++++ 2 files changed, 460 insertions(+)
(In reply to franklyfrank from comment #18) > Yep, -r2 works fine now and finishes in line with previous versions. Thanks for testing, I think the next linux-firmware should have this, which should be out any day now.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a64e2b27009e5b30f57d0270c7d9390e36ba041d commit a64e2b27009e5b30f57d0270c7d9390e36ba041d Author: Mike Pagano <mpagano@gentoo.org> AuthorDate: 2024-02-20 19:25:02 +0000 Commit: Mike Pagano <mpagano@gentoo.org> CommitDate: 2024-02-20 19:25:02 +0000 sys-kernel/linux-firmware: add 20240220, more accurate savedconfig support Closes: https://bugs.gentoo.org/913049 Signed-off-by: Mike Pagano <mpagano@gentoo.org> sys-kernel/linux-firmware/Manifest | 1 + .../linux-firmware/linux-firmware-20240220.ebuild | 401 +++++++++++++++++++++ 2 files changed, 402 insertions(+)
