Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 911305

Summary: sys-kernel/linux-firmware add wildcards support in savedconfig config file
Product: Gentoo Linux Reporter: Georgi <jorodrbr>
Component: Current packagesAssignee: Chí-Thanh Christopher Nguyễn <chithanh>
Status: UNCONFIRMED ---    
Severity: enhancement CC: bruce, flaviocamilo, joakim.tjernlund, jstein, kernel, kfm, sam, zerochaos
Priority: Normal Keywords: PATCH
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description Georgi 2023-07-27 07:53:51 UTC
Savedconfig can be very useful feature for many cases. The case with linux firmware however would benefit from wildcards support in the config file as there are plenty of files, some with with changing versions and thus filenames for plenty of hardware. It just loses its convenience if the user needs to check and correct the config every time a linux-firmware update is available.
Comment 1 Joakim Tjernlund 2023-11-08 22:57:31 UTC
Yes please. Nvidia just committed huge FW files that could be excluded this way
Comment 2 Joakim Tjernlund 2023-11-08 23:01:58 UTC
(In reply to Joakim Tjernlund from comment #1)
> Yes please. Nvidia just committed huge FW files that could be excluded this
> way

Some not operator(exclude all Nvidia but not these ....) would be great too
Comment 3 Joakim Tjernlund 2024-01-11 12:45:49 UTC
This just became a real problem. I have a trimmed saved config file
and now I noticed I am missing the latest iwlwifi FW as these were non
existing when I created the config file.
Comment 4 RumpletonBongworth 2024-02-27 04:05:06 UTC
I've created an ebuild in my personal repo that introduces initial globbing support. Here is the supported syntax, as described by the commit message of the relevant patch against the copy-firmware script.

Currently, ? and * are supported. The * metacharacter behaves as if FNM_PATHNAME were not in effect. In other words, both metacharacters behave as they do in the shell, except that * is also allowed to match the <slash> character.

Examples:

# Match all iwlwifi firmware
iwlwifi-*

# Match all bnx2 firmware
bnx2/bnx2-*.fw

# Match paths containing "bnx2" anywhere (thus also including bnx2x firmware)
*bnx2*

https://codeberg.org/kerframil/portage-overlay/commit/53dacb45cbfb16c2de1068fb8ff82e3d7f862808
Comment 5 RumpletonBongworth 2024-02-27 15:25:15 UTC
(In reply to Joakim Tjernlund from comment #2)
> (In reply to Joakim Tjernlund from comment #1)
> > Yes please. Nvidia just committed huge FW files that could be excluded this
> > way
> 
> Some not operator(exclude all Nvidia but not these ....) would be great too

I'm not sure that there is a concrete use case for such an operator. I would think a directory prefix combined with * to be sufficient.
Comment 6 Joakim Tjernlund 2024-02-27 15:32:59 UTC
(In reply to kfm from comment #5)
> (In reply to Joakim Tjernlund from comment #2)
> > (In reply to Joakim Tjernlund from comment #1)
> > > Yes please. Nvidia just committed huge FW files that could be excluded this
> > > way
> > 
> > Some not operator(exclude all Nvidia but not these ....) would be great too
> 
> I'm not sure that there is a concrete use case for such an operator. I would
> think a directory prefix combined with * to be sufficient.

Then one would have to list all dirs except Nvida(or AMD) I think.
If there is a new dir/file added to the firmware you might miss it.
Comment 7 RumpletonBongworth 2024-02-27 15:40:29 UTC
(In reply to Joakim Tjernlund from comment #6)
> (In reply to kfm from comment #5)
> > (In reply to Joakim Tjernlund from comment #2)
> > > (In reply to Joakim Tjernlund from comment #1)
> > > > Yes please. Nvidia just committed huge FW files that could be excluded this
> > > > way
> > > 
> > > Some not operator(exclude all Nvidia but not these ....) would be great too
> > 
> > I'm not sure that there is a concrete use case for such an operator. I would
> > think a directory prefix combined with * to be sufficient.
> 
> Then one would have to list all dirs except Nvida(or AMD) I think.
> If there is a new dir/file added to the firmware you might miss it.

Let me see if I understand correctly. You want to be be able to:

- install all firmware ("*")
- while subtracting "nvidia/*"
- while re-adding a particular sub-directory (say, "nvidia/ga107/*")

Is that right?
Comment 8 Joakim Tjernlund 2024-02-27 16:30:43 UTC
(In reply to kfm from comment #7)
> (In reply to Joakim Tjernlund from comment #6)
> > (In reply to kfm from comment #5)
> > > (In reply to Joakim Tjernlund from comment #2)
> > > > (In reply to Joakim Tjernlund from comment #1)
> > > > > Yes please. Nvidia just committed huge FW files that could be excluded this
> > > > > way
> > > > 
> > > > Some not operator(exclude all Nvidia but not these ....) would be great too
> > > 
> > > I'm not sure that there is a concrete use case for such an operator. I would
> > > think a directory prefix combined with * to be sufficient.
> > 
> > Then one would have to list all dirs except Nvida(or AMD) I think.
> > If there is a new dir/file added to the firmware you might miss it.
> 
> Let me see if I understand correctly. You want to be be able to:
> 
> - install all firmware ("*")
> - while subtracting "nvidia/*"
> - while re-adding a particular sub-directory (say, "nvidia/ga107/*")
> 
> Is that right?

Something like that. I think people generally knows what FW that don't want
so a way to say: install all but these(not AMD, not Nvidia etc.)
Comment 9 RumpletonBongworth 2024-02-27 16:34:00 UTC
(In reply to Joakim Tjernlund from comment #8)
> Something like that. I think people generally knows what FW that don't want
> so a way to say: install all but these(not AMD, not Nvidia etc.)

That seems reasonable. I'll mull it over.
Comment 10 RumpletonBongworth 2024-02-27 23:30:23 UTC
I have committed sys-kernel/linux-firmware-20240220-r5 to my personal repo.

https://codeberg.org/kerframil/portage-overlay/commit/7860e8f09afaea8e5e8b80cd4dcae50b4d8ed269

The filtering syntax has been extended, in accordance with the preceding discussion. Further details can be found within patch 0004. Below is an excerpt from the commit message, showing some sample policies.

---

A sample policy follows, the meaning of which is to install all
firmware, excluding all nvidia firmware with the exception of paths
beginning with "nvidia/ga107/".

+ *
- nvidia/*
+ nvidia/ga107/*

Below is another sample policy, the meaning of which is to install only
the iwlifi firmware, along with the Adaptec Starfire firmware (specified
literally).

adaptec/starfire_rx.bin
adapter/starfire_tx.bin
+ *iwlwifi*
Comment 11 RumpletonBongworth 2024-02-29 07:51:34 UTC
Several revisions have been issued since. The number of patches has been reduced from four to two. The awk code has been moved to its own script, rendered more idiomatic and commented, with the resulting changes to copy-firmware.sh being less intrusive. The scope of the second patch concerns only performance improvements, which are now better described.

https://codeberg.org/kerframil/portage-overlay/commit/4f0345bc2c2072289add0b80da4c141fc59dcdc9

Please note that it is possible to experiment with the filtering syntax without having to constantly re-install the package with portage. Below is an example of how to do so.

$ { printf '%s\n' '+ nvidia/tegra*' '- *186*' --; cat WHENCE; } | ./filter-whence.awk '(File|RawFile|Link)'
2 File nvidia/tegra124/vic03_ucode.bin
2 Link nvidia/tegra124/vic.bin vic03_ucode.bin
2 File nvidia/tegra124/xusb.bin
2 File nvidia/tegra194/vic.bin
2 File nvidia/tegra194/xusb.bin
2 File nvidia/tegra210/vic04_ucode.bin
2 Link nvidia/tegra210/vic.bin vic04_ucode.bin
2 File nvidia/tegra210/xusb.bin
Comment 12 Mike Pagano gentoo-dev 2024-03-12 17:06:31 UTC
(In reply to kfm from comment #11)
> Several revisions have been issued since. The number of patches has been
> reduced from four to two. The awk code has been moved to its own script,
> rendered more idiomatic and commented, with the resulting changes to
> copy-firmware.sh being less intrusive. The scope of the second patch
> concerns only performance improvements, which are now better described.
> 
> https://codeberg.org/kerframil/portage-overlay/commit/
> 4f0345bc2c2072289add0b80da4c141fc59dcdc9
> 
> Please note that it is possible to experiment with the filtering syntax
> without having to constantly re-install the package with portage. Below is
> an example of how to do so.
> 
> $ { printf '%s\n' '+ nvidia/tegra*' '- *186*' --; cat WHENCE; } |
> ./filter-whence.awk '(File|RawFile|Link)'
> 2 File nvidia/tegra124/vic03_ucode.bin
> 2 Link nvidia/tegra124/vic.bin vic03_ucode.bin
> 2 File nvidia/tegra124/xusb.bin
> 2 File nvidia/tegra194/vic.bin
> 2 File nvidia/tegra194/xusb.bin
> 2 File nvidia/tegra210/vic04_ucode.bin
> 2 Link nvidia/tegra210/vic.bin vic04_ucode.bin
> 2 File nvidia/tegra210/xusb.bin

Can you make this all into one file with the comments at the top and rename it to gentoo-copy-firmware.sh ?


Once that is done, I'll do a -rX with this included.
Comment 13 RumpletonBongworth 2024-03-14 02:38:53 UTC
(In reply to Mike Pagano from comment #12)
> Can you make this all into one file with the comments at the top and rename
> it to gentoo-copy-firmware.sh ?
> 
> 
> Once that is done, I'll do a -rX with this included.

Hi Mike.

As I understand it, you are requesting that the patch series begins by effectively forking the "copy-firmware.sh" shell script as "gentoo-copy-firmware.sh", leaving the original unmodified. Certainly, that would be straightforward to attend to. However, I could use some clarification as to the rest.

By making it all into one file, are you suggesting that the awk program be inlined? If so, that was split out into a separate script at the point that it became significant enough to warrant it. Initially, it began as a very short program to efficiently and correctly select literal pathnames according to a list being read from the standard input, only to grew in stature after implementing the extended filtering syntax. Having it be a separate script makes it easier to read, develop, test and debug. Further, since the program need not be conveyed as an argument, it eliminates another source of pressure towards the dreaded ARG_MAX threshold.

Additionally, I am uncertain as to whether you are referring to code comments or some other form of commentary (such as the content of the commit messages).
Comment 14 RumpletonBongworth 2024-03-14 03:11:29 UTC
Another potential interpretation just occurred to me, which is to supply the new/forked scripts for dropping directly into FILESDIR. Whatever the case may be, I shall wait for guidance before doing anything further.
Comment 15 Joakim Tjernlund 2024-04-24 19:38:23 UTC
ping? Please move this forward
Comment 16 RumpletonBongworth 2024-04-27 22:25:24 UTC
(In reply to Joakim Tjernlund from comment #15)
> ping? Please move this forward

The ball is in Gentoo's court. There's little I can do without further engagement.
Comment 17 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-04-27 22:47:57 UTC
From my conversation with mpagano a few weeks ago, I think the intention is to copy the upstream script into its own file, then give patches to that, as it's easier than maintaining patches which need rebasing to the existing upstream file.

mike?
Comment 18 Mike Pagano gentoo-dev 2024-04-28 10:52:17 UTC
I would like to see this upstreamed