Bug 920527 – app-containers/lxd: lxd will lose access to upstream image server hosted by linuxcontainers, which is default in Gentoo

• Product: Gentoo Linux
• Status: CONFIRMED ---

[Description Joonas Niilola 2023-12-22 08:18:08 UTC]

Earlier this year Canonical, the main contributor and owner of LXD, moved LXD from community-hosted LinuxContainers under Canonical's own management [1,2,3]. While doing so they removed administrative project access from non-Canonical employees, and forced Canonical's strict CLA on every future contributor. This caused a reaction in the community, and spawned a fork of LXD called Incus [4]. Incus was later transferred under LinuxContainers community's care. Incus would pull changes from upstream LXD to stay compatible, and follow LXD's release cycles.

Recently Canonical re-licensed LXD project aggressively, making it impossible for Incus to pull code from LXD and use Canonical's work on it [5,6]. Currently LinuxContainers community is responsible for building and publishing all container & virtual machine images found in https://images.linuxcontainers.org/ to LXD and Incus, while Canonical only provides containers and virtual machines for Ubuntu (note that e.g. https://uk.lxd.images.canonical.com/ is just a front for images.linuxcontainers.org). At the time of writing, it's unclear whether Canonical will start to provide images for other distributions.

Therefore the LinuxContainers community has decided to stop offering images for LXD. The support will be phased out in steps [7]. 
Non-LTS LXD users (unstable in Gentoo) will start having limited access to LinuxContainers's images on 2024-01-01, and they will completely lose access to all images on 2024-02-01. LTS LXD users (stable in Gentoo) will start losing access to some images on 2024-04-15, and will completely lose access to all images on 2024-05-01. This is so LTS users have a fair chance to migrate into Incus LTS release, which is expected to come out on April 2024. These times are a subject to change. Please follow upstream on this matter [7].

What should you do?

1: Move to Incus. 
Incus is already well-supported in Gentoo and it does come with a handy tool called "lxd-to-incus" to migrate your current system [8]. Unstable users (>=lxd-5.18) should move as soon as possible, while LTS users can wait for Incus LTS to be stabilized in Gentoo. This is expected to happen sometime during April 2024.

Migration with this tool should be pretty seamless, see upstream documentation at [8].
The tool works in both openrc and systemd host environments.

2: Build and deploy your own image server.
Gentoo has packaged Distrobuilder which is also used by upstream to build LXD/Incus container images. Currently Distrobuilder is capable of building both LXD and Incus images (both container and virtual machine), but obviously this may be a subject to change. Check Gentoo's Distrobuilder wiki page [9] to stay up to date with its support. Upstream container build "recipes" are found in [10].

3: Wait and see what Canonical does.
At the time of writing it's unclear what Canonical will do about the situation. It's possible Canonical will start offering distro images like LinuxContainers has done so far. If that happens, the only change you might need to do is update your remote image: server location. 

Please follow this bug for updates, and/or upstream communication channels - a news item about this situation will also be published for Gentoo users.

Sources and links:
1: https://linuxcontainers.org/lxd/
2: https://discourse.ubuntu.com/t/supporting-the-lxd-community/36764
3: https://discuss.linuxcontainers.org/t/lxd-is-no-longer-part-of-the-linux-containers-project/17593
4: https://linuxcontainers.org/incus/announcement/

5: https://discuss.linuxcontainers.org/t/lxd-has-been-re-licensed-and-is-now-under-a-cla/18454
6: https://stgraber.org/2023/12/12/lxd-now-re-licensed-and-under-a-cla/

7: https://discuss.linuxcontainers.org/t/important-notice-for-lxd-users-image-server/18479

8: https://linuxcontainers.org/incus/docs/main/howto/server_migrate_lxd/

9: https://wiki.gentoo.org/wiki/Distrobuilder
10: https://github.com/lxc/lxc-ci/tree/main/images


PS: Should LXD become a pure Canonical/Ubuntu thing, I don't see a future for it in Gentoo.

[Comment 1 Joonas Niilola 2023-12-26 06:26:47 UTC]

https://github.com/gentoo/gentoo/pull/34449 incus-0.4 - a bit hesitant to push due to idmapping "bug" in incus-user: https://github.com/lxc/incus/issues/329

[Comment 2 Joonas Niilola 2023-12-27 17:23:02 UTC]

Incus-0.4 pushed. I'd recommend a fresh installation, guide here:

https://wiki.gentoo.org/wiki/Incus
https://wiki.gentoo.org/wiki/Incus#Migrating_from_LXD

[Comment 3 Larry the Git Cow 2023-12-28 17:20:26 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/data/gentoo-news.git/commit/?id=1dc53c33d97053429b8c7a82aca0066c139d2da6

commit 1dc53c33d97053429b8c7a82aca0066c139d2da6
Author:     Joonas Niilola <[e-mail]>
AuthorDate: 2023-12-28 17:18:06 +0000
Commit:     Joonas Niilola <[e-mail]>
CommitDate: 2023-12-28 17:20:13 +0000

    2023-12-28-lxd-to-lose-access-for-its-image-server: add news item
    
    Bug: https://bugs.gentoo.org/920527
    Signed-off-by: Joonas Niilola <[e-mail]>

 ...-lxd-to-lose-access-for-its-image-server.en.txt | 44 ++++++++++++++++++++++
 1 file changed, 44 insertions(+)

[Comment 4 Larry the Git Cow 2024-01-04 07:03:00 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f8e8469a8b05e225f567701648223b24aae7c348

commit f8e8469a8b05e225f567701648223b24aae7c348
Author:     Joonas Niilola <[e-mail]>
AuthorDate: 2024-01-04 06:59:55 +0000
Commit:     Joonas Niilola <[e-mail]>
CommitDate: 2024-01-04 07:02:58 +0000

    app-containers/incus: add 'fuidshift' use flag that conflicts with lxd
    
     - apparently some weird systems are symlinking /usr/sbin into /usr/bin !
    
    Bug: https://bugs.gentoo.org/920527
    Signed-off-by: Joonas Niilola <[e-mail]>

 .../incus/{incus-0.4.ebuild => incus-0.4-r1.ebuild}   | 19 ++++++++++++++-----
 app-containers/incus/metadata.xml                     |  3 +++
 2 files changed, 17 insertions(+), 5 deletions(-)

[Comment 5 Francesco Riosa 2024-01-05 23:44:08 UTC]

>     
>      - apparently some weird systems are symlinking /usr/sbin into /usr/bin !
>     
That was probably sys-apps/merge-usr messing with it, because I do have those symlynk on all of my gentoo systems

[Comment 6 Joonas Niilola 2024-01-11 08:28:12 UTC]

Finally a word from Canonical:

https://discourse.ubuntu.com/t/an-update-on-the-licence-change-and-community-image-server/41549

> We have no intention of restricting LXD only to Ubuntu users and will be 
> providing a replacement image server serving images for other Linux 
> distributions. 

That's good news, although unfortunately some images are already unavailable for non-LTS lxd users and it's going to take some time for Canonical to set up a replacement. Still, good news overall in my opinion.



(In reply to Francesco Riosa from comment #5)
> >     
> >      - apparently some weird systems are symlinking /usr/sbin into /usr/bin !
> >     
> That was probably sys-apps/merge-usr messing with it, because I do have
> those symlynk on all of my gentoo systems

Heh, yeah, it was more of a joke. My own systems are old :)

[Comment 7 Larry the Git Cow 2024-01-26 06:51:32 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=07d9416a29bd333352f5bc72669d50e0bff523df

commit 07d9416a29bd333352f5bc72669d50e0bff523df
Author:     Joonas Niilola <[e-mail]>
AuthorDate: 2024-01-26 06:50:56 +0000
Commit:     Joonas Niilola <[e-mail]>
CommitDate: 2024-01-26 06:50:56 +0000

    app-containers/incus: add 0.5
    
    Bug: https://bugs.gentoo.org/920527
    Signed-off-by: Joonas Niilola <[e-mail]>

 app-containers/incus/Manifest         |   2 +
 app-containers/incus/incus-0.5.ebuild | 206 ++++++++++++++++++++++++++++++++++
 2 files changed, 208 insertions(+)

[Comment 8 Joonas Niilola 2024-04-07 06:35:26 UTC]

Incus-6.0.0 is out and in the repo. It is the first LTS release and if it works fine, the first stable candidate as well.

https://discuss.linuxcontainers.org/t/incus-6-0-lts-has-been-released/19576


On another news, LXD finally got their own image server up and running :

https://discourse.ubuntu.com/t/new-lxd-image-server-available-images-lxd-canonical-com/43824

As I don't have LXD installed anymore anywhere, I can't view what images are available. LXD-5.21.1 LTS has also been released and would be great to get someone still using LXD to maintain it in Gentoo. I'm gonna drop myself from the package metadata sometime soon.

When first Incus LTS is stabilized, I'll be closing this bug. Now's pretty much your final time to decide whether to migrate to Incus or stick with LXD, since the Incus's migration tool still works from Lxd-5.21 (but future compatibility is not promised).

[Comment 9 Joonas Niilola 2024-04-15 07:04:39 UTC]

HTML listing works now (apparently was a bug why it didn't work during my previous post): https://images.lxd.canonical.com/

Looks like they're offering the popular options (glad to see Gentoo there!) 
haven't tried the images but they're most likely based on same build configuration that incus uses:
https://github.com/lxc/lxc-ci/tree/main/images

[Comment 10 Brahmajit Das 2024-04-20 19:14:20 UTC]

(In reply to Joonas Niilola from comment #8)
> As I don't have LXD installed anymore anywhere, I can't view what images are
> available. LXD-5.21.1 LTS has also been released and would be great to get
> someone still using LXD to maintain it in Gentoo.

I've been playing with LXD 5.21.1, and these are the gentoo images that are available.

>lxc image list images: | grep gentoo
>| gentoo/openrc (3 more)                   | 1324d5cc934a | yes    | Gentoo current amd64 (20240420_0007)      | x86_64       | VIRTUAL-MACHINE | 1161.52MiB | Apr 20, 2024 at 12:00am (UTC) |
>| gentoo/openrc (3 more)                   | e3bb6d3f70e2 | yes    | Gentoo current amd64 (20240420_0007)      | x86_64       | CONTAINER       | 345.19MiB  | Apr 20, 2024 at 12:00am (UTC) |
>| gentoo/systemd (3 more)                  | 877e08758260 | yes    | Gentoo current amd64 (20240420_0007)      | x86_64       | CONTAINER       | 366.33MiB  | Apr 20, 2024 at 12:00am (UTC) |
>| gentoo/systemd (3 more)                  | ccadd5c83c4a | yes    | Gentoo current amd64 (20240420_0007)      | x86_64       | VIRTUAL-MACHINE | 1147.47MiB | Apr 20, 2024 at 12:00am (UTC) |

Looks like same as before?

[Comment 11 Larry the Git Cow 2024-05-05 06:28:06 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3f461592ad386529ea14e13785e573dc6416084a

commit 3f461592ad386529ea14e13785e573dc6416084a
Author:     Joonas Niilola <[e-mail]>
AuthorDate: 2024-05-05 06:27:32 +0000
Commit:     Joonas Niilola <[e-mail]>
CommitDate: 2024-05-05 06:28:02 +0000

    app-containers/incus: stabilize 6.0.0-r1 for amd64
    
    Bug: https://bugs.gentoo.org/920527
    Signed-off-by: Joonas Niilola <[e-mail]>

 app-containers/incus/incus-6.0.0-r1.ebuild | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)