Based on the discussion yesterday / some days ago on IRC here are the suggested points: 1) drop mips(el) n32 stages and (active) n32 support 2) drop mips(el) multilib stages [needed after 1) because the primary ABI is n32] 3) reduce multilib on mips to toolchain-only (as e.g. riscv) [ie., drop the abi_mips_... useflags globally] 4) downgrade mips(el) n32 and multilib profiles to dev Apart from the removal of the useflags, which will make everyone on other architectures shout at me again, this is all very unproblematic and easy to do. Comments, opinions, support, flames?
As Arch Tester, including for mips, I support all of those points. I didn't hear any user/developer wanting to support 32 bit mips (as main or as multilib), so I think we should limit the resources we pour into those.
(In reply to Arthur Zamarin from comment #1) > As Arch Tester, including for mips, I support all of those points. > > I didn't hear any user/developer wanting to support 32 bit mips (as main or > as multilib), so I think we should limit the resources we pour into those. That's a misunderstanding. n32 != 32bit mips. "n32" is a specific ABI for 64bit mips, comparable to x32 on amd64. It's (depending on whom you ask) either useful for very low memory 64bit applications, or utterly useless. We definitely want to keep 32bit mips (which is as ABI called o32).
As arch tester, vote aye on all of the above.
@ xen0n, kumba: any input?
(In reply to Andreas K. Hüttel from comment #4) > @ xen0n, kumba: any input? Is there a summary of what was discussed so I can get some context? I am making a speculative guess, but does it have anything to do w/ Rust support? Cause last I heard, Rust does not support the n32 ABI. If not, though, it'd help to get a quick synopsis on things. In any event, n32 should be kept, especially as a stand-alone ABI target. multilib is a nice concept, but I don't see any real harm in letting it go. I think it's primary use case was to bootstrap n64, which is MIPS' pure-64bit ABI. Though, I wouldn't exactly miss n64 support, either (but if we can keep it, we should, because re-bootstrapping support for it in the future may be really difficult). But n32 support should to stay, as it should be the primary ABI for 64-bit support. Unlike the ill-fated x32 experiment, n32 has been around since the early IRIX 6.x days, so this places it as far back as the mid-1990's, I believe w/ the release of IRIX 6.2, and it was the standard ABI for IRIX up until its end w/ IRIX 6.5.31 in August 2006. Once upon a time, it was envisioned to become the default ABI for all of MIPS; I believe because many thought that 32-bit MIPS would eventually go the way of the dodo bird. I am also only speaking in terms of big-endian mips, which is the only functional hardware platform I still have some access to. If folks want to take mipsel in a different direction than big-endian mips, I see no issue. tl;dr, ideal situation for default/supported ABI would be: - 32-bit userland: o32 ABI - 64-bit userland: n32 ABI
I was one of the ones involved in the IRC discussion, and my thoughts are as follows: * The n32 ABI is not as useful or interesting as it seems. It suffers from a lot of the same pain as x32 on x86_64. There are a lot of weird bugs and corner cases in upstreams when pointers are not the same size as long. That doesn't make them good, or right, and yes, they are bugs to fix. But MIPS is already a niche arch and uncovers a lot of bizarre behaviour - do we really need to throw a weird ABI in as well? * Even on low memory systems, the difference is small. The places where it makes the most difference are things like Plasma or Firefox - big C++ codebases with lots of pointers stored everywhere. These need more memory to start up than you would have available on systems where n32 would make a difference :) * The IRIX ABI is practically irrelevant to Linux because as far as I'm aware there is no user-space emulation for running IRIX binaries on Linux. Even if there were, they could implement their own n32 ABI. I very highly doubt they would be able to dlopen(3) a Linux shared object! Trying to chase the ABI of other systems is how we ended up with the horror show on ppc64: ELFv1. It's also how we ended up with the horror show on m68k: alignment to 2 bytes instead of 4, which mirrors SunOS 4's ABI but doesn't follow any other SysV Unix (i.e. NeXT, A/UX). See bug 882825 for more. * And if we really want to care about SGI's ideas, 64-bit versions of IRIX 6.2 and higher *do* support n64 and they recommended it for "high performance". MIPSpro used the -64 flag for n64 ABI and considered n32 to be the way to go for compatibility with 32-bit OSes. IRIX ran 32-bit mode on downlevel boards like the Indy and O2 (see http://www.sgistuff.net/software/irixintro/documents/irix6.2TR.html#HDR39 for more). This was due to the PROM. I know the PROM of the Indy, O2, and related (IP22, 24, 32 at least) need a binutils hack to boot 64-bit Linux kernels at all, but again, it isn't hardware but rather a PROM limitation. The hardware itself is more than capable of running n64 code once the OS is loaded. I really thought MIPSpro releases post 6.5.22 defaulted to n64 mode, but I can't recall right now, and I don't have access to an Octane any more to test with, just my Indy (which obviously stops at 6.5.22). * I wasn't even aware of the Rust situation, but I can't imagine Rust is alone here. Note that qemu-mips64 assumes N64 ABI, so crossing with binfmt_misc may get confused with n32. OpenBSD has only ever supported N64 (both /sgi and /octeon). I'm not entirely sure whether things like Java or dotnet support MIPS or not, but it would be useful to see what ABIs they support. If they only support N32 that would be an argument towards preserving it. In short: I see no good reason to keep N32, and very good reasons to prefer N64.
(In reply to A. Wilcox (awilfox) from comment #6) > I was one of the ones involved in the IRC discussion, and my thoughts are as > follows: > > * The n32 ABI is not as useful or interesting as it seems. It suffers from > a lot of the same pain as x32 on x86_64. There are a lot of weird bugs and > corner cases in upstreams when pointers are not the same size as long. That > doesn't make them good, or right, and yes, they are bugs to fix. But MIPS > is already a niche arch and uncovers a lot of bizarre behaviour - do we > really need to throw a weird ABI in as well? > Especially given there's no real interest in it AFAICT, both x32 and n32 seem to rot with things failing to build and nobody testing it. > * Even on low memory systems, the difference is small. The places where it > makes the most difference are things like Plasma or Firefox - big C++ > codebases with lots of pointers stored everywhere. These need more memory > to start up than you would have available on systems where n32 would make a > difference :) > > * The IRIX ABI is practically irrelevant to Linux because as far as I'm > aware there is no user-space emulation for running IRIX binaries on Linux. > Even if there were, they could implement their own n32 ABI. I very highly > doubt they would be able to dlopen(3) a Linux shared object! Trying to > chase the ABI of other systems is how we ended up with the horror show on > ppc64: ELFv1. It's also how we ended up with the horror show on m68k: > alignment to 2 bytes instead of 4, which mirrors SunOS 4's ABI but doesn't > follow any other SysV Unix (i.e. NeXT, A/UX). See bug 882825 for more. > And other platforms, like HPPA, have long abandoned ABI compatibility with HPUX because it just wasn't worth it. If this is the strongest argument being presented, I think it ought to go indeed.
Created attachment 903646 [details] MIPSpro N32 ABI Handbook > * The n32 ABI is not as useful or interesting as it seems. It suffers from a lot of > the same pain as x32 on x86_64. There are a lot of weird bugs and corner cases in > upstreams when pointers are not the same size as long. That doesn't make them good, or > right, and yes, they are bugs to fix. But MIPS is already a niche arch and uncovers a > lot of bizarre behaviour - do we really need to throw a weird ABI in as well? I've been maintaining an n32 userland on my SGI Octane for almost 10 years now, from back in ~2015, when I re-ressurected the system and got Linux to run on the machine again. I have encountered very few actual bugs specifically related to the n32 ABI in all of those years. I just converted that userland to the 23.0 mips profiles and did a full 'emerge -eD @world' using gcc-14, and out of 473 packages, only one, app-arch/sharutils, had a compilation failure that I haven't done a deep look into yet (but I don't think it's going to be due to n32). I currently run that userland via Qemu for now, as my Octane is just too slow to keep up with the hyperspeed pace of modern-day development that's occurred these past few years (e.g., compiling gcc on the machine is >19hrs+; qemu can do it in ~4hrs). However, the machine does work, so I can still test n32 on real hardware, if needed. Can you please cite some of these upstream cases that you're obliquely referring to? E.g., Bug IDs, mailing list discussions, etc? FWIW, I consider n32 to be a better ABI vs x32. Code executing on a 64-bit MIPS CPU cannot access the full set of CPU registers if it is compiled as 32-bits (o32) or the mips1/mips2 ISAs. Code has to be compiled for the mips3/mips4 ISAs at a minimum, which requires a full 64-bit toolchain. The more modern mips32rX/mips64rX ISAs derive from these original four ISAs and have the same limitations. However, SGI determined that running full 64-bit all the time was unnecessary on their systems, so they developed the n32 ABI as a "compromise" ABI that enables all of the benefits of the 64-bit architecture but allows programs to use 32-bit instructions for most things. x86 CPUs don't rely on CPU registers nearly as much as MIPS/RISC CPUs do, so the problem that n32 was created to solve for MIPS literally doesn't exist on x86. It feels like x32 is more of an experiment, and specifically one within the Linux ecosystem, because no other OS that I am aware of has attempted anything similar. I assume the intent was to see whether there were any benefits to be gained by using a hybrid ABI of that nature. > * Even on low memory systems, the difference is small. The places where it makes the > most difference are things like Plasma or Firefox - big C++ codebases with lots of > pointers stored everywhere. These need more memory to start up than you would have > available on systems where n32 would make a difference :) I can really only speak to the SGI MIPS ecosystem, as that's what I run/maintain/support (at least for as long as my hardware continues to function), but to the best of my knowledge, no supported SGI system can currently run an X11 graphical userland anymore under Linux, so citing Firefox and/or KDE Plasma is pointless here. I also believe these are both very old examples of n32 compatibility issues, and long-dead horses do not need to be continuously flogged. This is one case where multilib comes into play: if an app doesn't compile under n32, or exhibits difficult-to-debug runtime issues under the n32 ABI, but works fine under either o32 or n64, then the app can be blocked from n32 and target just o32/n64 and still be usable. However, I don't think we ever encountered too many cases where this actually happened. A lot of n32 ABI issues that I recall were actually bugs in gcc or binutils, and once fixed there, the apps compiled fine and/or ran fine. > * The IRIX ABI is practically irrelevant to Linux because as far as I'm aware there > is no user-space emulation for running IRIX binaries on Linux. Even if there were, > they could implement their own n32 ABI. I very highly doubt they would be able to > dlopen(3) a Linux shared object! Trying to chase the ABI of other systems is how we > ended up with the horror show on ppc64: ELFv1. It's also how we ended up with the > horror show on m68k: alignment to 2 bytes instead of 4, which mirrors SunOS 4's ABI > but doesn't follow any other SysV Unix (i.e. NeXT, A/UX). See bug 882825 for more. There was, once upon a time, very early code in the Linux kernel that was designed to read and execute IRIX ELF binaries, but I don't think much progress was ever made on getting it to work and Ralf Baechle eventually pulled it out at some point (he was the Linux/MIPS maintainer back then). That said, while you highlight some valid scenarios in other platforms and OSes, those do not matter here because they are entirely separate CPU architectures and/or OS families; each of which are entire ecosystems unto themselves. Problems in one might have similarities in another, but that's going to be more by accident than by intentional design. Please point to some real-world, *current* bugs/issues that are: 1) specific to the Linux/MIPS ecosystem 2) are directly caused by the n32 ABI (either compiling or running) 3) do not exist when an app is compiled for/ran under the o32 or n64 ABIs 4) cannot be easily resolved without painful compromises > * And if we really want to care about SGI's ideas, 64-bit versions of IRIX 6.2 and > higher *do* support n64 and they recommended it for "high performance". MIPSpro used > the -64 flag for n64 ABI and considered n32 to be the way to go for compatibility with > 32-bit OSes. IRIX ran 32-bit mode on downlevel boards like the Indy and O2 (see > http://www.sgistuff.net/software/irixintro/documents/irix6.2TR.html#HDR39 for more). > This was due to the PROM. I know the PROM of the Indy, O2, and related (IP22, 24, 32 > at least) need a binutils hack to boot 64-bit Linux kernels at all, but again, it isn't > hardware but rather a PROM limitation. The hardware itself is more than capable of > running n64 code once the OS is loaded. Yes, I am well aware of this history. All Indys and any O2s w/ R5K/RM7K CPUs need this "hack", because they run a 32-bit ARCS PROM image. The "hack" is really just a make(1) target ('make vmlinux.32') in arch/mips/Makefile that calls objcopy(1) to do an ELF format conversion and stuffs a 64-bit 'vmlinux' kernel into the "o64" ABI, which is the o32 ABI extended to 64-bits. This format can be loaded by the Indy & O2 PROMS for some reason. In truth, SGI O2 PROMs can load a 64-bit 'vmlinux' image, but that system's "unified memory architecture" does some weird things and it never liked loading a full 64-bit kernel. The vmlinux.32 target JustWorks(TM), and no one's ever taken the time to figure out why. On an O2 w/ R10K/R12K CPU, though, you have to use a 64-bit kernel. Not that it'll boot far; the R10K's infamous speculative execution nature rears its head pretty quickly (SGI would've gotten a laugh out of how that ghost came back to haunt everyone w/ Spectre/Meltdown). Same for the IP28 platform (Indigo2 Impact R10000), though that system can be made to boot with some careful compile switches and patches. As far as I know, it's not possible to boot a pure 32-bit kernel on any SGI system in Linux anymore, as support for that was removed quite a few years ago, during the 2008-2015 timeframe, I think. > I really thought MIPSpro releases post 6.5.22 defaulted to n64 mode, but I can't > recall right now, and I don't have access to an Octane any more to test with, just my > Indy (which obviously stops at 6.5.22). Does your Indy still work? The RTC chips in those things will eventually die and the PROM cannot handle that situation, so it'll cease booting until you either desolder the RTC and replace it (then reprogram the MAC address back into the PROM), or de-cap the RTC and solder a new battery onto it. My R5K Indy suffered this fate, so it's been retired to one of my closet shelves. Might try to resurrect it someday. > * I wasn't even aware of the Rust situation, but I can't imagine Rust is alone here. This is really digging into my memory banks from years past, but I believe this issue was more directly tied to how clang/llvm was handling MIPS targets in its early days. It was making some bad assumptions based on the MIPS CHOST triplet, in that "mips64*" meant pure 64-bits and anything otherwise meant pure 32-bits. This had downstream impacts, including for rust-bin, which was only built for o32 or n64 ABIs because of this. This meant you could not run rust-bin in a pure-n32 userland. That said, I *think* that's all been fixed now, at least in the clang/llvm world, and I think they have a full implementation of the n32 ABI by way of the new MIPS CHOST triplets that are suffixed with *-gnuabin32. This is also recently implemented in the mips 23.0 profiles, which I just converted my Octane's userland to under qemu. Dunno if current rust-bin ships an n32 target, though, but it may be possible for one to be built now. See bug #515694 for some long-running commentary on this topic. > Note that qemu-mips64 assumes N64 ABI, so crossing with binfmt_misc may get confused > with n32. This is incorrect. Qemu does not get confused; it works fine w/ n32, and I know because I've been using qemu to keep my Octane's userland up-to-date a bit easier, plus I sometimes update seed stages for six different catalyst targets (though I have not yet tried getting those to work w/ catalyst 4.x). As long as you copy all of the qemu-mips* binaries into the chroot into /usr/bin, binfmt_misc figures out which one is the correct one and it JustWorks(TM). It's actually quite nice, and I gotta give props to Andreas for helping me to get it to work. I've verified that these chroots that work fine under qemu: - mips2_o32-glibc - mips3_n32-glibc - mips4_n32-glibc - mips4-r12k_n32-glibc - mips3_multilib-glibc - mips2_o32-musl I've also chrooted into a musl/mips3/n32 userland built by the OpenADK toolchain, and that also works, so it's an eventual TODO to get a musl/n32 environment up and running. Kinda siked that the musl folks have working n32 support! > OpenBSD has only ever supported N64 (both /sgi and /octeon). OpenBSD, like all of the BSDs, is more than just a kernel. It's an entire OS, and a majority of the effort behind its now-deprecated SGI support was the effort of mostly one person, Miod Vallet (miod@). When he left the OpenBSD project, Theo tried to keep the hardware support working, but I believe he finally excised the entirety of SGI support w/ late OpenBSD 6.x or early 7.x after issues arose with the disk driver on the SGI Origin 300/350 dev machine they were using (at least, that's what I recall from Theo's CVS commit message that removed all SGI support). In Linux, support for the various MIPS ABIs is the effort of a *lot* of people. Linux kernel maintainers (especially Thomas Boegendorfer, current Linux/MIPS kernel maintainer), GNU glibc developers, GNU gcc developers, GNU binutils developers, and whomever got n32 working under musl libc. I'm pretty sure that if n32 was causing as many headaches as you're claiming, I'd have come across one or more of them by now. And if something was so grevious of a problem with n32 that necessitated its deprecation and removal from some upstream-level project (like gcc, glibc, or binutils), I'm sure I'd have seen a blurb in Phoronix about it. > I'm not entirely sure whether things like Java or dotnet support MIPS or not, but it > would be useful to see what ABIs they support. If they only support N32 that would be > an argument towards preserving it. Java and .NET supporting or not supporting a particular MIPS ABI being cited as a valid argument for keeping or abandoning n32 is pretty laughable as far as things go. If Java had proper garbage collection, it'd have deleted itself years ago, and the world today would be a brighter, happier place. > In short: I see no good reason to keep N32, and very good reasons to prefer N64. None of your points have cited any current-day issues exclusively caused by the existence of the n32 ABI, including referencing open bugs or issues in upstream project bug/issue trackers. You've either obliquely cited past issues, cited unrelated issues in other OS families or CPU architectures, or cited issues in scenarios that are not technically feasible in these modern times, as your only supporting evidence for the removal of a rather unique and quite interesting userland ABI. If someone else has hard evidence to support the discontinuation of n32 ABI support, then I hope they speak up and present facts and cite valid references, especially covering my four points above. So barring that, my opinion remains that we keep the n32 ABI as a supported userland target for mips. --- I am also including with this the SGI documentation on the N32 ABI. The SGI Document P/N is 007-2816-004. The first couple of pages contain some really good info about how the ABI works and what issues the ABI was trying to solve for MIPS. I also found some really good notes on MIPS' ABIs here: https://maskray.me/blog/2023-09-04-toolchain-notes-on-mips
I was asked to comment on this bug, but people should note that I haven't worked on mips in many years. Basically I have nothing to add to Joshua's comment. When I was working on mips, I used to make extensive use of n32 as the "compromise" ABI on 64-bit mips. So I agree with his assessment.
