| Summary: | Hosting for prospective MIPS and ia64 dev boxes | ||
|---|---|---|---|
| Product: | Gentoo Infrastructure | Reporter: | Marek Szuba (RETIRED) <marecki> |
| Component: | Dev box issues | Assignee: | Gentoo Infrastructure <infra-bugs> |
| Status: | RESOLVED WONTFIX | ||
| Severity: | normal | CC: | arthurzam, ia64, mgorny, mips |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
|
Description
Marek Szuba (RETIRED)
2022-03-18 14:57:50 UTC
Do you have a rough wattage for these? Onyx2 and Altix: insane (according to robbat2, it's ~10 kW for the former and around ~15 kW for the latter) Oigin 300: up to 460 W per base module Prism: about 1 kW in this configuration (In reply to Marek Szuba from comment #0) > There is a decommissioned compute cluster in Cambridge, UK whose owners are > looking to donate at least some of the hardware to a museum or other good > cause. These include: > * an SGI Onyx2 (MIPS), single-rack version but unsure of exact CPU > configuration at this point Really called a "deskside" unit, rather than "single rack". I have one out in my living room. They stand about ~8U-9U tall and have small wheels on the bottom for moving around (but only in straight lines; wheels are fixed). Onyx2's are the graphics-equipped versions of an Origin 2000. Though, the primary graphics interface, InfiniteReality ("Kona" in SGI jargon) is utterly unsupported in Linux. The SGI system class for these is IP27 overall...there's more subtle distinctions that I'll gloss over for now. > * an SGI Origin 300 (mips - 4x MIPS R14000 500 MHz), in double-2U > rack-mounted chassis This is an IP35 system class machine. Should only be a single 2U unit. If referring to "double 2U", then that sounds like two separate units linked together using CrayLink cables. That would mean there's really 8 CPUs available. Linux does not yet run on this hardware, though there have been active efforts in the past that have reached single-CPU booting on an initramfs image. I last spoke to the Linux/MIPS maintainer, Thomas Boegendorfer, a few weeks ago and he still has plans to try and get this support into better shape and included in the mainline Linux kernel at some point. > I found out this morning that they wouldn't be against letting Gentoo use > them. However, the hardware cannot stay where it currently is. > > Therefore: > - which one of these, if any, do we want? The nostalgic at heart says take both of them, as neither are produced anymore and you are literally holding onto a relic of a different era. But realistically-speaking, the Origin 300 is the more likely candidate for a future Gentoo/MIPS big-endian build/dev machine due to its compactness and power consumption. > - if we want any of the larger systems, could they be stripped down to > smaller size (like ditching some of the blades in case of the Altix) yet in > the end still be useful? Can't strip an Onyx2 Deskside unit down. They are pretty self-contained. (In reply to Marek Szuba from comment #2) > Onyx2 and Altix: insane (according to robbat2, it's ~10 kW for the former > and around ~15 kW for the latter) My Onyx2 runs off standard US 120V/60Hz power, though it really does prefer 240V/60Hz. As long as the Kona graphics module isn't fully spun up, the average wattage pull according to a Kill-A-Watt is around ~768W, so it'll drink 1kWH about every ~1hr 15mins, give or take. That's why I can run it off of a standard 120V outlet. If I boot IRIX 6.5, wattage jumps up a little because graphics is activated, but only 2D for desktop. However, I am told that to fully spin Kona up for a 3D demo would likely pop a breaker, since I should be using a 20A socket instead of a 15A one. Hence why it's really preferable to use a 240V outlet instead. > Oigin 300: up to 460 W per base module It's been a few years since I powered my Orin 300 on, but I recall the Kill-A-Watt clocked it in at around ~240W idle, not running any OS (just sitting at the PROM monitor over serial). (In reply to Joshua Kinard from comment #3) > > * an SGI Onyx2 (MIPS), single-rack version but unsure of exact CPU > > configuration at this point > > Really called a "deskside" unit, rather than "single rack". I have one out > in my living room. They stand about ~8U-9U tall and have small wheels on > the bottom for moving around (but only in straight lines; wheels are fixed). Umm, no. It's a single-rack version, installed in a 42U (I think? definitely taller than a person) rack with no wheels. Looks exactly as the one pictured on Wikipedia: https://en.wikipedia.org/wiki/SGI_Onyx2 (In reply to Marek Szuba from comment #5) > (In reply to Joshua Kinard from comment #3) > > > > * an SGI Onyx2 (MIPS), single-rack version but unsure of exact CPU > > > configuration at this point > > > > Really called a "deskside" unit, rather than "single rack". I have one out > > in my living room. They stand about ~8U-9U tall and have small wheels on > > the bottom for moving around (but only in straight lines; wheels are fixed). > > Umm, no. It's a single-rack version, installed in a 42U (I think? definitely > taller than a person) rack with no wheels. Looks exactly as the one pictured > on Wikipedia: https://en.wikipedia.org/wiki/SGI_Onyx2 Whoops, yeah, I forgot they made that configuration. I believe they are a bit rarer than the similar-looking Origin 2000 rack configs. The Onyx2 deskside units were more popular. It is actually possible to convert a rack node to a deskside unit if you have the available plastic skins. Still, a single-node unit should still have similar power requirements to the deskside model. If the rack comes with it, the power costs will be higher because there are some bits in the rack itself that also draw power (PDU, serial/MMSC console, fans, etc). In the space between the upper and lower nodes (just under the MMSC display), there's a spot where a "GIGAChannel" box goes -- really just a converted Origin 200 chassis -- any idea if that is included as well? To summarize, we're basically talking of hardware that's primarily a very expensive heater, and the only one that doesn't draw insane amounts of power does not run Linux? (In reply to Michał Górny from comment #7) > To summarize, we're basically talking of hardware that's primarily a very > expensive heater, and the only one that doesn't draw insane amounts of power > does not run Linux? *does not run Linux yet. But more or less, yeah. It is old hardware. These are the pitfalls of old hardware. A classic car lacks airbags, ABS, traction control, lane-assist, satellite radio, smartphone integration, emissions controls, and gets terrible fuel mileage. But for some, classic cars are still a viable, sometimes vibrant, hobby. Old computing hardware is not terribly different in this regard. I don't believe Gentoo should burn money on a hobby. If we're to host machines, they should be really beneficial to Gentoo users, and given all the data so far, I don't think we should take them unless someone makes a really strong case for doing so. (In reply to Michał Górny from comment #9) > I don't believe Gentoo should burn money on a hobby. If we're to host > machines, they should be really beneficial to Gentoo users, and given all > the data so far, I don't think we should take them unless someone makes a > really strong case for doing so. Not that I am disagreeing entirely -- I fully understand the cost of keeping around and using old hardware and I recognize that the foundation has limited resources available to spend. But I do disagree rather pointedly on the phrasing that Gentoo shouldn't burn money on a hobby. Half, if not more, of the arches we support are done so precisely because they ARE a hobby to someone out there, including fellow developers. By your logic, we should just drop all of those, especially MIPS and IA64, which are dead architectures now, and only support amd64, arm64, and riscv. Maybe not all of them require direct financial costs, but via one means or another, money, in some form, is still being burnt on supporting them. I somewhat agree, at least given that there's hasn't been any major progress on improving mips support for years now. However, that's a separate topic. (In reply to Michał Górny from comment #11) > I somewhat agree, at least given that there's hasn't been any major progress > on improving mips support for years now. However, that's a separate topic. MIPS support actually works quite fine, depending on how you set your goals. The install I use on my Octane is almost seven years old now. To say it needs "improving" is to state a nebulous goal. Different people are going to have different ideas about what that means. The installation side of the house does need some work. But once installed, the basic functionality is there. I've told people for years that if they want to help out, find an old SGI for sale and make it a project to get running, install Gentoo/MIPS, and help out. Suffice it to say, the interest isn't there. I support what I can within the time and resource limits available to me. Good support means: - regular stage builds - work towards non-exp status (depgraph consistency) - replies on bugs - ideally devbox access for devs to debug/test who aren't personally interested in MIPS These are the same metrics we use for other arches. (In reply to Sam James from comment #13) > Good support means: > - regular stage builds This may be resolved soon thanks to Andreas' efforts to find a qemu-based solution for stage-builds that can run on demeter. > - work towards non-exp status (depgraph consistency) We need more developers on the team to help resolve this. Due to the slow compilation time on my systems, I have to pick and choose what gets keyworded, and that's when the box isn't updating itself or doing stage builds. > - replies on bugs I do this when appropriate. I've resolved quite a number of keywording bugs recently for MIPS. > - ideally devbox access for devs to debug/test who aren't personally > interested in MIPS This is the hard part because there are so many flavors of MIPS. You simply cannot get all possible flavours on a single machine. So you would need multiple dev boxes for MIPS. And most modern MIPS hardware is in the embedded space, which isn't terribly suited for hosting environments or resource-equipped to handle multiple devs accessing it at the same time testing builds out. SGI systems are one of the few server-grade MIPS systems left that one can reasonably find. Like the Onyx2 system being mentioned in this bug. That's going to be suited for running in a server room and it supports a lights-out-like device for remote power up/down. But it's also power hungry, and my own testing indicates the hardware can be flaky at times (my Onyx2 is prone to I/O seizure when doing heavy disk activity; unknown if kernel-related or hardware). The more reliable system is an Octane, but that's a workstation and lacks remote management functions, and has a kernel bug where reboots don't always work, requiring one to physically power-cycle the box. Ideally, that's why it'd be nice for people to get interested in hacking on IP35 support so we can get at least an Origin 300 or Origin 350 working. Small size (2U), reasonable power consumption, reasonable specs for dev tasks, etc. Before I get distracted - okay, so we've had all the (IMHO very useful) feedback regarding MIPS hardware. Not a word from the ia64 camp, though... Anyone? Bueller? Bueller? (In reply to Joshua Kinard from comment #3) > This is an IP35 system class machine. Should only be a single 2U unit. If referring to "double 2U", then that sounds like two separate units linked together using CrayLink cables. That would mean there's really 8 CPUs available. Having asked the owner of the hardware, this indeed seems to be the case. Question, though - can such units be linked directly? When this cluster was operational they were connected via a NUMAlink hub, which however broke down quite a while ago. (In reply to Joshua Kinard from comment #6) > In the space between the upper and lower nodes (just under the MMSC display), there's a spot where a "GIGAChannel" box goes -- really just a converted Origin 200 chassis -- any idea if that is included as well? I think so! Or at least there is a box with several large drive bays, which looks similar to what an image search for "SGI gigachannel" has shown me, in the space you have indicated. (In reply to Joshua Kinard from comment #12) > The install I use on my Octane is almost seven years old now. Which reminds me - there are also an Octane and an Origin 200 (both deskside, with the plastic bits on the latter's case quite severely damaged)! I haven't mentioned them before because their CPU specs look quite pitiful comparing to the rack machines, that said if the choice is between a power-hungry Onyx2, an unsupported Origin 300 and an Octane... Plus we might actually be able to get away with keeping the Octane running on-site. (In reply to Joshua Kinard from comment #14) > Ideally, that's why it'd be nice for people to get interested in hacking on IP35 support so we can get at least an Origin 300 or Origin 350 working. Small size (2U), reasonable power consumption, reasonable specs for dev tasks, etc. I MIGHT perhaps be able to get one of the Origin 300 base units powered up and connected to the Internet, if it helps. Or possibly arrange for it/them to be donated to one of the Linux/MIPS people working on this, if THAT helps. Or at the very least stash them away, unpowered, somewhere out of the way until they can be used. (In reply to Marek Szuba from comment #15) > Before I get distracted - okay, so we've had all the (IMHO very useful) > feedback regarding MIPS hardware. Not a word from the ia64 camp, though... > Anyone? Bueller? Bueller? Not an IA64 expert by any means, but I can speak a *little* to SGI Altix systems. They are closely related to SGI Origins at heart, but with IA64 CPUs and addressing. Think of them as the evolution of SGI's MIPS hardware line, just with a different CPU architecture. In some cases, SGI literally took MIPS system chips and just stuck them into Altix chassis. There's something in them called "SAL", which acts as a hardware translation layer to convert various MIPS architecture specifics to IA64. Link to the Altix 4700 User Guide: https://irix7.com/techpubs/007-4823-003.pdf SGI Prism (not sure which one applies to the system mentioned): https://irix7.com/techpubs/007-4701-004.pdf https://irix7.com/techpubs/007-4772-002.pdf > (In reply to Joshua Kinard from comment #3) > > > This is an IP35 system class machine. Should only be a single 2U unit. If referring to "double 2U", then that sounds like two separate units linked together using CrayLink cables. That would mean there's really 8 CPUs available. > > Having asked the owner of the hardware, this indeed seems to be the case. > Question, though - can such units be linked directly? When this cluster was > operational they were connected via a NUMAlink hub, which however broke down > quite a while ago. Oh, that is a very good question. IP35 racks were built out of "bricks", at least the Origin 3000 ones were. C-Brick was for CPUs, P-Bricks were power, N-Bricks were NUMA, and X-Bricks were XIO, as far as I recall. I *think* the Origin 300 and 350 can directly NUMALink together, though, but you need two cables and they're crosslinked in an 'X' pattern on the back (right port to left port, vice-versa, etc). This would give you a maximum of 8 CPUs and 16GB RAM. I think the NUMALink Hub is if you want to expand beyond that configuration. This is actually a setup I have never been able to use or test, though. > (In reply to Joshua Kinard from comment #6) > > > In the space between the upper and lower nodes (just under the MMSC display), there's a spot where a "GIGAChannel" box goes -- really just a converted Origin 200 chassis -- any idea if that is included as well? > > I think so! Or at least there is a box with several large drive bays, which > looks similar to what an image search for "SGI gigachannel" has shown me, in > the space you have indicated. Yeah, looks like I got my terminology mixed up a bit. GigaChannel is the module that looks like an Origin 200 that lets you link up multiple Origin 200 towers. In the case of an Origin 2000 or Onyx2 rack, though, that drive bay is really just a spot for additional SCSI disk drives. You might want to fetch the "Silicon Graphics Onyx2 Rackmount Owner’s Guide" manual from this archive site here (go up one level for all SGI manuals ever): https://irix7.com/techpubs/007-3457-005.pdf That is the the goto manual for an SGI Onyx2 rackmount system, to give an idea of what we'd be getting into. Even the bare racks are a hard-to-find piece of kit. And for reference, the Origin 300 manual is here, which should cover the NUMALink question somewhere: https://irix7.com/techpubs/007-4240-002.pdf > (In reply to Joshua Kinard from comment #12) > > > The install I use on my Octane is almost seven years old now. > > Which reminds me - there are also an Octane and an Origin 200 (both > deskside, with the plastic bits on the latter's case quite severely > damaged)! I haven't mentioned them before because their CPU specs look quite > pitiful comparing to the rack machines, that said if the choice is between a > power-hungry Onyx2, an unsupported Origin 300 and an Octane... Plus we might > actually be able to get away with keeping the Octane running on-site. Yeah, finding Origin 200's in good condition is always a challenge. There is something with the chemical makeup of the plastics SGI used in both the Origin 200 and the smaller O2 systems, where they get extremely brittle with age and literally will shatter with the slightest impact or suffer almost a form of dry rot. The metal chassis underneath is pretty solid, though, so while it may look ugly, it would function. Spec-wise, though, yes, Origin 200's are on the slower side. The maximum specification for those is an R12000 @ 350MHz CPU I think. Maybe 400MHz, though I'd have to go and look at the one I have in a back room. My own experience, though, is that for the entire IP27 hardware class, Origin 200's are the most stable on Linux. My Onyx2 Deskside has had this weird bug where if you put too much I/O on the SCSI bus, it will lock hard. This is still current up to the 5.4 kernels. I haven't tried anything beyond 5.4 yet, though, to see if the issue is still present. I also haven't ruled out that it's bad hardware in the midplane board, where the two XBOW chips are at that handle all system I/O. Could be bad heat dissipation on those chips due to the thermal glue on the heatsink being too old. If that's the case, I have yet to figure out how to extract the midplane board to work on it. That's the kinda stuff there ain't much documentation on because that was reserved to SGI field technicians. The Octane is the most stable of the SGI machines overall, IMHO. It's my primary workhorse for my MIPS work. Currently have two working and just acquired a third one that needs some work (bad PSU that likely needs recapping). Downside with it is it lacks any remote management functions whatsoever, so power cycles and all will require a human to manually reboot it from the front panel. What worse, is there is a bug in Linux, likely one I inadvertently created (because some of the code I've written for the thing is in the kernel), where after a soft-reboot, it will not post back to PROM. I have been slowly experimenting with trying to find it in Linux 5.4, but odds are likely I just need to give up and move forward to 5.10 or 5.15 LTS and see if the newer code in mainline still exhibits the problem. If it does, that's going to be another factor that needs fixing to make the thing reliable from a remote power standpoint. Because, when it locks up during the reboot cycle, you either have to yank the power cable or you have to hit the reset button multiple times in a row to give the HEART system ASIC chip a jump-start. On top of that....I am literally now in the process of trying to find a recent aggravating Kernel oops in __update_load_avg_se() that has returned when the system is under heavy compilation load. First ran into it in 5.4.17x, but then it went away, and now in 5.4.186, it's back again. Somewhere between 5.4.180 and 5.4.186, I hope, it got reintroduced. So, yeah, fun times. > > > (In reply to Joshua Kinard from comment #14) > > > Ideally, that's why it'd be nice for people to get interested in hacking on IP35 support so we can get at least an Origin 300 or Origin 350 working. Small size (2U), reasonable power consumption, reasonable specs for dev tasks, etc. > > I MIGHT perhaps be able to get one of the Origin 300 base units powered up > and connected to the Internet, if it helps. Or possibly arrange for it/them > to be donated to one of the Linux/MIPS people working on this, if THAT > helps. Or at the very least stash them away, unpowered, somewhere out of the > way until they can be used. As far as I know, outside of Gentoo, there are only two other people with the kernel knowledge to tackle IP35 support. I have some kernel spelunking skills, but I am an abject amateur considered to them. One is the current Linux/MIPS kernel maintainer, Thomas Boegendorfer, and in a recent e-mail exchange, he mentioned he was looking for an Origin 300 or 350 to work on, so I can reach back out to him and see if he'd be interested. He's located in Germany, so getting it to him from the UK would be trivial I think. Let me know if this is workable. The other guy, Alastair, is US stateside somewhere, though I lost contact with him a few years ago. He was able to do an initial bring-up of Linux-4.0 to 4.2 on his Origin 350, but one of the blockers was extracting the ethernet MAC address from an EEPROM chip hidden behind the L1 lights-out device that has stumped quite a few people for a long time. As far as I know, his interest in SGI gear was a tangential motivation from another hardware project he was working on at the time. Not sure if he ever got any farther, though. (In reply to Marek Szuba from comment #15) > Before I get distracted - okay, so we've had all the (IMHO very useful) > feedback regarding MIPS hardware. Not a word from the ia64 camp, though... > Anyone? Bueller? Bueller? There's really not an ia64 camp. We've got an ia64 system that works fine. We just don't have the manpower or expertise to investigate the myriad of ia64-specific bugs. (In reply to Joshua Kinard from comment #16) > As far as I know, outside of Gentoo, there are only two other people with > the kernel knowledge to tackle IP35 support. I have some kernel spelunking > skills, but I am an abject amateur considered to them. One is the current > Linux/MIPS kernel maintainer, Thomas Boegendorfer, and in a recent e-mail > exchange, he mentioned he was looking for an Origin 300 or 350 to work on, > so I can reach back out to him and see if he'd be interested. He's located > in Germany, so getting it to him from the UK would be trivial I think. Let > me know if this is workable. If the goal of this donation ends up to be having support for this hardware added to the Linux kernel, this is definitely an option! Could you contact Thomas, please? Hopefully shipping arrangements will not turn out to be too difficult. On the Gentoo side of things however, I guess we are simply not interested. Will close this as WONTFIX soon if no-one says anything to the contrary. (In reply to Marek Szuba from comment #18) > (In reply to Joshua Kinard from comment #16) > > > As far as I know, outside of Gentoo, there are only two other people with > > the kernel knowledge to tackle IP35 support. I have some kernel spelunking > > skills, but I am an abject amateur considered to them. One is the current > > Linux/MIPS kernel maintainer, Thomas Boegendorfer, and in a recent e-mail > > exchange, he mentioned he was looking for an Origin 300 or 350 to work on, > > so I can reach back out to him and see if he'd be interested. He's located > > in Germany, so getting it to him from the UK would be trivial I think. Let > > me know if this is workable. > > If the goal of this donation ends up to be having support for this hardware > added to the Linux kernel, this is definitely an option! Could you contact > Thomas, please? Hopefully shipping arrangements will not turn out to be too > difficult. I will reach out to him and CC you and we can pursue it from there. Thanks! (In reply to Marek Szuba from comment #19) > On the Gentoo side of things however, I guess we are simply not interested. > Will close this as WONTFIX soon if no-one says anything to the contrary. I am almost curious how much it'd be to disassemble just the rack itself and ship that to the states for me to keep in storage (that I honestly don't have), if only because finding complete SGI Origin 2k/Onyx2 racks are really hard these days. Ah, cest la vie... |
