Hi!
I was thinking about writing some document "There is an e10k in ebay, should I buy it?". So I write small FAQ here.
--- Decision making support questionsQ: Can I make it work?
A: Probably yes. Starfire has nearly all duplicated hardware and most things even if missing doesn't stop you doing at least something with it.
Q: I have some experience with Sun E3x00..E6x00 computers. Is Starfire same but bigger?
A: No. Stafire has same CPUs, same memory modules, same disk boxes but lot of things are different. Other Sun/Sparc/Solaris know-how is useful but be ready to relearn.
Q: So, what is different?
A: Hardware is incompatible with E3x00...E6x00. No graphics board, no serial port console, no even a chance to connect keyboard. You can run different operating systems on different system boards. System boards can be joined into single computer. No realtime clock in hardware (because unclear how many you need them). Generally everything is better than Exx00 but weights much more and uses more power too. Also you may be surprised to hear that this computer has no CDROM nor chance to connect it. Starfire domains get installed over network boot. Different operating systems tested failed or acted improperly. It is possible to make them work but Solaris is only OS which supports features of this monster.
Q: Why should I buy one?
A: Starfire has unique features you may want to learn or use. Major difference from all other Suns of same era is having ability physically separate system boards and run different operating systems on groups of them. Those are referred as "domains" in all Starfire documentation. You also can remove boards from running domain with price of some 1 second suspend only and also add them back. This computer is very flexible also about power saving and you have still freedom to add tens of CPUs on the fly when needed. One reason might be just owning real piece of history. This is one of most powerful and legendary computers ever made and improved version of Cray at your house. Many records were beaten with Starfire. Hey and it looks good! :)
Q: Why should I do NOT buy it
A: This machine has lot of problems. You may be not able physically fit some of them. Main troubles:
Q: What is an SSP?
A: SSP is a controlling computer without which Starfire is a brick. you cannot use it without this computer. You need to run "power -on" from his command line for example. Normal Starfire setup had two of them. No problem if you have just one.
Q: There is a e10k for sale but no SSP included. Can I make it work?
A: Yes! But you need to get another Sun Sparc computer and install some special software on it. Sparc computer+ Solaris OS+SSP software is all you need. Original SSP computers from Sun are Ultra5, E250 or Netra X1. I see no reason why you cannot use any other Sparc too.
Q: So, how they communicate....SSP and Starfire?
A: Ethernet and IP. When switched "on" Starfire boards issue RARP request and need to get IP for their boards. It is your responsibility to have RARP daemon. Usually it may run on SSP computer. After that Starfire control boards have IP address, they try to download firmware over TFTP protocol from same computer which answered by rarp. Firmware establishes communication with SSP special daemons after few minutes. From this point you make all tasks from SSP console commands. No physical access needed for Starfire. BTW, not all is IP encapsulated there. There are some exceptions - for example resetting board is done using weird specially crafted ethernet packet.
Q: I heard something that Starfire cannot run without special license or "keys"
A: This is true. You need keys and usually it is very difficult to learn if your starfire has them or not. Sellers in ebay are often scrapyard companies who have not much idea about technical details. Even more - they usually wipe disks to make sure you cannot get any customer data. Meanwhile I would suggest you a try buy starfire anyway. Seems like their protection method is weak and in worst case you can make those keys yourself. Not sure about you country laws but in many places you are legally allowed to make reverse engineering if this is necessary to make broken code to work. For example I got starfire all legal ways with licenses (but no keys) but Sun was too lazy to speak with me. So I reverse engineered related code and found it trivial to make keys yourself. Also I wonder if wrong keys work on it. Anyway I think this problem can be worked off if you really want. Sure I never suggest such solutions for business users. Sure always try to get keys from seller of computer first. You need those keys in stage where you generate EEPROM file to be uploaded into starfire domain. Every domain needs one key.
Q: Will it run Linux? NetBSD?
A: Maybe. Actually be prepared that they will not run or you have lot of trouble with it. Currently I have experience running linux on big Suns and only Starfire is place where I never seen it working yet. I get all different reasons why it crashes and have no workaround yet. I think problems can be solved but be warned - this is mainly for Solaris only. All other OS-s are even less supported. I do not want to begin flamewar here but under Solaris you can add boards on the fly on this platform and under linux there is no support for such features under Sparc64. You cannot use some coolest features of this hardware with linux yet.
Q: I may buy one from ebay but not sure how much transport will cost. I heard about $3000.
A: No, you probably get it for much less. I paid around 500 EUR from UK to Estonia. This is from one border of EU to another! I know it is difficult to estimate cost but call into some cargo company and ask for "pallet price". Notice, not DHL, not UPS but some company which moves big goods with big trucks. They carry nothing less than pallet. Those companies do lot of business and always ready to tell price for "pallet". "Pallet" means fixed footprint of goods on wooden pallet and limited to weight of 770kg (1500lbs). When you know price for pallet, this is good indication. They charge you little extra for weight over this one but you can live with it. I used company DFDS Transport and they handled all from pick up in UK till bringing this box down to my village within one week and decent price. One more detail - there are EUR and FIN type of pallets. FIN is bit bigger and more expensive. Because Starfire size exceeds both pallets and weigth limits, FIN pallet price is more proper. Expect something like FIN pallet price + 10..25%.
----- Other stuff
Q: Many CPUs....it should be FAST
A: Not necessarily. Most of jobs just won't scale over many CPUs, so your application probably runs like on single or dual 400Mhz computer with 8MB cache on CPU. You get real speed boost only if you have many concurrent users or your algorithms/apps scale over many CPUs. Most of them don't. You have to consider also that big computer in low-end configuration is usually slower than low end computer in high configuration. Memory latency depends on maximum amount supported by computer, not by actual amount installed for example. It is better to use maximum supported amount to get best results.
Most e10k-s on sale have 8..12 CPUs or similar.
If someone wants to upgrade e10k them main problem is not memory ($4000 per gig on Sun pricelist) and CPUs ($12000 for 400Mhz CPU in Sun) but board itself. It costs $58,000 and you need extra $10,000 memory board to use memory on it. While memory and CPUs are decently prices in ebay then e10k boards are very rare and expensive. And notice - there is no replacement. Just e10k boards go.
Meanwhile yes, e10k uses 13,5KW (never measured myself as my house has 20A main fuse, i.e. around 5KW only.). But e10k can be run in pretty economic way. It runs right now and wattmeter shows 1200W plus I have A5200 box with disks and not counted in. e10k consumes some 10W only per board when they are switched "off" from software. Not so bad. On E6500 they went down to 50% of usual only.
If you want to see how it looks in action, read about running StarFire
I will make some stuff how to make it work. Some QuickStart. I think for now it is enough. My linux running attempts with Aurora 2.0 and Gentoo 2006.x are here:
http://no.spam.ee/~tonu/sparc/gentooe10k.txt
http://no.spam.ee/~tonu/sparc/aurorae10k.txt
I also provide shell access for people with any serious intention. But please understand that this computer is in my bedroom and making noise of jet engine. I do not run it 24x7.