Linux on SUN Enterprise 6500
I am running Linux on Sun Enterprise 6500 server at home. I found very few information about this platform running Linux, so I share own experience.
Update: Not anymore! Sun 6500 is sold to people in masks who paid in cash and left me without single Sun computer at home. So I had to shop and now I have Sun Enterprise 10000 StarFire instead. Sorry :)
Hardware
I have two similar products in house: Enterprise 4500 and Enterprise 6500. They should be very similar except number of system boards allowed (read: "maximum CPUs and Memory") but in practice I found some differences in installing Linux. As rule of thumb - having more than 4G of RAM causes trouble.
Exact configuration relies what boards I insert (I have extra boards available) but as rule of thumb 6500 is fully loaded with 30 CPUs (approximately 10 of 400Mhz, 8MB cache and many 336Mhz, 4MB cache) and single IO board. I have also disk box A5200 but missing cables and disks, so it is unused. Just connected single 18GB 10000RPM IBM to external SCSI connector with flat ribbon cable and feeding it with ATX power supply. Not sure about total memory installed yet but soon I get missing SIMMs, so I can load it fully up to 30GB of RAM.
Ultra Enterprise 4500 is running in minimal configuration with few CPUs to conserve power.
Electricity consumption of Enterprise 6500 is between 1KW and 2,2KW without hard disks. I have 3/4 CPU boards switched down and now it takes 1,4KW and generates lot of heat too.
I do not use rack power distribution unit but feed 6500 base directly. I get some warnings related to nonworking fans in rack but it works without problem.
Have no digital camera to make pictures of my own 6500 but some images I grabbed from Internet can be browsed here
Software
It took me while to understand some issues. I tried different Linux distributions and landed with Aurora 2.0. Every one (including Gentoo, Debian) had some problem I had to solve but Aurora was only which installed with all hardware included without any workarounds.
Gentoo 2006.0
In beginning I was most successful with Gentoo.
Gentoo 2006.0 had problem booting on computers with more than 4G of RAM. Even having SILO argument mem=1024M does not help. Extra memory needs to be disabled from OpenBoot:
{ok} setenv disabled-memory-list 3456789abcdef
Those numbers and letters are board identificators. I leave memory only on boards 0 and 2 running and exclude them from 3...15. Maybe you wonder why I do not name board no 1 here? This is because board no 1 is IO board and should be in slot 1.
To conserve power I also disabled some CPU boards completely for now. This can be done in similar way:
{ok} setenv disabled-board-list 56789abcdef
CPU Boards 0,2,3 and 4 with total 8 CPUs are enough for now. So after booting Gentoo up situation:
livecd ~ # free
total used free shared buffers cached
Mem: 2056808 74032 1982776 0 5456 28728
-/+ buffers/cache: 39848 2016960
Swap: 0 0 0
livecd ~ #
livecd ~ # cat /proc/cpuinfo
cpu : TI UltraSparc II (BlackBird)
fpu : UltraSparc II integrated FPU
promlib : Version 3 Revision 2
prom : 3.2.30
type : sun4u
ncpus probed : 8
ncpus active : 8
Cpu0Bogo : 671.74
Cpu0ClkTck : 000000001406f400
Cpu1Bogo : 671.74
Cpu1ClkTck : 000000001406f400
Cpu4Bogo : 671.74
Cpu4ClkTck : 000000001406f400
Cpu5Bogo : 671.74
Cpu5ClkTck : 000000001406f400
Cpu6Bogo : 671.74
Cpu6ClkTck : 000000001406f400
Cpu7Bogo : 671.74
Cpu7ClkTck : 000000001406f400
Cpu8Bogo : 671.74
Cpu8ClkTck : 000000001406f400
Cpu9Bogo : 671.74
Cpu9ClkTck : 000000001406f400
MMU Type : Spitfire
State:
CPU0: online
CPU1: online
CPU4: online
CPU5: online
CPU6: online
CPU7: online
CPU8: online
CPU9: online
livecd ~ #
You also might be interested to know that you can clear disabled board list in OpenBoot with:
set-default disabled-board-list
With all boards enabled it looks like this:
[root@e6500 ~]# cat /proc/cpuinfo cpu : TI UltraSparc II (BlackBird) fpu : UltraSparc II integrated FPU promlib : Version 3 Revision 2 prom : 3.2.30 type : sun4u ncpus probed : 30 ncpus active : 30 D$ parity tl1 : 0 I$ parity tl1 : 0 Cpu0Bogo : 669.69 Cpu0ClkTck : 000000001406f400 Cpu1Bogo : 669.69 Cpu1ClkTck : 000000001406f400 Cpu4Bogo : 669.69 Cpu4ClkTck : 000000001406f400 Cpu5Bogo : 669.69 Cpu5ClkTck : 000000001406f400 Cpu6Bogo : 669.69 Cpu6ClkTck : 000000001406f400 Cpu7Bogo : 669.69 Cpu7ClkTck : 000000001406f400 Cpu8Bogo : 669.69 Cpu8ClkTck : 000000001406f400 Cpu9Bogo : 669.69 Cpu9ClkTck : 000000001406f400 Cpu10Bogo : 669.69 Cpu10ClkTck : 000000001406f400 Cpu11Bogo : 669.69 Cpu11ClkTck : 000000001406f400 Cpu12Bogo : 669.69 Cpu12ClkTck : 000000001406f400 Cpu13Bogo : 669.69 Cpu13ClkTck : 000000001406f400 Cpu14Bogo : 669.69 Cpu14ClkTck : 000000001406f400 Cpu15Bogo : 669.69 Cpu15ClkTck : 000000001406f400 Cpu16Bogo : 669.69 Cpu16ClkTck : 000000001406f400 Cpu17Bogo : 669.69 Cpu17ClkTck : 000000001406f400 Cpu18Bogo : 669.69 Cpu18ClkTck : 000000001406f400 Cpu19Bogo : 669.69 Cpu19ClkTck : 000000001406f400 Cpu20Bogo : 669.69 Cpu20ClkTck : 000000001406f400 Cpu21Bogo : 669.69 Cpu21ClkTck : 000000001406f400 Cpu22Bogo : 669.69 Cpu22ClkTck : 000000001406f400 Cpu23Bogo : 669.69 Cpu23ClkTck : 000000001406f400 Cpu24Bogo : 669.69 Cpu24ClkTck : 000000001406f400 Cpu25Bogo : 669.69 Cpu25ClkTck : 000000001406f400 Cpu26Bogo : 669.69 Cpu26ClkTck : 000000001406f400 Cpu27Bogo : 669.69 Cpu27ClkTck : 000000001406f400 Cpu28Bogo : 669.69 Cpu28ClkTck : 000000001406f400 Cpu29Bogo : 669.69 Cpu29ClkTck : 000000001406f400 Cpu30Bogo : 669.69 Cpu30ClkTck : 000000001406f400 Cpu31Bogo : 669.69 Cpu31ClkTck : 000000001406f400 MMU Type : Spitfire State: CPU0: online CPU1: online CPU4: online CPU5: online CPU6: online CPU7: online CPU8: online CPU9: online CPU10: online CPU11: online CPU12: online CPU13: online CPU14: online CPU15: online CPU16: online CPU17: online CPU18: online CPU19: online CPU20: online CPU21: online CPU22: online CPU23: online CPU24: online CPU25: online CPU26: online CPU27: online CPU28: online CPU29: online CPU30: online CPU31: online [root@e6500 ~]#
Compiling kernel is FAST. Some screenshots:
TODO
All this stuff is in very early stage. I have lot of small problems to solve and yet installing software. One focus of work is to find out ways to influence power consumption and make it lower. Now I cannot run two servers simultaneously as house fuses will burn off. At least then I cannot use even oven anymore at kitchen.
livecd ~ # mount /dev/sda2 /mnt/gentoo/ livecd ~ # chroot /mnt/gentoo/ /bin/bash livecd / # cd /usr/src/linux livecd linux # make menuconfig
bash-3.00# /usr/sbin/psrinfo -v
Status of virtual processor 0 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:51.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:55.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 4 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:55.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 5 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:55.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 6 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:55.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 7 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:55.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 8 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:55.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 9 as of: 07/28/2006 10:24:58
on-line since 07/28/2006 10:21:55.
The sparcv9 processor operates at 336 MHz,
and has a sparcv9 floating point processor.
bash-3.00# prtdiag
System Configuration: Sun Microsystems sun4u 16-slot Sun Enterprise E6500
System clock frequency: 84 MHz
Memory size: 4608Mb
========================= CPUs =========================
Run Ecache CPU CPU
Brd CPU Module MHz MB Impl. Mask
--- --- ------- ----- ------ ------ ----
0 0 0 336 8,0 US-II 10.0
0 1 1 336 8,0 US-II 10.0
2 4 0 336 4,0 US-II 2.0
2 5 1 336 4,0 US-II 2.0
3 6 0 336 4,0 US-II 2.0
3 7 1 336 4,0 US-II 2.0
4 8 0 336 4,0 US-II 2.0
4 9 1 336 4,0 US-II 2.0
========================= Memory =========================
Intrlv. Intrlv.
Brd Bank MB Status Condition Speed Factor With
--- ----- ---- ------- ---------- ----- ------- -------
0 0 1024 Active OK 60ns 4-way A
0 1 1024 Active OK 60ns 4-way A
2 0 256 Active OK 60ns 2-way B
2 1 256 Active OK 60ns 2-way B
3 0 1024 Active OK 60ns 4-way A
3 1 1024 Active OK 60ns 4-way A
========================= IO Cards =========================
Bus Freq
Brd Type MHz Slot Name Model
--- ---- ---- ---------- ---------------------------- --------------------
1 SBus 25 0 QLGC,isp/sd (block) QLGC,ISP1000U
1 SBus 25 1 QLGC,isp/sd (block) QLGC,ISP1000U
1 SBus 25 2 network SUNW,sbus-gem
1 SBus 25 3 SUNW,hme
1 SBus 25 3 SUNW,fas/sd (block)
1 SBus 25 13 SUNW,socal/sf (scsi-3) 501-3060
Detached Boards
===============
Slot State Type Info
---- --------- ------ -----------------------------------------
5 disabled cpu
6 disabled cpu
7 disabled cpu
8 disabled cpu
9 disabled cpu
10 disabled cpu
11 disabled cpu
12 disabled cpu
13 disabled cpu
14 disabled cpu
15 disabled cpu
No failures found in System
===========================
Detected System Faults
======================
Rack Exhaust Fan failure
Detected Fri Jul 28 10:21:51 2006
PROM detected failure
Detected Fri Jul 28 10:21:51 2006
bash-3.00#