some notes on IBM OpenPower LPARs and VIO channels

I thought I’d share something I just learned about VIO disks by a process of experimentation, based upon some comments I found in a wiki, but couldn’t confirm anyplace in documentation.

You can run:

mkvdev -vdev lv_cayenne_home -vadapter vhost9 -dev dev_cayenne_h mkvdev -vdev lv_athabasca_sl -vadapter vhost9 -dev dev_athab_sl

I.e. set two virtual disks to be on the same host/server adapter. Linux, sees, properly:

root@cayenne:~ # dmesg | grep 'Attached scsi' Attached scsi disk sda at scsi0, channel 0, id 1, lun 0 Attached scsi disk sdb at scsi1, channel 0, id 1, lun 0 Attached scsi disk sdc at scsi2, channel 0, id 1, lun 0 Attached scsi disk sdd at scsi3, channel 0, id 1, lun 0 Attached scsi disk sde at scsi3, channel 0, id 2, lun 0

This is a nice thing, because you can then add disks to running LPARs, and get the SCSI bus rescanned by doing: root@cayenne:~ # echo >/sys/bus/scsi/devices/3:0:1:0/rescan

that means you can also do things like perform rescue operations on a sick LPAR’s root disk by mounting it on another LPAR, and you can do this without rebooting.

In particular, if you have provisioned your VIO server with enough spare SCSI server adapters, you can easily add LPARs without restarting the VIO server.

The instructions we read suggesting creating a new VIO channel per disk, which seemed dumb. We much prefer to have a seperate LV on the VIO server per partition, because ideally we can resize them as well.