Michael's musings

Thu, 25 Oct 2007

How to extend a file system using LVM/XEN

You have a nifty XEN domU running on top of an LVM backed dom0. You run out of space, and you need to extend. It's not hard, but it can be tedious if you don't want to the domU down.

The hardest part is making sure that there are no processes using the partition you want to extend, as you have to umount it. While ext2online (available in RH kernels, but not, as of 2.6.18, stock) permits you to expand the file system, unfortunately, with xen 3.1.0, extending the Logical Volume underneath does not communicate the new length to the domU.

Usually, people miss the NFS exports. To unexport temporarily (which does screw any client using it), you can use exportfs. You need to give it the IP addresses (possibly with netmasks) that you wish to remove. They have to match. This is still easier than running exportfs -a -u, editing the /etc/exports and running exportfs -a, as it doesn't screw quite so many NFS clients.

In these examples, tbm800 is a guest domU, which happens to also be an NFS server. airbus is the dom0.

tbm800-[~] root 1 #df -H /proj/thintropy/skeletons
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sde5               11G   8.0G   2.1G  80% /proj/thintropy/skeletons

tbm800-[~] root 7 #umount /proj/thintropy/skeletons
umount: /proj/thintropy/skeletons: device is busy
umount: /proj/thintropy/skeletons: device is busy

tbm800-[~] root 9 #exportfs -u 192.168.7.169:/proj/thintropy/skeletons
tbm800-[~] root 10 #umount /proj/thintropy/skeletons

Now, we need to know what the backend block device which was used to mount the device. Unfortunately, xm block-list does not show us anything useful, but it's in the logs:

airbus-[/var/log/xen] root 10 #grep tbm800 xend.log | grep skeletons | grep DevController
[2007-10-09 09:32:02 xend 3682] DEBUG (__init__:1072) DevController: writing
{'domain': 'tbm800', 'frontend': '/local/domain/1/device/vbd/2117', 'dev':
'sde5', 'state': '1', 'params': '/dev/AirbusGroup0/tbm800skeletons', 'mode':
'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to
/local/domain/0/backend/vbd/1/2117.
[2007-10-09 10:43:49 xend 27521] DEBUG (__init__:1072) DevController: writing
{'domain': 'tbm800', 'frontend': '/local/domain/14/device/vbd/2117', 'dev':
'sde5', 'state': '1', 'params': '/dev/AirbusGroup0/tbm800skeletons', 'mode':
'w', 'online': '1', 'frontend-id': '14', 'type': 'phy'} to
/local/domain/0/backend/vbd/14/2117.

Note the 2117. We need to know the domain-ID, as well.

airbus-[/var/log/xen] root 11 #xm list tbm800
Name                                      ID Mem(MiB) VCPUs State   Time(s)
tbm800                                    14      192     1 -b----   7812.5

airbus-[/var/log/xen] root 12 #xm block-detach
Error: 'xm block-detach' requires 2 arguments.

Usage: xm block-detach <Domain> <DevId>

Destroy a domain's virtual block device.
[1]    31565 exit 1     xm block-detach
airbus-[/var/log/xen] root 13 #xm block-detach 14 2117

Now, we can manipulate the LV:

airbus-[/var/log/xen] root 14 #lvextend -L +10G /dev/AirbusGroup0/tbm800skeletons
  Extending logical volume tbm800skeletons to 20.00 GB
  Logical volume tbm800skeletons successfully resized

Next we can attach the block device again. We need to know some details, but they are in the config file:

airbus-[/var/log/xen] root 15 #grep tbm800skeletons /etc/xen/tbm800
        'phy:/dev/AirbusGroup0/tbm800skeletons,sde5,w'

airbus-[/var/log/xen] root 16 #xm block-attach
Error: 'xm block-attach' requires between 4 and 5 arguments.

Usage: xm block-attach <Domain> <BackDev> <FrontDev> <Mode>

Create a new virtual block device.
[1]    31646 exit 1     xm block-attach
airbus-[/var/log/xen] root 17 #xm block-attach 14 phy:/dev/AirbusGroup0/tbm800skeletons sde5 w

On the domU, again, now we can resize things:

tbm800-[~] root 12 #e2fsck -f /dev/sde5
e2fsck 1.40-WIP (14-Nov-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sde5: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sde5: 323289/1310720 files (1.4% non-contiguous), 1979906/2621440 blocks

And finally, the resize operation:

tbm800-[~] root 14 #resize2fs /dev/sde5
resize2fs 1.40-WIP (14-Nov-2006)
Resizing the filesystem on /dev/sde5 to 5242880 (4k) blocks.
The filesystem on /dev/sde5 is now 5242880 blocks long.
tbm800-[~] root 15 #mount -a
tbm800-[~] root 16 #df -H /proj/thintropy/skeletons
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sde5               22G   8.0G    13G  40% /proj/thintropy/skeletons
tbm800-[~] root 17 #exportfs -a

The device is now 22G in size.