Qemu and other emulators

This note covers setting up Qemu on an x86-based development system running Linux. This allows native (rather than cross) development tools to be run, which can be useful where the target system has performance/resource issues (e.g. some ARM systems), is not run natively due to company policy (older versions of Microsoft Windows) or is quite simply unavailable at a reasonable price (e.g. SGI MIPS systems). It also briefly mentions User Mode Linux and the Hercules emulator for IBM zSeries mainframes despite the fact that these are not particularly relevant to Free Pascal, it does not consider x86-on-x86 virtualisation systems such as VMware.

The Host System

In the current case, the host is a Compaq rack-mount server running at around 3GHz. It has two internal drive cages, the first is connected to a RAID controller and is used for the host operating system and tools, the second is connected to a SCSI controller and contains 6x discs each of which is used for a different guest system.

The host IP address is 192.168.1.22 and the system is named pye-dev-07, the default gateway and name server are on 192.168.1.1. Guest systems are on the 192.168.22.x subnet and are named pye-dev-07a (192.168.22.16), pye-dev-07b (192.168.22.17) and so on, they have their own gateway 192.168.22.1 which is known to the site router and firewalls.

The host operating system is Debian "Squeeze", the host normally runs headless and may be accessed by SSH, X using XDMCP, or VNC. The display manager is gdm since this has a better XDMCP implementation than the alternatives, however in practice graphical login is most often handled by VNC.

The following guests are implemented:

pye-dev-07a

Debian on ARM (big-endian) using Qemu

pye-dev-07b

Debian on ARM (little-endian, armel) using Qemu

pye-dev-07c

Debian on MIPS (little-endian, mipsel) using Qemu

pye-dev-07d

Slackware 13.37 using User Mode Linux

pye-dev-07e

Windows 2K using Qemu

pye-dev-07f

Debian on zSeries using the Hercules emulator

In general, multiple guests can run simultaneously although this has not been exhaustively tested recently.

In the case of Linux the guest systems are each installed on an 18Gb disc, in the case of Windows a 36Gb disc is used. Each disc is assigned a label using e2label (arm, armel and so on), so that the startup script can mount it by name irrespective of which drive cage slot it's in.

Debian Guest using Qemu

Select a suitable Debian mirror and version, for example

 http://ftp.de.debian.org/debian/dists/squeeze/main/...

Fetch a kernel and initrd image for Debian Squeeze, as below.

For ARM (big-endian):

For ARM (little-endian):

 .../main/installer-armel/current/images/versatile/netboot/vmlinux-2.6.32-5-versatile
 .../main/installer-armel/current/images/versatile/netboot/initrd.gz

In addition for this architecture you also need:

 http://people.debian.org/~aurel32/qemu/armel/initrd.img-2.6.32-5-versatile

For MIPS (little-endian):

.../main/installer-mipsel/current/images/malta/netboot/vmlinux-2.6.32-5-4kc-malta
.../main/installer-mipsel/current/images/malta/netboot/initrd.gz

Copy these to the disc reserved for the guest, e.g. /export/mipsel.

Create a filesystem for Qemu, e.g.:

# qemu-img create -f qcow mipsel_hda.img 16G

Expect that to round up to around 17.1 Gb, if it doesn't then experiment. Start Qemu, telling it what kernel, initrd and filesystem to use:

For ARM (big-endian):

For ARM (little-endian):

# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.gz \
-hda armel_hda.img -append root=/dev/ram

Note that the above command requires X access, e.g. ssh with the -X option.

For MIPS (little-endian):

# qemu-system-mipsel -M malta -kernel vmlinux-2.6.32-5-4kc-malta -initrd initrd.gz \
-hda mipsel_hda.img -append "root=/dev/ram console=ttyS0" -nographic

Install the guest operating system as usual, splitting the disc into 16.5Gb for / with the remainder (around 700Mb) as swap. This will be slow, 8 or 9 hours is not implausible, so make sure that nobody's about to turn off your mains or disconnect you from the Internet.

Don't worry if it tells you it's not installing a loader- it's not needed since the kernel and initrd are loaded into memory by the host.

Boot the operating system and set network addresses etc. Use 192.168.22.16 or similar, with a gateway of 192.168.22.1.

For ARM (big-endian):

For ARM (little-endian):

# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.32-5-versatile \
        -initrd initrd.img-2.6.32-5-versatile -hda armel_hda.img -append "root=/dev/sda1"

For MIPS (little-endian):

# qemu-system-mipsel -M malta -kernel vmlinux-2.6.32-5-4kc-malta \
        -hda mipsel_hda.img -append "root=/dev/sda1 console=ttyS0" -nographic

Finally, you should be able to boot the operating system with an operational network. This relies on having /etc/qemu-ifup and /etc/qemu-ifdown files (see below), and passes additional parameters to them in shell variables. In outline:

For ARM (big-endian):

For ARM (little-endian):

# qemu-system-arm -M versatilepb -m 256 -hda armel_hda.img \
        -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile \
        -append 'root=/dev/sda1' \
        -net nic,macaddr=00:16:3e:00:00:01 -net tap,ifname=tun1

For MIPS (little-endian):

# qemu-system-mipsel -M malta -m 256 -hda mipsel_hda.img \
        -kernel vmlinux-2.6.32-5-4kc-malta \
        -append 'root=/dev/sda1 console=ttyS0' -nographic \
        -net nic,macaddr=00:16:3e:00:00:02 -net tap,ifname=tun2

Remember that if you change the network interface type or MAC address you will probably need to delete entries from the guest's /etc/udev/rules.d/z??_persistent-net.rules file.

Windows 2K Guest using Qemu

Use dd to save a .iso image of the installation CD. Create a filesystem image:

# qemu-img create -f qcow2 win2k.img 32G

Boot using startup script as below. Note that this must specify a non-default network card, since Qemu's current (as of 2011) default is not supported by Windows 2K.

TODO: run with kernel support module.

Common Qemu startup, ifup and ifdown scripts

There is much commonality irrespective of whether the guest is running Linux or Windows.

First startup script (e.g. /export/C):

#!/bin/sh

mount -L mipsel
cd /export/mipsel
. ./C-2

Second startup script for ARM (big-endian):

Second startup script for ARM (little-endian):

#!/bin/sh

# Routine startup of a Qemu guest relies on (the host) running /etc/qemu-ifup
# to condition ARP, forwarding etc.

QEMU_ID=1
QEMU='qemu-system-arm -M versatilepb'
QEMU_RAM='-m 256'
QEMU_HD='-hda armel_hda.img'
QEMU_CD=
QEMU_BOOT="-kernel vmlinuz-2.6.32-5-versatile -append 'root=/dev/sda1' -initrd initrd.img-2.6.32-5-versatile"
# QEMU_MONITOR='-monitor stdio -nographic'
# QEMU_MONITOR='-nographic'
QEMU_VGA=
VNC_ID=$(($QEMU_ID+1))
# QEMU_VNC="-vnc :$VNC_ID -k en-gb"
QEMU_VNC=
QEMU_NET="-net nic,macaddr=00:16:3e:00:00:0$QEMU_ID -net tap,ifname=tun$QEMU_ID"
QEMU_GUEST_IP_ADDRESS=192.168.22.17
QEMU_GUEST_IP_GATEWAY=192.168.22.1
QEMU_HOST_GATEWAY_IF=eth1
export QEMU_GUEST_IP_ADDRESS QEMU_GUEST_IP_GATEWAY QEMU_HOST_GATEWAY_IF

echo \* $QEMU $QEMU_RAM $QEMU_HD $QEMU_CD $QEMU_BOOT \
        $QEMU_MONITOR $QEMU_VGA $QEMU_NET $QEMU_VNC

screen -S QEMU_$QEMU_ID \
sh -c "$QEMU $QEMU_RAM $QEMU_HD $QEMU_CD $QEMU_BOOT \
        $QEMU_MONITOR $QEMU_VGA $QEMU_NET $QEMU_VNC"

cd ..

Second startup script for MIPS (little-endian):

#!/bin/sh

# Routine startup of a Qemu guest relies on (the host) running /etc/qemu-ifup
# to condition ARP, forwarding etc.

QEMU_ID=2
QEMU='qemu-system-mipsel -M malta'
QEMU_RAM='-m 256'
QEMU_HD='-hda mipsel_hda.img'
QEMU_CD=
QEMU_BOOT="-kernel vmlinux-2.6.32-5-4kc-malta -append 'root=/dev/sda1 console=ttyS0'"
# QEMU_MONITOR='-monitor stdio -nographic'
QEMU_MONITOR='-nographic'
QEMU_VGA=
VNC_ID=$(($QEMU_ID+1))
# QEMU_VNC="-vnc :$VNC_ID -k en-gb"
QEMU_VNC=
QEMU_NET="-net nic,macaddr=00:16:3e:00:00:0$QEMU_ID -net tap,ifname=tun$QEMU_ID"
QEMU_GUEST_IP_ADDRESS=192.168.22.18
QEMU_GUEST_IP_GATEWAY=192.168.22.1
QEMU_HOST_GATEWAY_IF=eth1
export QEMU_GUEST_IP_ADDRESS QEMU_GUEST_IP_GATEWAY QEMU_HOST_GATEWAY_IF

echo \* $QEMU $QEMU_RAM $QEMU_HD $QEMU_CD $QEMU_BOOT \
       $QEMU_MONITOR $QEMU_VGA $QEMU_NET $QEMU_VNC

screen -S QEMU_$QEMU_ID \
sh -c "$QEMU $QEMU_RAM $QEMU_HD $QEMU_CD $QEMU_BOOT \
       $QEMU_MONITOR $QEMU_VGA $QEMU_NET $QEMU_VNC"

cd ..

Second startup script for Windows:

#!/bin/sh

# Routine startup of a Qemu guest relies on (the host) running /etc/qemu-ifup
# to condition ARP, forwarding etc.

QEMU_ID=4
QEMU=qemu
QEMU_RAM='-m 256'
QEMU_HD='-hda win2k.img'
QEMU_CD='-cdrom Windows2k-SP4.iso'
QEMU_BOOT='-boot c'
QEMU_MONITOR='-monitor stdio'
QEMU_VGA='-vga cirrus'
VNC_ID=$(($QEMU_ID+1))
QEMU_VNC="-vnc :$VNC_ID -k en-gb"
QEMU_NET="-net nic,macaddr=00:16:3e:00:00:0$QEMU_ID,model=rtl8139 -net tap,ifname=tun$QEMU_ID"
QEMU_GUEST_IP_ADDRESS=192.168.22.20
QEMU_GUEST_IP_GATEWAY=192.168.22.1
QEMU_HOST_GATEWAY_IF=eth1
export QEMU_GUEST_IP_ADDRESS QEMU_GUEST_IP_GATEWAY QEMU_HOST_GATEWAY_IF

echo \* $QEMU $QEMU_RAM $QEMU_HD $QEMU_CD $QEMU_BOOT \
        $QEMU_MONITOR $QEMU_VGA $QEMU_NET $QEMU_VNC

screen -S QEMU_$QEMU_ID \
$QEMU $QEMU_RAM $QEMU_HD $QEMU_CD $QEMU_BOOT \
        $QEMU_MONITOR $QEMU_VGA $QEMU_NET $QEMU_VNC

cd ..

/etc/ifup (for both Linux and Windows):

#!/bin/bash

# if-up file for qemu, heavily cribbed from the command sequence embedded in
# User Mode Linux. MarkMLl.

echo Running /etc/qemu-ifup $1 $2...

# For compatibility with UML the only parameter here is $1 which is the
# interface name. I've put in a reference to $2 so we can see it if anything
# changes.

# I'm going to assume that qemu is always run by root. This is fairly
# reasonable since it allows guest OSes to be fired up which themselves might
# give access to confidential data etc. if compromised.

# Here's my equivalent to the host-side UML setup for Qemu. We're hamstrung
# here by the fact that the emulator is not telling us what IP address it's
# trying to enable, there isn't a 1:1 correspondence between IP addresses and
# interfaces since the latter depends on the order the sessions are started.
#
# As a hack, assume that the caller exports QEMU_GUEST_IP_ADDRESS (e.g.
# 192.168.17.16), QEMU_GUEST_IP_GATEWAY (e.g. 192.168.17.1) and
# QEMU_HOST_GATEWAY_IF (e.g. eth0).

echo \* modprobe tun
modprobe tun

echo \* ifconfig $1 $QEMU_GUEST_IP_GATEWAY netmask 255.255.255.255 up
ifconfig $1 $QEMU_GUEST_IP_GATEWAY netmask 255.255.255.255 up

X=`cat /proc/sys/net/ipv4/ip_forward`
if [ "$X" == "0" ]; then

  # Use either this...

#  echo Global forwarding is not enabled. Please refer to the administrator
#  echo responsible for this machine, enabling it might be a security hazard.

  # ...or this.

  echo Forcibly enabling global forwarding, note that this might be a security hazard.
  echo \* echo 1 \> /proc/sys/net/ipv4/ip_forward
  echo 1 > /proc/sys/net/ipv4/ip_forward
  X=`cat /proc/sys/net/ipv4/ip_forward`
  if [ "$X" == "0" ]; then
    echo Unable to enable global forwarding. Please refer to the administrator
    echo responsible for this machine.
  fi
fi

echo \* route add -host $QEMU_GUEST_IP_ADDRESS dev $1
route add -host $QEMU_GUEST_IP_ADDRESS dev $1

echo \* echo 1 \> /proc/sys/net/ipv4/conf/$1/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/$1/proxy_arp
X=`cat /proc/sys/net/ipv4/conf/$1/proxy_arp`
if [ "$X" == "0" ]; then
  echo -n Retrying
  while [ "$X" == "0" ]; do
    sleep 1
    echo -n .
    echo 1 > /proc/sys/net/ipv4/conf/$1/proxy_arp
    X=`cat /proc/sys/net/ipv4/conf/$1/proxy_arp`
  done
  echo OK
fi

echo \* arp -Ds $QEMU_GUEST_IP_ADDRESS $1 pub
arp -Ds $QEMU_GUEST_IP_ADDRESS $1 pub

echo \* arp -Ds $QEMU_GUEST_IP_ADDRESS $QEMU_HOST_GATEWAY_IF pub
arp -Ds $QEMU_GUEST_IP_ADDRESS $QEMU_HOST_GATEWAY_IF pub

# Set up experimental UDP proxies. Depending on the protocol of interest
# messages in one or both directions might need to be relayed.
#
# UDP port 79 is used for Dialarm signals, a unidirectional proxy is
# adequate for this but detection of hosts changing state (i.e. being
# added to or removed from the population of cooperating systems) is far
# more responsive if a bidirectional proxy is available.

PROXY_ID=1
case "$1" in
  tun1)
    PROXY_ID=2
    ;;
  tun2)
    PROXY_ID=3
    ;;
  tun3)
    PROXY_ID=4
    ;;
  tun4)
    PROXY_ID=5
    ;;
  tun5)
    PROXY_ID=6
    ;;
esac

# echo \* udp-broadcast-relay -f $PROXY_ID 79 $QEMU_HOST_GATEWAY_IF $1
# /usr/local/src/udp-broadcast-relay/udp-broadcast-relay-0.3/udp-broadcast-relay \
        -f $PROXY_ID 79 $QEMU_HOST_GATEWAY_IF $1

# Alternatively use this one which is oriented towards IP addresses
# rather than interfaces.

# Note attempt to counteract any niceness applied to Qemu itself.

ps ax | grep 'udp-proxy[ ]-z 79 ' >/dev/null 2>&1
if [ $? != 0 ]; then
  echo \* udp-proxy -z 79 $QEMU_GUEST_IP_ADDRESS
  /usr/bin/nice --adjustment=20 /usr/local/src/udp-proxy/udp-proxy -z 79 $QEMU_GUEST_IP_ADDRESS
else
  echo \* Already running udp-proxy -z 79 $QEMU_GUEST_IP_ADDRESS
fi

# echo \* udp-proxy -z 13264 $QEMU_GUEST_IP_ADDRESS
# /usr/local/src/udp-proxy/udp-proxy -z 13264 $QEMU_GUEST_IP_ADDRESS

echo .../qemu/qemu-ifup completed.

/etc/ifdown (for both Linux and Windows):

#!/bin/sh

echo \* route del -host $QEMU_GUEST_IP_ADDRESS dev $1
route del -host $QEMU_GUEST_IP_ADDRESS dev $1

echo \* ifconfig $1 down
ifconfig $1 down

In actual fact, these operations were cribbed from User Mode Linux (below) where they are embedded inside a host library.

Slackware Guest using User Mode Linux

User Mode Linux runs a guest kernel as a standard program, i.e. there is no emulation or virtualisation involved. The guest kernel can be allocated either physical discs or filesystems contained in files.

Put a .iso corresponding to a recent Slackware DVD in /export/uml. Unpack the initrd using zcat and cpio, save it as an ext3 image initrd_unpacked. Create an empty file root_fs_slackware which will be partitioned and formatted.

Use the sources from e.g. a recent Slackware to compile kernel plus modules with ARCH=um using a suffix -uml. Save the kernel to /export/uml/linux, install the modules and then copy them into the initrd filesystem.

Boot the UML kernel, telling it to use the initrd image and DVD iso:

# ./linux ubd0=initrd_unpacked ubd1=root_fs_slackware fake_ide ubd2r=slackware-13.37-install-dvd.iso rw

Run fdisk and setup as normal, you might need to tell it to install to /dev/ubd1 and use /dev/ubd2 for source. Finally, copy the modules onto the target filesystem.

When complete start up like this:

# Routine startup of a UML guest relies on (the host) running /usr/lib/uml/uml_net
# to condition ARP, forwarding etc.

echo \* ./linux ubd0=initrd_unpacked ubd1=root_fs_slackware fake_ide ubd2r=slackware-13.37-install-dvd.iso \
        root=/dev/ubdb1 eth0=tuntap,,,192.168.1.22

screen -S UML_3 \
./linux ubd0=initrd_unpacked ubd1=root_fs_slackware fake_ide ubd2r=slackware-13.37-install-dvd.iso \
        root=/dev/ubdb1 eth0=tuntap,,,192.168.1.22

cd ..

Note that this is usually run from an X session, since the multiple virtual consoles appear as separate xterms.

Debian Guest using Hercules

Hercules is a commercial-grade emulator for IBM mainframes. Once the emulator is running, enter

ipl 120

to boot Linux from device 120. Hopefully SSH will be operational so you won't need to interact with the console, but if you do then prefix each line that is to go to the guest operating system (i.e. rather than to the console itself) with a dot.

Refer to the URL in the script below for more details.

Startup:

#!/bin/sh

# PREREQUISITE: Boot with  ipl 120

# Note that this makes no attempt to support IPv6.

iptables -t nat -A POSTROUTING -o eth1 -s 192.168.22.0/24 -j MASQUERADE
iptables -A FORWARD -s 192.168.22.0/24 -j ACCEPT
iptables -A FORWARD -d 192.168.22.0/24 -j ACCEPT
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp

# http://www.josefsipek.net/docs/s390-linux/hercules-s390.html

screen -S HERC_5 \
hercules

cd ..

Configuration:

CPUSERIAL 000069        # CPU serial number
CPUMODEL  9672          # CPU model number
MAINSIZE  256           # Main storage size in megabytes
XPNDSIZE  0             # Expanded storage size in megabytes
CNSLPORT  3270          # TCP port number to which consoles connect
NUMCPU    2             # Number of CPUs
LOADPARM  0120....      # IPL parameter
OSTAILOR  LINUX         # OS tailoring
PANRATE   SLOW          # Panel refresh rate (SLOW, FAST)
ARCHMODE  ESAME         # Architecture mode ESA/390 or ESAME

# .-----------------------Device number
# |     .-----------------Device type
# |     |       .---------File name and parameters
# |     |       |
# V     V       V
#---    ----    --------------------

# console
001F    3270

# terminal
0009    3215

# reader
000C    3505    /export/zlinux/rdr/kernel.debian /export/zlinux/rdr/parmfile.debian /export/zlinux/rdr/initrd.debian autopad eof

# printer
000E    1403    /export/zlinux/prt/print00e.txt crlf

# dasd
0120    3390    /export/zlinux/dasd/3390.LINUX.0120
0121    3390    /export/zlinux/dasd/3390.LINUX.0121

# tape
0581    3420

# network                               s390     realbox
# 0A00,0A01  CTCI -n /dev/net/tun -t 1500 10.1.1.2 10.1.1.1
0A00,0A01  CTCI -n /dev/net/tun -t 1500 192.168.22.21 192.168.1.22

Note that the guest network is configured as SLIP. Best not fooled with.

Relative Performance

(TBD)

Further Reading

http://www.aurel32.net/info/debian_arm_qemu.php http://www.aurel32.net/info/debian_mips_qemu.php http://en.wikibooks.org/wiki/QEMU/Windows_XP http://user-mode-linux.sourceforge.net/ http://www.josefsipek.net/docs/s390-linux/hercules-s390.html