Simple VoIP Demo on OKL4

This page describes Ryan Heffernan's work which was used as the ERTOS demo at Techfest 2008, UNSW Open Day and as case study in papers. Please note: This is the result of a student project and is by no means "production quality" code, but it might be helpful to see how OKL4 might be used.

Summary

This is essentially a bare-bones version of voice over IP (VoIP), allowing for two-way voice and text communication across a network using two Gumstix motherboards (with appropriate peripherals). It is not meant to be a robust instant messaging system, but rather a working demo and an application used for benchmarking the RBED scheduler work done by Martin Lawitzky.

The Code

There are two tarballs you will need to run this application: On one hand the application code with unimplemented serial and network functionality. On the other hand the network driver and serial functionality code which we put separately.

Besides the application you obviously need the OKL4 kernel. In particular version 1.5.2 which can be downloaded from Open Kernel Labs wiki under PreviousReleases.

Quick Start

All the bits are here (links above) so here's a step by step guide:

  • Download and untar the OKL4 kernel. Make sure you follow the general build directions in terms of tool chain etc.
  • Download and extract the application tarball in the root directory of the OKL4 tree
  • Run patch -p0 < build-patch-vanilla from the root directory to patch the iguana build files so that the application can be built. You can now build the application if you'd like (see instructions further down), but the results will be pretty disappointing since you don't have the drivers yet.
  • Download and extract the serial and network driver tarball in the root directory.
  • Run patch -p0 < serial-network-patch-vanilla in the root directory to patch the driver functionality into the code.
  • Now you should be able to build successfully following the instructions below.

Components of the Application

Network Server (iguana/network)

This thread utilizes the LwIP network stack (libs/lwip) with some iguana-specific modifications to act as a network server for multiple applications at once. Clients wishing to send or receive from a network must do it through this server, since the implementation of the LwIP stack we used is not safe for multiple threads to access.

Network Driver/Packet Filter (drivers/smc91x, iguana/packet)

The driver handles data transfer to and from the hardware as well as buffer management. Currently all received packets are passed to the network server for demultiplexing, and all transmissions are sent to the driver from the network server. In the future some functions should be added here to avoid uneccessary packet copying (ie. support for ARP queries)

DMA Driver (iguana/dmamon)

The DMA driver can be used by clients to set up certain DMA functionality (not complete) and monitor for interrupts. If requested, the driver will notify clients of DMA interrupts or keep a tally of received interrupts for later use by a client. This has been included for scalability purposes, as it monitors channels for multiple clients at once.

Streaming Audio (iguana/audio_sender)

Although DMA is used to transfer audio data to and from the chip, this server sets up network functionality with the network server and controls when audio data is sent using interrupts from the DMA driver. Also, in the case of dropped packets, which could potentially cause audio buffer underruns (the audio chip never stops making DMA requests, and underruns cause really nasty audio playback), this thread tells the DMA driver to adjust the current DMA read location so that it stays a constant length behind where valid data is being written.

Serial Driver (iguana/vserial, drivers/uart_8250)

The native iguana vserial driver.

Chat Server (iguana/chat)

The chat Server Accepts serial text input and sends it to the network server to be sent to a remote host, and accepts text packets from the network to be output on the serial line.

Building

Before you build a boot image, you need to specify the IP address of the gumstix you will be communicating with. This is embarrassing, but I did not get time to implement a proper dialling protocol, so the IP address of the gumstix that will be receiving your message needs to be hardcoded, and this means you have to build two separate boot images, one for each gumstix (yeah, it's annoying. Sorry!).

Anyway, here's how you do it:

  • iguana/audio_sender/src/main.c: Line 457
    r = network_set_destination(network_thread, 10, 13, 1, 144, PORT, NULL);

This is an IPC to the network server to set the destination IP address for all packets coming from the audio sender. In this case the destination is 10.13.1.144, but you're probably going to want to change that. This IP address reflects the IP address of the gumstix you will be sending and receiving audio from. You can also change the PORT, which is #defined at the top of the file if you want. Just make sure the port you send to is the same as the port that the audio sender on the other gumstix is listening on. Also, there's a weird thing (bug maybe?) in LwIP where only packets from an IP address you are sending to will be accepted (so in this case, if a packet arrives for PORT but its source isn't 10.13.1.144, it will be discarded). I really don't know if this is a bug or on purpose, maybe since LwIP is targeted for embedded devices it enforces this constraint to prevent random unwanted packets from wasting cycles.

There is also a network_set_destination IPC at:

  • iguana/chat/src/chat_server.c: Line 314

Note on IP addresses

Currently, the IP address is set in the user-level driver code.

  • iguana/packet/src/packet.c. The relevant line (195) is:
    IP4_ADDR(&serv_info->ip_addr, 10, 13, 1, driver_mac[5]);

As you can see, the IP is currently hardcoded with a local network mask as the first 3 bytes and the last byte of the MAC address of the network chip as the last byte of the IP. For example, if the MAC address is A6:76:75:06:E1:D0, then the IP address will be 10.13.1.208. I do it this way since it's easy and sufficient for our purposes. LWIP's DHCP functionality can also be used (make sure you are listening on the DHCP port (67 or 68) before you start the DHCP query) during the Network Server initialization to replace the default MAC-based IP, but I currently don't do this as it causes headaches when hardcoding the destination IPs :).

Building a Boot Image

To build an image, execute the following command (assuming you've set up your toolchain correctly - see the README in the source root directory)

  • tools/build.py machine=gumstix project=iguana

Now you have a boot image: build/images/image.boot. See the section on booting to get it running on a gumstix.

Hardware Setup

Ideal Situation

  • Two gumstix connex motherboards
  • Two Audiostix 2 attachments
  • Two NetCF attachments
  • Two tweener boards
  • Two 5V power supplies

The pieces are layered as follows:

Audiostix 2 on the bottom -> tweener -> motherboard ->NetCF on top (do this twice to make two seperate setups)

When we first put this together we found that the power supply box on the tweener board got in the way, so we removed it. Be careful if you do this since the power supply also bridges a connection which should remain intact. Get someone who knows what they are doing with electronics to do this.

It's quite fiddly to get all this together (beware the connectors between boards slipping out unexpectedly) so I recommend using tape to keep different parts together, especially since you will be attaching a power supply, serial cable and ethernet cable to this contraption.

Non-Ideal Situation

When we did this we didn't actually have two netCF boards, so while one of the setups was done exactly like the one above, the other was done differently. Instead of a NetCF we used an old Etherstix which has the ethernet port on its underside that makes it impossible to use a tweener board and thus get no serial interface. Once the demo is loaded onto the gumstix you don't really need a serial interface aside from debugging purposes, but it's good to have anyway. What we did to replace the function of the tweener board was to connect a serial-to-USB converter to the FFUART pins on one of the Audiostix 2 boards and use a USB cable for the interface. Using this board, the setup looks like:

Audiostix 2 + serial-to-USB converter on bottom -> motherboard -> Etherstix on the top.

Connecting the two Hardware Blobs Together

All you need is some kind of simple ethernet switch to connect the two gumstix setups to and assuming you've set up the gumstix boot environment correctly everything should just work when you plug the gumstix in. Since life is never that nice you should probably read the next section on setting up the boot environment.

Gumstix Boot Environment

I'm writing this assuming the reader is completely unfamiliar with the gumstix, so some of this may seem fairly basic to experienced users.

Anyway, you first want to make sure you can get a serial console from the gumstix. Assuming you are using kermit and the serial cable is attached to ttyS0, your ~/.kermrc should look like:

  • set line /dev/ttyS0
  • set speed 115200
  • set reliable
  • fast
  • set carrier-watch off
  • set flow-control none
  • set prefixing all
  • set file type bin
  • set rec pack 4096
  • set send pack 4096
  • set window 5

Note that when using the serial-to-USB converter the first line should be

  • set line /dev/ttyUSB0

You can also setup separate kermrc files for different ports and run kermit to use different initialisation files like this:

  • kermit kermrc-ttyS0
  • kermit kermrc-USB0

When using the serial-to-USB converter sometimes there is a conflict with braille tty, in which case uninstall brltty

sudo apt-get remove brltty

Start kermit, type connect and then turn on the gumstix. You should see something like:

  • *** Welcome to Gumstix ***
  •  
  • DRAM: 64 MB
  • Flash: 16 MB
  • SMC91C1111-0
  • Can't overwrite "serial#"
  • Can't overwrite "ethaddr"
  • Net: SMC91C1111-0
  • Hit any key to stop autoboot: 0

Hit any key to stop the autobooting process and you should get a command line prompt. Type printenv and hit enter. The output shows your current gumstix environment settings. We'll adjust these to get the demo to boot on the gumstix. There are two ways to do this: the easy way and the hard way.

The Easy Way

If you have access to the internet, your life will be made easy by an tftpboot server on the machine {your.tftpboot.host}. After building a boot image, execute the following:

  • arm-linux-objcopy -O binary build/images/image.boot /tmp/boot
  • scp /tmp/boot {your.tftpboot.host}:/tftpboot/{your path}/bootimg.bin

This just takes an object copy of the boot image and puts it in your personal directory in the tftpboot server of {your.tftpboot.host}. Now you need to tell the gumstix what to boot. Use the command setenv to set environment variables. Ie. if you want to change 'bootfile' to {your path}/bootimg.bin, the command is

  • GUM> setenv bootfile "{your path}/bootimg.bin"

Below is a list of the variables you should change and what their values should be.

  • bootfile={your path}/bootimg.bin
  • gatewayip=10.13.0.1
  • netmask=255.255.254.0
  • serverip=10.13.0.1
  • bootcmd=tftpboot a0000000 {your username}/bootimg.bin; go a0000000

After setting the variables to their proper values, use saveenv to permanently save them.

Now type reset, hit enter, and hopefully the demo will boot via TFTP from your tftpboot server. There is a bunch of debugging output at the start which can be disregarded. To get rid of this output search through the source tree for #define DEBUG and comment it out.

The Hard Way

So if you don't have internet access where you're using the application, you need to load the binary into flash beforehand and tell the gumstix to boot from flash at startup. Do the setup from somewhere you can access your tftpboot server (ie. not 5 minutes before showing the demo where you don't have internet access), since it's easy to use TFTP to get the image onto flash. Anyway, here are the steps:

  • Compile your bootimg and copy onto tftpserver as explained in "Building a Boot Image".
  • Make sure the gumstix is connected to the network, start kermit and connect to the gumstix.
  • Plug in the gumstix (if it's already plugged in unplug then plug it back in)
  • Quickly hit any key to stop the autoboot and get a command line.
  • Load the boot image from tftpboot server into RAM:
    GUM> tftpboot a0000000 {your path}/bootimg.bin
  • Copy the image into flash:
    GUM> katinstall 200000 a0000000
  • Okay so now your image is permanently in flash and you can boot from flash whenever you want by loading the image into RAM and running it:
    GUM> katload 200000 a0000000; go a0000000

The Really Stupid Part

When you try to run the demo from flash by setting the autoboot command (bootcmd environment variable) to katload 200000 a0000000; go a0000000, you'll probably find that it will boot but nothing will happen. It turns out that the gumstix bootloader is weird and doesn't properly initialize the network card unless it actually makes use of it. When booting via tftp, U-Boot initializes the card so it can get the image over the network, but when booting from flash it has no reason to do this. There's probably a very elegant way to solve this, but the solution I used was to use this as my boot command:

  • bootcmd=ping 10.13.1.104; katload 200000 a0000000; go a0000000

10.13.1.104 is the IP address I set for my laptop. So, all I need to do is connect my laptop to the switch that the two gumstix are connected to, they both ping it, get a response, and happily boot my image with the network card properly initialized.

Buffer/DMA Alignment

A recurring problem with the voice sending and receiving was that the buffer read/write pointers wouldn't stay aligned with DMA, and every so often an overlap would occur that temporarily caused nasty audio playback. This is most likely due to interrupts being missed, which would allow the DMA pointers to traverse the audio data buffers at a faster rate than the user application which is reading from or writing to the buffers. Due to time constraints my solution to this was to periodically reset all the pointers to their starting positions. Theoretically this seems like a huge hack job, but it turns out it worked nicely and I could pretty much run the IP phone indefinitely without hearing any glitches. (I tested this my playing my MP3 player through the phone while I was working - for about 4 or 5 straight hours). Another thing that probably helped was reducing the audio sampling rate. This is probably something that could be looked into in the future.

Version Notes

Some notes on the version of the demo: This worked quite well. The chat functionality was a last-minute addition since we realized that just voice was boring from a scheduling perspective since it is purely time-driven. So, we added chat, which introduces random serial interrupts from the keyboard and random network packets arriving from the remote chat partner. Due to how quickly this had to be implemented anyone looking at the code will be very shocked to find that it pretty much uses no protocol at all aside from storing text, and sending it when it gets the signal. If a message is received while you are typing, the message wont be displayed until you send your message, or the storage buffers overflow (which leads to nasty situations). It has to be noted that we chose deliverately not to use synchronous IPC to make it suitable for a real-time scheduler we have implemented.

The Source

Relevant directories:

  • iguana/packet: Network Driver/Packet Filter
  • libs/packet: Network Driver user library
  • drivers/smc91x: SMC Chip-specific driver code
  • iguana/audio: Audio Driver
  • libs/audio: Audio Driver user library
  • drivers/ac97: AC97-specific driver code
  • iguana/audio_sender: Streaming Audio
  • libs/audio_sender: Streaming Audio user Library
  • iguana/network: Network Server
  • libs/network: Network server user Library
  • libs/lwip: LwIP stack (used only by network server)
  • iguana/vserial: Native Iguana Serial Driver
  • libs/vserial: Iguana Serial driver user library
  • drivers/uart_8250: PXA255 FFUART-specific code
  • iguana/chat: Chat server
  • libs/chat: Chat user library
  • iguana/dmamon: DMA driver
  • libs/dmamon: DMA driver user library
  • libs/driver: Iguana Driver Framework

Other related code:

  • platform/pxa/iguana/devicecore: Thread that intiializes all devices, allocates resources at startup, provides device reference to threads when requested. Code was added here to support audio driver, but Network driver has not yet been added since it has its own initialization procedure that worked and I didn't want to break it.
  • platform/pxa/tools/machines.py: This file tells the build system which drivers should be built, when building for a PXA target.
  • projects/iguana/SConstruct: Main build file for iguana. Except for drivers and driver threads (which are handled by machines.py), all new iguana libraries or applications need to be built from this file. Usually you can use existing code in this file as a template for adding new stuff.