Porting wifibroadcast to Android

This post shows my approach on getting my FPV wifi transmission system to run on an Android device

—

Motivation sometimes comes through strange ways. In this case it was my laptop’s battery that died completely. I just had finished my quad and one week later my laptop was bound to the wall due to the failed battery. Thus I could not fly anymore since then 😦 That was the perfect excuse to port the wifibroadcast stuff over to Android. To be honest: I wouldn’t want to do this a second time. It was quite painful. I can’t give give you exact reproduction steps since this is higly device dependend. But you should gather enough info to redo the stuff for your device (I used a Nexus 7 2013 running Android 5). You’ll find a screenshot from the android device here (so the system has more or less photographed itself. How I love these recursive worm holes :):

This project needed to archive the following sub-goals:

* Root the device (many manuals available on how to do that)
* Support TL-WN722N wifi card in android kernel
* Cross-compile wifibroadcast for android
* Decode of raw h264 streams
* Displaying video data

TL-WN722N wifi card kernel support

Although most modern Android devices support host mode over USB OTG the number of activated USB drivers in the vanilla kernels is quite low. You can be pretty sure that there will be no USB wifi drivers enabled. Thus there is no way around a custom kernel build. This page has a detailed description on how to build your own kernel for the official “Google” android. It seems as if kernel compilation for Cyanogenmod is a bit more complicated. That’s why I went this way. Using “make menuconfig” to enable the driver “ath_htc” for the TL-WN722N is all there is to do. After installing the new kernel in your device you also need the firmware for the card. Note that you probably can’t use the firmware from your Ubuntu machine (or similar) because that probably uses the newest v1.4 firmware. The android kernel only supports v1.3 which you can get here. To copy it to the right location enter on the android shell (root required):

#make system writeable
mount -o rw,remount /system
cp /sdcard/htc_9271.fw /system/etc/firmware

Now your wifi card should run. Note that the card is by default “down”. So you won’t notice anything when you plug it in. Look into dmesg and see if everything went well. You can safely ignore error messages in dmesg about the “ASSERT in hdd_netdev_notifier_call”. That originates from some unclean qualcom wifi drivers for host mode…

Running wifibroadcast on Android

Now that the wifi card runs, how do we use it with wifibroadcast? Well, I was lazy and did not bother to port it over to Android. Instead I used a Ubuntu chroot environment where I could easily build and run it like I would be on a real linux machine. Again, you find a lot of pages describing how to create an “Ubuntu in a file”. I won’t repeat these steps here. Just the two lines which get me from the android shell into the system:

#this mounts the ubuntu image into the directory "ubuntu"
mount -o loop -t ext4 ubuntu.img ubuntu

#this chroots into the system
unset LD_PRELOAD
export USER=root
export HOME=/root
chroot ubuntu /bin/bash

Now you have a full powered ubuntu system available on your android device 🙂 In there you can do the usual things:

apt-get install mercurial
hg clone https://bitbucket.org/befi/wifibroadcast
cd wifibroadcast
make

And then you have a usable wifibroadcast. Inside the ubuntu system you can then setup your wifi card:

ifconfig wlan1 down
iw dev wlan1 set monitor otherbss fcsfail
ifconfig wlan1 up
iwconfig wlan1 channel 13

Decode and display video

Remember, the rx program will receive raw h264 frames that need to be decoded and displayed. My first attempt was to install the “XServer XSDL” app (excellept app by the way) which gives your chroot system graphical output capabilities. From that I just used the standard ubuntu gstreamer tools to decode and display the video. Unfortunately this was a bit too slow and had a latency of around 500ms (too much to be usable). I also tried some of the video streaming apps from the Play store but I found that none of them was suitable. So I had to write my own… 😦

I wrote an app that opens a UDP socket on port 5000 for receiving raw h264 data. The incoming data is parsed for “access units” and these are forwarded into a MediaCodec instance. The decoded frames are then directly rendered onto a Surface object. You can grab the source code of it here:

hg clone https://bitbucket.org/befi/h264viewer

This repository contains an “Android Studio” project. Use “import project” to open it. It targets Android API 21 (aka 5 Lollipop). There is also a ready-to-install .apk under “app/build/outputs/apk/app-debug.apk”

How to use that program? Very simple: Just send raw h264 data over UDP to port 5000 of your android device and it gets displayed. If your Raspi is in the same network as your android device you could use something like this:

raspivid -t 0 -w 1280 -h 720 -fps 30 -b 3000000 -n -pf baseline -o - | socat -b 1024 - UDP4-DATAGRAM:192.168.1.100:5000

where 192.168.1.100 is the IP address of your android device. Why have I used “socat” instead of “netcat”? Well, netcat is a line-based tool. Since we are sending binary data it usually takes some data until something similar to a newline is found (netcat triggers the packet transmission on this condition). Most often this will be more data than the maximum packet size. Therefore, netcat packets are usually fragmented IP packets (something you should avoid). Socat just sends unfragmented packets when the threshold (in this case 1024bytes) is reached.

How to use that with wifibroadcast? Again, very simple. On the raspberry execute the usual command to broadcast your video into the air:

raspivid -t 0 -w 1280 -h 720 -fps 30 -b 3000000 -n -pf baseline -o - | sudo ./tx -r 3 -b 8 wlan1

Inside the chrooted Ubuntu on your android device execute:

./rx -b 8 wlan1 | socat -b 1024 - UDP4-DATAGRAM:localhost:5000

All this does is to forward the packets from “rx” to the UDP port 5000 of your device. And voilà, you should see your camera’s image.

Summary

The latency is ok and should be usable. My PC is still a little bit faster but that is not too surprising. I have some ideas on how to optimize everything and keep you posted. Concerning battery life I was quite surprised. The app and wifi dongle drained the battery 14% after one hour of runtime. So I’m quite confident that this would allow me to at least stream for 5 hours with a full battery.

Another alternative I’m thinking about is to use a Raspberry with a HDMI display to show the video stream. I’ve fiddled around a bit and maybe in the next post I’ll present my OMX pipeline that does the decoding and displaying of the video on the Raspberry’s GPU (thus the CPU idles). I was actually surprised that this gave me even lower latency compared to a gstreamer pipeline running on my PC. The downside is that this is a bit more complicated hardware-wise. raspi+display+display-controller+wifi-card+battery+5v-dc-dc-converter could get a bit messy… we’ll see 🙂