Gregor Santner

Debugging is an essential part of software development. In case of Ubuntu Touch apps there are basically two ways to debug an app. This blog post showcases both ways.

Some basic knowledge of GDB is assumed as well as of Clickable which is the recommended tool to build Ubuntu Touch apps. If not, search the web for GDB tutorials and/or read the Clickable Documentation.

Desktop Mode

Clickable provides a desktop mode to run an Ubuntu Touch app on desktop. This mode also enables debugging via GDB. Again there are two ways:

1) Run GDB in a terminal 2) Start gdbserver and debug with your preferred IDE.

GDB on Command Line

From the project root directory run:

clickable desktop --gdb

This compiles the app with debug symbols (provided that CMake or qmake is used) and starts GDB inside a docker container. Use GDB commands like break main and run to start debugging. You can find more examples, like how to set breakpoints on a specific lines of code here.

Any IDE via gdbserver

From the project root directory run:

clickable desktop --gdbserver 3333

This starts the gdbserver on port 3333. Check for an option to do GDB Remote Debugging in your IDE and connect to localhost:3333.

On Device

Debugging in desktop mode is convenient, but it does not cover all use cases, e.g. those involving Content Hub, App Armor or arm specific issues. In such a case you need to debug directly on an Ubuntu Touch device.

You can always watch the logs from the device on your desktop via

clickable logs

This works independently of the debugging described below.

Clickable Debugging

Apps that are started through their own elf binary can be debugged on-device via Clickable’s Debug feature, as described in its docs. If this does not match your app, continue reading.

Preparation

Note: The steps described below make your rootfs writable and use apt to install packages. Do only proceed if you know what you are doing and how to reinstall Ubuntu Touch in case of trouble. Also note that your changes might get lost after system updates.

Make sure to install a debug build of your app, e.g. via Clickable:

clickable --debug

Make sure you have started the app at least once normally with that version.

Enter a shell on your device via SSH and run all subsequent commands on the device.

Remount the root filesystem writable and install gdb:

sudo mount -o rw,remount /
sudo apt update
sudo apt install gdb libc6-dbg

You may need to repeat this step after an OTA update. Installing libc6-dbg is necessary due to a bug that probably won’t be closed, soon.

Find the app’s desktop file in the ubuntu-app-launch cache

ls /home/phablet/.cache/ubuntu-app-launch/desktop/

Let’s use Webber as an example. In this example /home/phablet/.cache/ubuntu-app-launch/desktop/webber.timsueberkrueb_webber_0.5.1.desktop. You will need that path in a second.

Change directory to the installation path of the app, which can be found at /opt/click.ubuntu.com/<appname>.<maintainer>/current. In case of Webber

cd /opt/click.ubuntu.com/webber.timsueberkrueb/current

Look for the Exec line in the desktop file to find out how to start the app

cat /home/phablet/.cache/ubuntu-app-launch/desktop/webber.timsueberkrueb_webber_0.5.1.desktop | grep Exec

In case of Webber this returns Exec=webber %U. This means the app is started by the command webber (we can omit the %U). This executable may be located directly in the app installation directory or in the sub directory lib/<arch-triplet>/bin, where <arch-triplet is arm-linux-gnueabihf or aarch64-linux-gnu depending on your device. In case of Webber on a armhf device the executable is located at lib/arm-linux-gnueabihf/bin/webber.

Sourcing the app environment

When an app is started, some environment variables are set and the app may rely on them to work properly. That’s why you may or may not need to do this step before starting gdb on your app, too. In this sections you will learn how it can be done.

Start your app normally. Then find out the process ID (PID)

ps aux | grep <command>

where <command> is the executable that starts the app. Example for Webber:

phablet  32384  8.6  2.2 340256 62620 ?        Ssl  08:41   0:01 webber
phablet  32576  0.0  0.0   5684   812 pts/47   R+   08:41   0:00 grep --color=auto webber

We ignore the grep command we just ran and find the PID to be 32384. While the app is still running execute:

. <(xargs -0 bash -c 'printf "export %q\n" "$@"' -- < /proc/<PID>/environ)

replacing <PID> with the the PID identified above:

. <(xargs -0 bash -c 'printf "export %q\n" "$@"' -- < /proc/30649/environ)

Now the environment is set up as it would be on a normal app start. Close the app and start debugging as described in the next section.

Debugging

Now you are set up to start your debugging session:

gdb -ex "handle SIGILL nostop" --args <executable> --desktop_file_hint=<cached-desktop-file>

In case of Webber this is

gdb -ex "handle SIGILL nostop" --args lib/arm-linux-gnueabihf/bin/webber --desktop_file_hint=/home/phablet/.cache/ubuntu-app-launch/desktop/webber.timsueberkrueb_webber_0.5.1.desktop

The -ex "handle SIGILL nostop" part avoids stopping at a SIGILL signal caused by OpenSSL probing the abilities of the architecture. Use GDB commands like break main and run to start debugging.

Notice: Jonatan Hatakeyama Zeidler is the author of this article. The content was originally written in March 2020 and first published on this (gsantner’s) blog. ‘I’ refers to the original author.

Read more