Generating Dynamically Linked Programs with Cargo

Eric Le Bihan

2016-04-02 18:42

Also available in:

Big, Statically Linked Programs

In a previous article, a Rust program was generated using Cargo: hello-rust.

$ cd $HOME/src/hello-rust
$ stat -c "%s" target/arm-buildroot-linux-gnueabihf/release/hello-rust
218732

The size of the binary file generated is 213 KB. Stripped, it will go down to 115 KB. This is indeed a big "Hello World"! Due to the architecture of the runtime, all I/O handling code is included in any statically linked binary, which is the default for rustc (running strings on the file ends up with a scary result).

If multiple Rust programs should be included in the target root filesystem of an embedded system, this would be a terrible waste of space.

It is possible to get a smaller file by adding -C prefer-dynamic to the rustc command line. But for the program to run properly, the cross-compiled versions of shared libraries with the missing symbols will have to be copied in the target root file system.

These libraries were built along with the Rust compiler and are available in $HOME/build/demo-rust/qemu/arm/host/usr/lib/rustlib/arm-buildroot-linux-gnueabihf/lib/.

Generating Dynamically Linked Programs with Cargo

It is possible to generate dynamically linked programs with Cargo. But it requires a feature only available in version 0.10, which is not available yet.

Build Cargo from Master Branch

Go to the Buildroot base directory and grab the current version of Cargo (master branch):

git clone --recursive https://github.com/rust-lang/cargo \
    $HOME/build/demo-rust/qemu/arm/build/host-cargo-lastest

Configure, build and install:

export PATH=$HOME/build/demo-rust/qemu/arm/host/usr/bin:$PATH
export PKG_CONFIG=$HOME/build/demo-rust/qemu/arm/host/usr/bin/pkgconf
export LIBRARY_PATH=$HOME/build/demo-rust/qemu/arm/host/usr/lib
pushd $HOME/build/demo-rust/qemu/arm/build/host-cargo-latest
./configure --prefix=$HOME/build/demo-rust/qemu/arm/host/usr \
            --localstatedir=$HOME/build/demo-rust/qemu/arm/host/var/lib \
            --sysconfdir=$HOME/build/demo-rust/qemu/arm/host/etc
make
make install
popd

Build Test Program

Then, create a new version of the Cargo configuration file for the "hello-world" crate:

cd $HOME/src/hello-rust
cat <<EOF > .cargo/config
[target.arm-buildroot-linux-gnueabihf]
linker = "arm-buildroot-linux-gnueabihf-gcc"

[build]
rustflags = ["-C", "prefer-dynamic"]
EOF

Note the last two lines, where Cargo is instructed to pass an additional flag to the Rust compiler when building the project. The new flag is quite explicit.

Tell the Rust compiler where the target specification can be found:

export RUST_TARGET_PATH=$HOME/build/demo-rust/qemu/arm/host/etc/rustc

Now, build the crate:

cargo build \
      --target=arm-buildroot-linux-gnueabihf \
      --release

Once the build is finished, check the size of the generated binary:

$ stat -c "%s" target/arm-buildroot-linux-gnueabihf/release/hello-rust
6880

The binary file has drastically shrinked! Once stripped, it will shrink to 3680 bytes. Copy it into the target root filesystem:

cp target/arm-buildroot-linux-gnueabihf/release/hello-rust \
   $HOME/build/demo-rust/qemu/arm/target/usr/bin

Copying the Missing Shared Libraries

Usually, ldd is used to list the dependencies of a dynamically linked program. Unfortunately, this tool is not available in the Buildroot environment. But it is possible to use the cross-compiled version of ld.so, the dynamic linker/loader, to achieve the same goal.

On the target, the dynamic loader is HOME/build/demo-rust/qemu/arm/target/lib/ld-2.22.so. As it is only executable on an ARM machine, QEMU userspace emulator will be used to run it.

$ qemu-arm -r 4.5.0 -E LD_TRACE_LOADED_OBJECTS=1 -E LD_VERBOSE=1 \
> $HOME/build/demo-rust/qemu/arm/target/lib/ld-2.22.so \
> target/arm-buildroot-linux-gnueabihf/release/hello-rust
        libstd-ca1c970e.so => not found
        libgcc_s.so.1 => not found
        libc.so.6 => not found

        Version information:
                target/arm-buildroot-linux-gnueabihf/release/hello-rust:
                libgcc_s.so.1 (GCC_3.5) => not found
                libc.so.6 (GLIBC_2.4) => not found

Note

QEMU userspace emulator for ARM is invoked with -r 4.5.0, which is the Linux version used, to avoid the error "FATAL: kernel too old".

As shown, the program depends on the C runtime (libc.so.6), the GCC low-level runtime library (libgcc_s.so.1) and libstd-ca1c970e.so, the Rust standard library.

The dependencies are listed, but as the shared libraries have not been found, the inspection is incomplete. Here is the result when passing the location of the libraries:

$ qemu-arm -r 4.5.0 -E LD_TRACE_LOADED_OBJECTS=1 -E LD_VERBOSE=1 \
> $HOME/build/demo-rust/qemu/arm/target/lib/ld-2.22.so \
> --library-path $HOME/build/demo-rust/qemu/arm/host/usr/lib/rustlib/arm-buildroot-linux-gnueabihf/lib:$HOME/build/demo-rust/qemu/arm/target/lib \
> target/arm-buildroot-linux-gnueabihf/release/hello-rust
        libstd-ca1c970e.so => /home/stewie/build/demo-rust/qemu/arm/host/usr/lib/rustlib/arm-buildroot-linux-gnueabihf/lib/libstd-ca1c970e.so (0xf64be000)
        libgcc_s.so.1 => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1 (0xf6492000)
        libc.so.6 => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6 (0xf6356000)
        libdl.so.2 => /home/stewie/build/demo-rust/qemu/arm/target/lib/libdl.so.2 (0xf6342000)
        libpthread.so.0 => /home/stewie/build/demo-rust/qemu/arm/target/lib/libpthread.so.0 (0xf6319000)
        libm.so.6 => /home/stewie/build/demo-rust/qemu/arm/target/lib/libm.so.6 (0xf629f000)
        /lib/ld-linux-armhf.so.3 => /home/stewie/build/demo-rust/qemu/arm/target/lib/ld-2.22.so (0xf6fcf000)

        Version information:
        target/arm-buildroot-linux-gnueabihf/release/hello-rust:
                libgcc_s.so.1 (GCC_3.5) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1
                libc.so.6 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
        /home/stewie/build/demo-rust/qemu/arm/host/usr/lib/rustlib/arm-buildroot-linux-gnueabihf/lib/libstd-ca1c970e.so:
                libdl.so.2 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libdl.so.2
                libgcc_s.so.1 (GCC_4.3.0) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1
                libgcc_s.so.1 (GLIBC_2.0) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.0) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1
                libgcc_s.so.1 (GCC_4.0.0) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.3.1) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1
                libgcc_s.so.1 (GCC_3.5) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1
                libm.so.6 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libm.so.6
                libpthread.so.0 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libpthread.so.0
                libc.so.6 (GLIBC_2.17) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
                libc.so.6 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
        /home/stewie/build/demo-rust/qemu/arm/target/lib/libgcc_s.so.1:
                libc.so.6 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
        /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6:
                ld-linux-armhf.so.3 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/ld-2.22.so
                ld-linux-armhf.so.3 (GLIBC_PRIVATE) => /home/stewie/build/demo-rust/qemu/arm/target/lib/ld-2.22.so
        /home/stewie/build/demo-rust/qemu/arm/target/lib/libdl.so.2:
                ld-linux-armhf.so.3 (GLIBC_PRIVATE) => /home/stewie/build/demo-rust/qemu/arm/target/lib/ld-2.22.so
                libc.so.6 (GLIBC_PRIVATE) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
                libc.so.6 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
        /home/stewie/build/demo-rust/qemu/arm/target/lib/libpthread.so.0:
                ld-linux-armhf.so.3 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/ld-2.22.so
                ld-linux-armhf.so.3 (GLIBC_PRIVATE) => /home/stewie/build/demo-rust/qemu/arm/target/lib/ld-2.22.so
                libc.so.6 (GLIBC_PRIVATE) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
                libc.so.6 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
        /home/stewie/build/demo-rust/qemu/arm/target/lib/libm.so.6:
                libc.so.6 (GLIBC_PRIVATE) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6
                libc.so.6 (GLIBC_2.4) => /home/stewie/build/demo-rust/qemu/arm/target/lib/libc.so.6

That is quite an exhaustive list! But it shows that only libstd-ca1c970e.so is required, as all the other dependencies are already in the target root filesystem.

To copy it, execute:

cp $HOME/build/demo-rust/qemu/arm/host/usr/lib/rustlib/arm-buildroot-linux-gnueabihf/lib/libstd-ca1c970e.so \
   $HOME/build/demo-rust/qemu/arm/target/usr/lib
chmod a+wx $HOME/build/demo-rust/qemu/arm/target/usr/lib/libstd-ca1c970e.so

As this library has been originally installed in $HOME/build/demo-rust/qemu/arm/host/usr/lib, the runtime search path in the ELF file is incorrect. To check and modify it, build patchelf:

make O=$HOME/build/demo-rust/qemu/arm host-patchelf

The current value of rpath is:

$ patchelf --print-rpath $HOME/build/demo-rust/qemu/arm/target/usr/lib/libstd-ca1c970e.so
/home/stewie/build/demo-rust/qemu/arm/host/usr/lib/rustlib/arm-buildroot-linux-gnueabihf/lib

To set it to an empty value, execute:

patchelf --set-rpath '' $HOME/build/demo-rust/qemu/arm/target/usr/lib/libstd-ca1c970e.so

Run Test Program from System

Go back to the Buildroot directory. Before rebuilding the system image, generate the cache of the loader (this operation is not automagically performed by Buildroot):

qemu-arm -r 4.5.0 $HOME/build/demo-rust/qemu/arm/staging/sbin/ldconfig -v \
         -r $HOME/build/demo-rust/qemu/arm/target

Then, rebuilt the system image:

make O=$HOME/build/demo-rust/qemu/arm

Now, start the system with QEMU:

qemu-system-arm \
    -M vexpress-a9 \
    -m 256 \
    -kernel $HOME/build/demo-rust/qemu/arm/images/zImage \
    -dtb $HOME/build/demo-rust/qemu/arm/images/vexpress-v2p-ca9.dtb \
    -drive file=$HOME/build/demo-rust/qemu/arm/images/rootfs.ext2,if=sd,format=raw \
    -append "console=ttyAMA0,115200 root=/dev/mmcblk0" \
    -serial stdio \
    -net nic,model=lan9118 \
    -net user

Log as "root" (no password) and execute the test program:

Welcome to Buildroot
buildroot login: root
# hello-rust
Hello World!

Excellent!

Note

When using QEMU from Debian Jessie (version 2.1.2), the test program can not be executed because the loader fails to read libstd-ca1c970e.so. Using strace shows that the kernel attempts to read beyond the limits of the filesystem, which is an EXT2 image on an emulated SD card. dmesg also shows complaints about the SD card having a bad geometry.

This is a QEMU bug, which does not seem to align the size of the filesystem (8192 KB) correctly. Setting the "exact size in blocks" to 16384 in the "Filesystem images" menu of Buildroot and regenerating the image solves the problem.