How to GDB step debug a dynamically linked executa

For example for ARM, if I compile statically, all works fine:

sudo apt-get install gdb-multiarch gcc-arm-linux-gnueabihf qemu-user
printf '
#include <stdio.h>
#include <stdlib.h>

int main() {
    puts("hello world");
    return EXIT_SUCCESS;
}
' >  hello_world.c
arm-linux-gnueabihf-gcc -ggdb3 -static -o hello_world hello_world.c
qemu-arm -L /usr/arm-linux-gnueabihf -g 1234 ./hello_world

On another terminal:

gdb-multiarch -q --nh \
  -ex 'set architecture arm' \
  -ex 'set sysroot /usr/arm-linux-gnueabihf' \
  -ex 'file hello_world' \
  -ex 'target remote localhost:1234' \
  -ex 'break main' \
  -ex continue \
;

This leaves me at main, and I can see the source and step debug as usual.

However, if I remove the -static, and keep everything else unchanged, my breakpoint never gets hit, and the program runs until completion:

The target architecture is assumed to be arm
Reading symbols from hello_world...done.
Remote debugging using localhost:1234
Reading symbols from /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3...(no debugging symbols found)...done.
0xff7b3b80 in ?? () from /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3
Breakpoint 1 at 0x50c: file hello_world.c, line 5.
Continuing.
[Inferior 1 (Remote target) exited normally]

The executable itself does work fine however:

qemu-arm -L /usr/arm-linux-gnueabihf ./hello_world

prints:

hello world

I have seen: How to single step ARM assembler in GDB on Qemu? but it didn't cover the case of dynamically linked executables specifically.

Tested on Ubuntu 18.04, gdb-multiarch 8.1-0ubuntu3, gcc-arm-linux-gnueabihf 4:7.3.0-3ubuntu2, qemu-user 1:2.11+dfsg-1ubuntu7.3.

Edit: working pristine crosstool-ng setup

As a sanity check, I tried to get a clean toolchain with crosstool-ng, and it worked:

git clone https://github.com/crosstool-ng/crosstool-ng
cd crosstool-ng
git checkout d5900debd397b8909d9cafeb9a1093fb7a5dc6e6
export CT_PREFIX="$(pwd)/.build/ct_prefix"
./bootstrap
./configure --enable-local
./ct-ng arm-cortex_a15-linux-gnueabihf
# Has to be older than host kernel, which is 4.15.
printf "
CT_LINUX_V_4_14=y
CT_LINUX_VERSION=\"4.14.0\"
" >> .config
./ct-ng oldconfig
env -u LD_LIBRARY_PATH time ./ct-ng build -j`nproc`
cd ..
crosstool-ng/.build/ct_prefix/arm-cortex_a15-linux-gnueabihf/bin/arm-cortex_a15-linux-gnueabihf-gcc -ggdb3 -o hello_world hello_world.c -ggdb3 -static -o hello_world hello_world.c
qemu-arm -L crosstool-ng/.build/ct_prefix/arm-cortex_a15-linux-gnueabihf/arm-cortex_a15-linux-gnueabihf/sysroot -g 1234 ./hello_world

And on another shell:

./.build/ct_prefix/arm-cortex_a15-linux-gnueabihf/bin/arm-cortex_a15-linux-gnueabihf-gdb \
  -q --nh \
  -ex 'set architecture arm' \
  -ex 'set sysroot crosstool-ng/.build/ct_prefix/arm-cortex_a15-linux-gnueabihf/arm-cortex_a15-linux-gnueabihf/sysroot' \
  -ex 'file hello_world' \
  -ex 'target remote localhost:1234' \
  -ex 'break main' \
  -ex continue \
;

And it also works with the distro provided gdb-multiarch.

The failure is due to -pie -fpie, and there is a bug report for it at: https://bugs.launchpad.net/qemu/+bug/1528239

Explicitly setting -no-pie -fno-pie makes it work on both crosstool-ng and Ubuntu host.

The difference is that the Ubuntu one has GCC built to use -fpie by default, while my crosstool-ng build did not.

This can be checked with: