Including objects to a shared library from a C++ a

Question

I am trying to include some object files into a shared library I am building. Take the following command (things in [ETC] have been omitted for brevity):

/usr/bin/c++ -fPIC -std=c++14 -pthread -Iinclude/ext/liveMedia -Iinclude/ext/groupsock [ETC] -g -shared -Wl,-soname,libValkka.so -o lib/libValkka.so CMakeFiles/Valkka.dir/src/avthread.cpp.o CMakeFiles/Valkka.dir/src/opengl.cpp.o [ETC] CMakeFiles/Valkka.dir/src/decoders.cpp.o -lX11 -lGLEW -lGLU -lGL -Wl,--whole-archive lib/libavcodec.a -Wl,--no-whole-archive

So basically I am just creating a shared library where most of the objects come from my own source code (i.e. CMakeFiles/Valkka.dir/src/*.o), but some of them come from an external static library, located at "lib/libavcodec.a". I get the following error:

/usr/bin/ld: lib/libavcodec.a(h264_cabac.o): relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status

But that is so untrue! I can extract "libavcodec.a" with

ar x libavcodec.a

And after that check that

readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)' 

does give some **it:

00000000175d 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4 000000001926 003100000004 R_X86_64_PLT32 0000000000000000 __stack_chk_fail - 4

...

As does

objdump -r h264_cabac.o | grep -i "relocation"

So, indeed, the object files in "libavcodec.a" have been compiled to get PIC (position independent code).

Why does the linker believe otherwise!?

Related links:

How to include all objects of an archive in a shared object?

Linking archives (.a) into shared object (.so)

Is there a way to determine that a .a or .so library has been compiled as position indepenent code?

How can I tell, with something like objdump, if an object file has been built with -fPIC?

Answer 1

TL;DR

Add -Wl,-Bsymbolic to the gcc linkage options for your shared library.

Why?

You're testing the PICness of h264_cabac.o with:

readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)

and concluding the the object file was compiled with -fPIC if you get any hits. Presumably you got this test from the favourite answer to How can I tell, with something like objdump, if an object file has been built with -fPIC?

You got some hits, and I can reproduce that more than one way:

From the source code

$ git clone https://github.com/FFmpeg/FFmpeg.git
$ cd FFmpeg
$ ./configure --enable-shared
$ make

Then:

$ cd libavcodec
$ readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)'
00000000175d  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001926  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
00000000259f  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000002f0d  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003216  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000003460  00330000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_chroma422_dc_s - 4
000000003afc  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003fb6  00360000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_i_mb_type_info - 4
000000004031  00370000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_mb_sizes - 4
00000000409a  003800000004 R_X86_64_PLT32    0000000000000000 ff_init_cabac_decoder - 4
000000004248  00390000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_b_mb_type_info - 4
000000004299  003a00000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000004a31  003b00000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra4x4 - 4
000000004bd5  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000004f85  003c0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_p_mb_type_info - 4
0000000050fd  003d0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_b_sub_mb_type_ - 4
000000005233  003a00000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
00000000544a  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000005bef  003a00000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000006db5  003e00000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
000000006de9  003f0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_p_sub_mb_type_ - 4
000000007171  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000008b1b  003e00000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
00000000ad41  004000000009 R_X86_64_GOTPCREL 0000000000000000 ff_h264_chroma_dc_scan - 4
00000000ad84  004000000009 R_X86_64_GOTPCREL 0000000000000000 ff_h264_chroma_dc_scan - 4
00000000b758  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4

From the Ubuntu 16.04 dev package

$ sudo apt-get install libavcodec-dev
$ dpkg -S libavcodec.a
libavcodec-dev:amd64: /usr/lib/x86_64-linux-gnu/libavcodec.a
$ mkdir ~/deleteme
$ cd ~/deleteme
$ ar x /usr/lib/x86_64-linux-gnu/libavcodec.a h264_cabac.o
$ readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)'
0000000000c7  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
0000000002fa  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
00000000179d  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001966  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001b09  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001d4a  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001ee5  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
00000000265f  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000002fcd  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
0000000032f6  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000003305  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003bdc  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003cb5  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000004121  00320000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_mb_sizes - 4
000000004187  003300000004 R_X86_64_PLT32    0000000000000000 ff_init_cabac_decoder - 4
000000004381  003400000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000004afe  003500000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra4x4 - 4
000000005556  003400000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
00000000576a  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000005acf  003400000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000006e31  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000006e58  003600000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
000000009c20  003600000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
00000000b425  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
00000000b5ab  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4

The results aren't identical, and I get 26 relocations the first way, 25 the second way. But there are plenty of PIC-safe relocations either way and I'm happy to believe both compilations of h264_cabac.o had -fPIC, whatever other options they had.

I'll state the obvious: The symbol ff_h264_cabac_tables, about which your linkage complains:

relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' can not be used when making a shared object

isn't in either of those lists. That means this object file - from both provenances - contains both PIC-safe and PIC-unsafe relocations. How could GCC get that wrong, without anybody noticing till now? And if it did, how did I just run the shared library build of FFmpeg and link libavcodec.so successfully?

Let's have a look at the PIC-unsafe relocations:

$ readelf --relocs h264_cabac.o | egrep -v '(GOT|PLT|JU?MP_SLOT)'
000000000017  002c00000002 R_X86_64_PC32     0000000000000000 ff_h264_cabac_tables - 4
...
...

Well I'll elide ~160 lines of that, but they all describe PC-relative type R_X86_64_PC32 relocations and the only symbol mentioned, discounting section names and local labels, is our friend ff_h264_cabac_tables, about which the symbol table says:

$ readelf -s h264_cabac.o | grep ff_h264_cabac_tables
    44: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND ff_h264_cabac_tables

It's a global variable, not defined in this object file.

GCC's -fPIC isn't broken. However, knowing that an object file was compiled with -fPIC can't absolutely guarantee that it contains no PC-relative type R_X86_64_PC32 relocations that reference undefined global symbols. relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' is such a relocation. The R_X86_64_PC32 type relocation employs a 32-bit PC-relative addressing mode, which is efficient but has a critical limitation in the setting of a 64-bit linkage. The linker can't guarantee the referenced symbol will not be dynamically resolved to an address that's unrepresentable in this addressing mode. It won't have that, so it says:

relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' can not be used when making a shared object

and its advice:

recompile with -fPIC

is based on the hypothesis that the culprit object file was not compiled with -fPIC. Which is probably, but necessarily, the right hypothesis and is not right about your culprit libavcodec.a(h264_cabac.o)

Compiling with -fPIC will guarantee you PIC-safe relocations provided that the compiler is allowed to do all the assembly and code-generation. But it wasn't allowed to with your specimen of h264_cabac.o or either of my specimens. All those specimens were compiled from FFmpeg/libavcodec/x86/h264_cabac.c in the FFmpeg source tree. Look at that file and you'll see that it defines functions that refer to the extern global variable ff_h264_cabac_tables and are implemented in inline, hand-crafted assembly. GCC can be told to compile these functions -fPIC, but it doesn't get a chance. The position-independence of these functions is the responsibility of the author of the assembly code.

We can show that GCC is able to compile a h264_cabac.o that has only PIC-safe relocations, if it's allowed to. That will collaterally prove that your linkage failure stems from the hand-assembly of our specimens of the file and will also show you one fix for the linkage failure. FFmpeg's ./configure script has the option:

--disable-asm            disable all assembly optimizations

which has the effect, among others, of causing h264_cabac.o to be compiled from the pure C source file FFmpeg/libavcodec/h264_cabac.c instead of the the inline assembly source FFmpeg/libavcodec/x86/h264_cabac.c. So let's try that:

$ cd FFmpeg
$ make clean
$ ./configure --enable-shared --disable-asm
$ make
$ cd libavcodec
$ readelf --relocs h264_cabac.o | grep ff_h264_cabac_tables
00000000000a  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000000ca  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000001eb5  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000021c6  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000026fe  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000002a17  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000002f13  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000324c  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000003509  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000362a  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000037d7  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000592b  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4

Now, all the relocations referencing ff_h264_cabac_tables are PIC-safe. We might as well prove that this h264_cabac.o can be linked in a shared library. We know that ff_h264_cabac_tables is undefined in h264_cabac.o, so we'll also need to link the object file in which it is defined. It happens to be ./cabac.o.

$ gcc -shared -o libfoo.so h264_cabac.o cabac.o

Voila:

$ file libfoo.so
libfoo.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ed63107b715b357853da94d4a031c0b06c30c5f2, not stripped

You might still feel a little aggrieved, however, if you have to link your own shared library with this unoptimized h264_cabac.o and a little disappointed by the assembly coding flaw that obliges you to. These feelings would be premature.

Remember, I've already built FFmpeg successfully with plain vanilla ./configure --enable-shared. I said that linker's objection in your failing linkage is that the R_X86_64_PC32 relocation referencing ff_h264_cabac_tables might not be feasible at runtime, if ff_h264_cabac_tables is dynamically resolved. It is not an objection to the type R_X86_64_PC32 relocation as such. It is a precautionary objection, based on ignorance of how ff_h264_cabac_tables will ultimately be resolved.

But we know that ff_h264_cabac_tables is in fact defined in cabac.o and that we'll include it the same linkage with h264_cabac.o, just as they're both included in the linkage of libavcodec.so. And we can tell the linker that any global references in the linkage are to be statically resolved to definitions in the shared library being linked, if at all, by passing it the argument:

-Bsymbolic

This will remove its precautionary objection to any R_X86_64_PC32 relocation. It knows it will be able to decide at linktime whether the R_X86_64_PC32 relocation against ff_h264_cabac_tables is feasible. If not, it will give a different error: relocation truncated to fit:... Otherwise it will succeed without comment.

Inevitably, that's how libavcodec.so is successfully linked in the stock FFmpeg build. Once more from the top:

$ cd FFmpeg
$ make clean
$ ./configure --enable-shared
$ make

Then force a relink of libavcodec.so:

$ rm libavcodec/h264_cabac.o
$ $ make libavcodec/libavcodec.so V=1
gcc -I. -I./ -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
-D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DPIC -DZLIB_CONST -DHAVE_AV_CONFIG_H \
-std=c11 -fomit-frame-pointer -fPIC -pthread  -g -Wdeclaration-after-statement \
-Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wwrite-strings \
-Wtype-limits -Wundef -Wmissing-prototypes -Wno-pointer-to-int-cast -Wstrict-prototypes \
-Wempty-body -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign \
-O3 -fno-math-errno -fno-signed-zeros -fno-tree-vectorize -Werror=format-security \
-Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type \
-Werror=vla -Wformat -fdiagnostics-color=auto -Wno-maybe-uninitialized  \
-MMD -MF libavcodec/h264_cabac.d -MT libavcodec/h264_cabac.o -c \
-o libavcodec/h264_cabac.o libavcodec/h264_cabac.c
sed 's/MAJOR/57/' libavcodec/libavcodec.v | cat > libavcodec/libavcodec.ver
gcc -shared -Wl,-soname,libavcodec.so.57 -Wl,-Bsymbolic ... etc. etc. ...
                                         ^^^^^^^^^^^^^^   

So there is no assembly coding flaw. To link the hand-optimized h264_cabac.o in a shared library you just need to add -Wl,-Bsymbolic to the gcc linkage options. It's a requirement of the optimization.

Let's prove that minimally:

$ cd libavcodec/
$ gcc -shared -o libfoo.so h264_cabac.o cabac.o
/usr/bin/ld: h264_cabac.o: relocation R_X86_64_PC32 against symbol `ff_h264_cabac_tables' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

There's your failure again. And:

$ gcc -shared -Wl,-Bsymbolic -o libfoo.so h264_cabac.o cabac.o
$ file libfoo.so 
libfoo.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=7dc86aeae353c4d92cdb5fa35d169bf019b47eb2, not stripped

success.