Can branch prediction crash my program?

Going trough chapter 3 of this book called Computer Systems Architecture: A programmer's perspective, it is stated that an implementation like

testl %eax, %eax
cmovne (%eax), %edx

is invalid because if the prediction fails, then we'll have NULL dereferencing. It is also stated that we should use branching code.

Still, wouldn't using conditional jumps lead to the same result? For example:

.L1:
jmp *%eax

testl %eax, %eax
jne .L1

Is it possible to trick gcc to output something like that for an x86-32? Suppose I have an array of pointers to functions of which some are valid and some aren't and I call each one that's not NULL.

No. You should not be able to detect out-of-order operand fetch of a jmp instruction if it is part of a speculative execution that proves invalid due to a test and jump.

The cmove__ instruction is precisely documented to cause a fault if a memory access operand would cause a fault, even if the condition is not met. In other words, this is not speculative execution. It's part of the instruction semantics. It's the move to desination that's conditional, not the fetch.

The jmp instruction is not so-documented.

I'm not getting the point of your example code because there is no condition on the memory operation *%eax. If %eax contains zero, certainly the fetch in the unconditional execution of jmp *%eax will cause a fault. This is correct behavior. If you test %eax and jump around the bad reference.

testl %eax, %eax
je .L1
jmp *%eax
.L1:

There can't be a problem. Speculative execution of the *%eax cannot cause a fault unless the speculation turns out to be valid, i.e. the true control path. This is similar to behavior for bad opcodes, division by zero and the like: normal program semantics are not affected by speculative execution.

Where out-of-order fetches and stores really do cause all kinds of interesting problems is in multi-processing. This article and also its first part in the preceeding issue are great discussions of this topic.