How does the instruction decoder differentiate bet

Question

In 32-bit mode Intel solves the VEX prefix vs LDS/LES conflict by inverting the high bits of register extension, because the mod field of ModRM byte can't be 11b

The VEX prefix's initial-byte values, C4h and C5h, are the same as the opcodes of the LDS and LES instructions. These instructions are not supported in 64-bit mode. To resolve the ambiguity while in 32-bit mode, VEX's specification exploits the fact that a legal LDS or LES's ModRM byte can not be of the form 11xxxxxx (which would specify a register operand). Various bit-fields in the VEX prefix's second byte are inverted to ensure that the byte is always of this form in 32-bit mode.

https://en.wikipedia.org/wiki/VEX_prefix#Technical_description

However in EVEX the R and X bits aren't inverted, which results in mod=00b, which also indicates a memory operand in BOUND instruction

Four bits R, X, B, and W from the REX prefix. W expands the operand size to 64 bits or serves as an additional opcode, R expands reg, B expands r/m or reg, and X and B expand index and base in the SIB byte. Comparing to the VEX prefix, RXB are provided in non-inversed forms, just like in the REX prefix.

https://en.wikipedia.org/wiki/EVEX_prefix

So how can they decode that instruction overlap cleanly?

---

I checked the Intel manual and they seem to mention about the inversion of the bits only in VEX but not EVEX.

OTOH the table in sandpile says that those RXB bits in EVEX should be inverted as well.

Which of them is correct?

Answer 1

The trick with inverted bits works because of two facts:

1. In the 32-bit mode, only 8 of 32 ZMM registers are available, therefore EVEX.R' bit never flips, and thus never aliases with allowed BOUND encodings.
2. In the 64-bit mode, there is no BOUND instruction recognized so the EVEX bits can take any value.

An excerpt from a patent on AVX clarifies this:

[0111] REX′ field —this is the first part of the REX′ field and is the EVEX.R′ bit field (EVEX Byte 1, bit [4]—R′) that is used to encode either the upper 16 or lower 16 of the extended 32 register set. In one embodiment of the invention, this bit, along with others as indicated below, is stored in bit inverted format to distinguish (in the well-known x86 32-bit mode) from the BOUND instruction, whose real opcode byte is 62, but does not accept in the MOD R/M field (described below) the value of 11 in the MOD field; alternative embodiments of the invention do not store this and the other indicated bits below in the inverted format. A value of 1 is used to encode the lower 16 registers. In other words, R′Rrrr is formed by combining EVEX.R′, EVEX.R, and the other RRR from other fields.

However, the Intel SDM is unclear on this fact. I've looked through the SDM and indeed, in the EVEX section there are no traces of mentioning of the complement meaning of EVEX encodings. One must deduce it somehow from the fact that EVEX is an extension of VEX, and for the latter there is a statement of inverted meaning (volume 2A, section 2.3.5, the first bullet):

This field is encoded using 1’s complement form (inverted form), i.e. XMM0/YMM0/R0 is encoded as 1111B, XMM15/YMM15/R15 is encoded as 0000B.