In the Intel 64 & IA-32 architecutures manual vol 3A, Chapter 9 Processor Management and Initialization, I found the the following:
Compatibility mode execution is selected on a code-segment basis. This mode allows legacy applications to coexist with 64-bit applications running in 64-bit mode. An operating system running in IA-32e mode can execute existing 16-bit and 32-bit applications by clearing their code-segment descriptor's CS.L bit to 0.
Does this mean that legacy 16-bit & 32-bit application can coexist with 64-bit application on an operating system running in IA-32e mode.
But as I know, legacy 16-bit code is generally not supported by 64-bit operating system. If it is supported, how can I startup a 16-bit application?
16-bit DOS apps can't run under 64-bit Windows, because virtual-8086 mode isn't available in long mode
However 16-bit protected mode is still available, so technically it's possible to run 16-bit Windows 3.x apps. That's how Wine runs 16-bit Windows apps in 64-bit Linux. Unfortunately 64-bit Windows doesn't have the same capability, although the reason is not because 64-bit mode cannot run 16-bit instructions but because the significant part has been increased.
The primary reason is that handles have 32 significant bits on 64-bit Windows. Therefore, handles cannot be truncated and passed to 16-bit applications without loss of data.
https://docs.microsoft.com/en-us/windows/win32/winprog64/running-32-bit-applications
So if you want to run 16-bit apps on 64-bit Windows you have to use a virtual machine
For more detailed information please read Peter Cordes' answer
See also Can a 64-bit computer (x86) run a 16-bit OS natively, without emulation?
Under a 64-bit kernel, 16-bit protected-mode user-space is available, but virtual-8086 mode isn't.
Most 16-bit software is written for DOS, and expects to run in real mode. vm86 (virtual-8086) mode is basically hardware virtualization for real mode, allowing the guest to use cli / sti without affecting the real IF bit in EFLAGS for example.
In 16-bit protected mode, cli would only work if the IO privilege level was 0 (like ring 0), and it would disable interrupts on that actual CPU core, not just inside the 16-bit emulated environment. Thus it's not useful for running 16-bit DOS programs under a modern OS.
So yes, you can run 16-bit user-space code under a 64-bit kernel, but only in 16-bit protected mode which nobody ever uses (AFAIK). I don't think Linux natively supports 16-bit processes, although perhaps you could create a custom 16-bit code segment with the modify_ldt system call, and jmp far to it.
It might have been possible to build a software-virtualization system that used 16-bit protected mode to run 16-bit DOS guests, trapping to the kernel/hypervisor when the guest ran instructions like cli. @Lưu Vĩnh Phúc's answer is one reason that MS didn't try to do that.
Software-virtualization of x86 is non-trivial, though, because some instructions that you need to emulate (like pushf) don't actually trap. The guest could notice (or break because of) the Interrupts-enabled flag (IF) in the pushf result not matching a cli it just ran.
Even the slowest x86-64 CPUs are plenty fast enough to run software like DOSBOX that fully emulates an x86 PC with directly-accessible old hardware, so there wasn't much demand for being able to run 16-bit code natively, or much demand for running it in a way that looked "native", i.e. able to launch other programs under the main OS instead of just inside the emulated environment.
Apparently there was a Linux patch (last updated in 2008) to run vm86 under a 64-bit kernel, maybe by switching the kernel to protected mode temporarily? Or by just running software emulation.
It's possible for a 64-bit kernel to switch back to legacy mode, making it possible to use vm86 mode.
Some kernels in real life actually do run some/most of the kernel in 32-bit mode, perhaps only switching into long mode when 64-bit user-space needs to run, or to read/write RAM that can't be mapped into the limited 32-bit virtual address space.
I think I've read that MacOS did / does this, perhaps to support 32-bit binary kernel drivers.
32 Bit application is supported by 64 bit architectire, so they can coexist, regarding 16 Bit application you need some tools to emulate addresing space shuch as dosbox on windows
