64-bit syscall documentation for MacOS assembly

Question

I'm having trouble finding the good documentation for writing 64-bit assembly on MacOS.

The 64-bit SysV ABI says the following in section A.2.1 and this SO post quotes it:

• A system-call is done via the syscall instruction. The kernel destroys registers %rcx and %r11.

• Returning from the syscall, register %rax contains the result of the system-call. A value in the range between -4095 and -1 indicates an error, it is -errno.

Those two sentences are ok on Linux but are wrong on macOS Sierra with the following code:

global _start
extern _exit

section .text
_start:

; Align stack to 16 bytes for libc
and rsp, 0xFFFFFFFFFFFFFFF0

; Call write
mov rdx, 12             ; size
mov rsi, hello          ; buf
mov edi, 1              ; fd
mov rax, 0x2000004      ; write ; replace to mov rax, 0x1 on linux
syscall

jc .err                 ; Jumps on error on macOS, but why?
jnc .ok

.err:
mov rdi, -1
call _exit              ; exit(-1)

.ok:
; Expect rdx to be 12, but it isn't on macOS!
mov rdi, rdx
call _exit              ; exit(rdx)

; String for write
section .data
hello:
.str db `Hello world\n`
.len equ $-hello.str

Compile with NASM:

; MacOS: nasm -f macho64 syscall.asm && ld syscall.o -lc -macosx_version_min 10.12 -e _start -o syscall
; Linux: nasm -f elf64 syscall.asm -o syscall.o && ld syscall.o -lc -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o syscall

Run on macOS:

./syscall      # Return value 0
./syscall >&-  # Return value 255 (-1)

I found out that:

• A syscall return errno an sets the carry flag on error, instead of returning -errno in rax
• rdx register is clobbered by syscall
• On Linux, everything works as expected

Why is rdx clobbered? Why doesn't a syscall return -errno? Where can I find the real documentation?

The only place I found where someone talks about the carry flag for syscall errors is here

Answer 1

I used this:

# as hello.asm -o hello.o
# ld hello.o -macosx_version_min 10.13 -e _main -o hello  -lSystem
.section __DATA,__data
str:
  .asciz "Hello world!\n"

.section __TEXT,__text
.globl _main
_main:
  movl $0x2000004, %eax           # preparing system call 4
  movl $1, %edi                    # STDOUT file descriptor is 1
  movq str@GOTPCREL(%rip), %rsi   # The value to print
  movq $13, %rdx                 # the size of the value to print
  syscall

  movl %eax, %edi
  movl $0x2000001, %eax           # exit 0
  syscall

and was able to catch return value into eax. Here return value is the number of bytes actually written by write system call. And yes MacOS being a BSD variant it is the carry flag that tells you if the syscall was wrong or not (errno is just an external linkage variable).

# hello_asm.s
# as hello_asm.s -o hello_asm.o
# ld hello_asm.o -e _main -o hello_asm
.section __DATA,__data
str:
        .asciz "Hello world!\n"
good:
        .asciz "OK\n"

.section __TEXT,__text
.globl _main
_main:
        movl $0x2000004, %eax           # preparing system call 4
        movl $5, %edi                    # STDOUT file descriptor is 5
        movq str@GOTPCREL(%rip), %rsi   # The value to print
        movq $13, %rdx                 # the size of the value to print
        syscall

        jc err

        movl $0x2000004, %eax           # preparing system call 4
        movl $1, %edi                    # STDOUT file descriptor is 1
        movq good@GOTPCREL(%rip), %rsi   # The value to print
        movq $3, %rdx                 # the size of the value to print
        syscall
        movl $0, %edi
        movl $0x2000001, %eax           # exit 0
        syscall
err:    
        movl $1, %edi
        movl $0x2000001, %eax           # exit 0
        syscall

This will exits with error code one because descriptor 5 was used, if you try descriptor 1 then it will work printing another message and exiting with 0.