Detect NUL file descriptor (isatty is bogus)

Question

The following C file gives a bogus result when NUL is piped to it:

int main()
{
  printf("_isatty = %d\n", _isatty(0));
}

the result is:

C:\Users\Edward\Dev\nulltest> test.exe < NUL
_isatty = 64

I'm pretty sure NUL (aka /dev/null) is not a terminal device! So I need to detect in another way whether or not the file descriptor corresponds to NUL. The number doesn't have any specific meaning; I see it when I actually do have a terminal attached.

What should I do? This question suggests using a sketchy undocumented function to get the underlying name, presumably comparing it to NUL, but that feels less than ideal to me. Is there a better way?

P.S. This would help solve this GHC bug.

Answer 1

Here is a possible solution, but I'm not convinced it works all the time. I believe it will work for the specific case of a NUL file descriptor:

int real_isatty(int fd) {
    DWORD st;
    HANDLE h;
    if (!_isatty(fd)) {
        /* TTY must be a character device */
        return 0;
    }
    h = (HANDLE)_get_osfhandle(fd);
    if (h == INVALID_HANDLE_VALUE) {
        /* Broken handle can't be terminal */
        return 0;
    }
    if (!GetConsoleMode(h, &st)) {
        /* GetConsoleMode appears to fail when it's not a TTY. */
        return 0;
    }
    return 1;
}

Answer 2

From msdn:

_isatty returns a nonzero value if the descriptor is associated with a character device. Otherwise, _isatty returns 0.

NUL is like /dev/null on Unix, it's a char device.

Note that on Linux, isatty is different:

The isatty() function tests whether fd is an open file descriptor referring to a terminal.

What you can do is try to compare STDIN_FILENO (0) with ${cwd}/NUL (using stat or stat).

Update:

int ret = GetFileType(GetStdHandle(STD_INPUT_HANDLE));

It will return FILE_TYPE_CHAR for NUL or tty.

See GetFileType documentation for other values. You can detect files/char device/pipes.

Update Final:

Use GetConsoleMode for input and GetConsoleScreenBufferInfo for output.

CONSOLE_SCREEN_BUFFER_INFO sbi;
DWORD mode;
if (!GetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), &mode))
   fprintf(stderr, "not console\n");
else
   fprintf(stderr, "console\n");
if (!GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &sbi))
   fprintf(stderr, "not console\n");
else
  fprintf(stderr, "console\n");

Answer 3

You can use fstat on the file descriptor and compare the device member of the resulting stat structure with that for /dev/null and see if they match.