Timestamp outgoing packets

问题:

I'm trying to get accurate timestamps for outgoing packets (sent using raw sockets). According to Linux/Documentation/networking/timestamping.txt, "For send time stamps the outgoing packet is looped back to the socket's error queue with the send time stamp(s) attached. It can be received with recvmsg(flags=MSG_ERRQUEUE).".

Unfortunately, recvmsg is always returning -1 when called on a raw socket (created with socket(PF_INET, SOCK_RAW, IPPROTO_RAW) and with SO_TIMESTAMP set to 1 with setsockopt). What am I doing wrong? Is there a better way of getting an accurate timestamp for an outgoing packet?

Addendum (information):

I also tried getting the timestamp from a packet sent through an UDP socket (source code below) and recvmsg returns -1: the error is "Resource temporarily unavailable" (EAGAIN).

Addendum (source code):

#include <arpa/inet.h>
#include <linux/net_tstamp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>

void die(char* s)
{
    perror(s);
    exit(1);
}

int main(int argc, char* argv[])
{
    char* destination_ip = "10.0.0.1";
    int destination_port = 1234;

    int sock;
    if ((sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
        die("socket()");
    }

    int timestamp_flags = SOF_TIMESTAMPING_TX_SOFTWARE;
    if (setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &timestamp_flags, sizeof(timestamp_flags)) < 0) {
        die("setsockopt()");
    }

    struct sockaddr_in si_server;
    memset(&si_server, 0, sizeof(si_server));
    si_server.sin_family = AF_INET;
    si_server.sin_port = htons(destination_port);
    if (inet_aton(destination_ip, &si_server.sin_addr) == 0) {
        die("inet_aton()");
    }

    const int buffer_len = 256;
    char buffer[buffer_len];

    const int n_packets = 10;
    for (int i = 0; i < n_packets; ++i) {
        sprintf(buffer, "Packet %d", i);
        if (sendto(sock, buffer, buffer_len, 0, (const sockaddr*) &si_server, sizeof(si_server)) < 0) {
            die("sendto()");
        }

        // Obtain the sent packet timestamp.
        char data[256];
        struct msghdr msg;
        struct iovec entry;
        struct sockaddr_in from_addr;
        struct {
            struct cmsghdr cm;
            char control[512];
        } control;
        int res;

        memset(&msg, 0, sizeof(msg));
        msg.msg_iov = &entry;
        msg.msg_iovlen = 1;
        entry.iov_base = data;
        entry.iov_len = sizeof(data);
        msg.msg_name = (caddr_t)&from_addr;
        msg.msg_namelen = sizeof(from_addr);
        msg.msg_control = &control;
        msg.msg_controllen = sizeof(control);        
        if (recvmsg(sock, &msg, MSG_ERRQUEUE) < 0) {
            die("recvmsg()");
        }
    }
    return 0;
}

回答1:

Looking into the Linux kernel source code, I found that the function responsible for putting the message containing the timestamp of the packet on the error queue is skb_tx_timestamp. This function is supposed to be called by the NIC driver and unfortunately, the e1000 driver doesn't call it (there's a similar function for hardware timestamping, but this is obviously dependent on the NIC driver supporting it).

According to this NetDev discussion from last September, "no driver calls skb_tx_timestamp()" and "You'll need to tweak your NIC driver to play with this TX timestamps". After adding a call to skb_tx_timestamp to e1000_xmit_frame on e1000_main.c, I was able to obtain timestamps for outgoing packets (through an UDP socket). I wasn't able to obtain timestamps for outgoing packets on a RAW socket, though (I still get EAGAIN).

回答2:

It's hard to know what you are doing wrong, since we cannot see your code.

However: The documentation says that SO_TIMESTAMP is for incoming packets, while SO_TIMESTAMPING is for outgoing packets.

The kernel documentation contains a full example which you could use as a base - though it's using UDP, but you should be able to adjust it to using a RAW socket. See the linux kernel Documentation/networking/timestamping/timestamping.c

EDIT: It seems transmit timestamping isn't universally supported, see e.g. here. Even today just a handful of nic drivers implement software support, and a few have hardware support.

回答3:

sock_tx_timestamp is only called for SOCK_DGRAM sockets in current kernel code.

BTW, the document Documentation/networking/timestamping/timestamping.c isn't very accurate.

SO_TIMESTAMP / SO_TIMESTAMPNS / SO_TIMESTAMPING / SIOCGSTAMP / SIOCGSTAMPNS are similar. Anyone of them will enable application getting the timestamp of a received packet.

With SOF_TIMESTAMPING_TX_SOFTWARE, any one of the above flags will also provide the application a CMSG in MSG_ERRQUEUE, indicating the timestamp of a sent packet.

But SOF_TIMESTAMPING_RX_SOFTWARE is useless at all. It can not even be used to disable the reporting of timestamp of received packets.

回答4:

I think, you have to mention the timestamping flag as

int timestamp_flags |= SOF_TIMESTAMPING_TX_SOFTWARE;
timestamp_flags |= SOF_TIMESTAMPING_SOFTWARE;

And also you have to check the strerror for all system call before getting the data from the msg queue error. This will help you to get rid off of the EAGAIN message.