How to ptrace a multi-threaded application?

Question

EDIT (MADE PROGRESS):

I am trying to ptrace a vsftpd daemon. I have the following code which is attaching to the daemon. Then it successfully displays the PID of the first spawned process. However, for the children of this spawned process it returns the PIDs as 2,3,.. The program does catch the exiting of the spawned processes though, which makes me think I am close.

Any ideas?

void * trace_process(void * pid){
    pid_t child = atoi((char *) pid);
    long orig_eax, eax;
    int status;
    int callmade = FALSE;
    long opt = PTRACE_O_TRACEFORK;
    long newpid;

    long trace = ptrace(PTRACE_ATTACH,child,NULL,NULL);
    ptrace(PTRACE_SETOPTIONS,child,NULL,opt);
    if(trace == FALSE)
        printf("Attached to %d\n",child);

    while(TRUE) {
        child = waitpid(-1, &status, __WALL);

        if (status >> 16 == PTRACE_EVENT_FORK) {
            ptrace(PTRACE_GETEVENTMSG, child, NULL, (long) &newpid);
            ptrace(PTRACE_SYSCALL, newpid, NULL, NULL);       

            printf("Attached to offspring %ld\n", newpid);  
        }
        else{
            if(WIFEXITED(status))
                printf("Child %d exited\n", child);
        }

        ptrace(PTRACE_SYSCALL,child, NULL, NULL);
    }  
}

Sample output:

Attached to 2015 // daemon
Attached to offspring 5302 // new connection handler
Attached to offspring 2 // should be authenticator
Child 5303 exited       // authenticator exiting on successful login
Attached to offspring 3 // should be process serving files
Child 5304 exited       // logout: process serving files
Child 5302 exited       // connection closed
Attached to offspring 5305 // new connection handler
Attached to offspring 2    // ... repeat
Child 5306 exited
Attached to offspring 3
Child 5307 exited
Child 5305 exited

Answer 1

While reading the Playing with ptrace article, I found this comment from a user who also struggled with this:

/* After struggled a long time, I got a true way to make my ptrace worked 
 * correct with multi-thread application. Here're my sample codes, hope it 
 * can help others whom have the same confusion. */    

char trapCode[] = {0, 0, 0, 0};
int status;

ptrace(PTRACE_ATTACH, childProcess, NULL, NULL); //childProcess is the main thread
wait(NULL);

printf("\nchild %d created\n", childProcess);
fflush(stdout);

long ptraceOption = PTRACE_O_TRACECLONE;
ptrace(PTRACE_SETOPTIONS, childProcess, NULL, ptraceOption);

struct user_regs_struct regs;

for(unsigned int i = 0; i < m_breakPoints.size(); i++)
{
    BreakPoint_Info breakPointInfo = m_breakPoints[i];
    if(!breakPointInfo.m_enabled)
        continue;

    unsigned int index = breakPointInfo.m_checkPointIndex;
    if(m_bytesBackup.find(m_checkPoints[index].m_offset) != m_bytesBackup.end())
        continue;

    unsigned long readAddr = m_checkPoints[index].m_offset;
    One_Byte_With_Result *oneByte = new One_Byte_With_Result;
    getData(childProcess, readAddr, trapCode, 4);
    oneByte->m_char = trapCode[0];
    trapCode[0] = 0xcc;
    putData(childProcess, readAddr, trapCode, 4);

    m_bytesBackup.insert(std::make_pair(m_checkPoints[index].m_offset, oneByte));
}

std::set allThreads;
std::set::iterator allThreadsIter;
allThreads.insert(childProcess);

int rec = ptrace(PTRACE_CONT, childProcess, NULL, NULL);

while(true)
{
    pid_t child_waited = waitpid(-1, &status, __WALL);

    if(child_waited == -1)
        break;

    if(allThreads.find(child_waited) == allThreads.end())
    {
        printf("\nreceived unknown child %d\t", child_waited);
        break;
    }

    if(WIFSTOPPED(status) && WSTOPSIG(status) == SIGTRAP)
    {
        pid_t new_child;
        if(((status >> 16) & 0xffff) == PTRACE_EVENT_CLONE)
        {
            if(ptrace(PTRACE_GETEVENTMSG, child_waited, 0, &new_child) != -1)
            {        
                allThreads.insert(new_child);
                ptrace(PTRACE_CONT, new_child, 0, 0);

                printf("\nchild %d created\t", new_child);
            }

            ptrace(PTRACE_CONT, child_waited, 0, 0);
            continue;
        }
    }

    if(WIFEXITED(status))
    {
        allThreads.erase(child_waited);
        printf("\nchild %d exited with status %d\t", child_waited, WEXITSTATUS(status));

        if(allThreads.size() == 0)
            break;
    }
    else if(WIFSIGNALED(status))
    {
        allThreads.erase(child_waited);
        printf("\nchild %d killed by signal %d\t", child_waited, WTERMSIG(status));

        if(allThreads.size() == 0)
            break;
    }
    else if(WIFSTOPPED(status))
    {
        int stopCode = WSTOPSIG(status);
        if(stopCode == SIGTRAP)
        {
            ptrace(PTRACE_GETREGS, child_waited, NULL, &regs);
            unsigned long currentEip = regs.eip;
            //printf("%d\t%08x\n", child_waited, currentEip);

            Address_Bytes_Map::iterator iter = m_bytesBackup.find(currentEip - 1);
            if(iter != m_bytesBackup.end())
            {
                iter->second->m_result = true;

                regs.eip = regs.eip - 1;
                getData(child_waited, regs.eip, trapCode, 4);
                trapCode[0] = iter->second->m_char;
                putData(child_waited, regs.eip, trapCode, 4);
                rec = ptrace(PTRACE_SETREGS, child_waited, NULL, &regs);
            }
        }
    }

    rec = ptrace(PTRACE_CONT, child_waited, 1, NULL);

    continue;
}

Answer 2

After going further with my code, I realize that it does actually work to capture all the system calls that are coming from the parent and its children. The only issue is that the PIDs are returned as relative numbers, rather than actual PIds. This results in not being certain that a wait PID was actually generated from the parent. Either way, the code will get you all the system calls. I would still like to know why the PID is relative, for my own knowledge, but the code works fine.

Answer 3

Let the thread run before the next wait().

Try:

ptrace(PTRACE_SYSCALL,child, NULL, NULL);

Before:

while(TURE)