How to read mesos task stdout/stderr from Mesos fr

Question

I am developing a Mesos framework, it is working perfectly fine, my only issue is that I am unable to read task stdout or stderr from inside the the Scheduler class.

I am providing a code sample below, I would like to read the stdout and stderr of a finished task, preferably in the statusUpdate function but anywhere would be useful. How can I reach that info? I tried getting executorInfo or executorId from TaskInfo or TaskStatus objects without any luck.

If someone can provide a code sample it would be greatly appreciated. I know that I can read the stderr and stdout from the master UI at url:master:5050 and also from the file system on the slaves, but I REALLY need to read it inside the framework, in the Java class as it will influence future scheduling decisions made in the resourceOffers function. As a last resort I was thinking of making a web call to the master by constructing the required URL, but that would be way too inconvenient. The mesos API must be providing a way to do it from inside the Scheduler class.

public class TfScheduler implements Scheduler {

    public TfScheduler(String volumes, String container) {

    }

    @Override
    public void registered(SchedulerDriver driver,
            Protos.FrameworkID frameworkId,
            Protos.MasterInfo masterInfo) {
        System.out.println("Registered! ID = " + frameworkId.getValue());
    }

    @Override
    public void reregistered(SchedulerDriver driver, Protos.MasterInfo masterInfo) {
    }

    @Override
    public void disconnected(SchedulerDriver driver) {
    }

    @Override
    public void resourceOffers(SchedulerDriver driver, List<Protos.Offer> offers) {

    }

    @Override
    public void offerRescinded(SchedulerDriver driver, Protos.OfferID offerId
    ) {
    }

    @Override
    public void statusUpdate(SchedulerDriver driver, Protos.TaskStatus status
    ) {
        System.out.println("Status update: task " + status.getTaskId().getValue()
                + " is in state " + status.getState().getValueDescriptor().getName());

        if (status.getState() == Protos.TaskState.TASK_FINISHED) {

            /* !!!!! */
            //Can I read the task stdout here?
            /* !!!!! */
        }

    }

    @Override
    public void frameworkMessage(SchedulerDriver driver,
            Protos.ExecutorID executorId,
            Protos.SlaveID slaveId,
            byte[] data
    ) {
    }

    @Override
    public void slaveLost(SchedulerDriver driver, Protos.SlaveID slaveId
    ) {
    }

    @Override
    public void executorLost(SchedulerDriver driver,
            Protos.ExecutorID executorId,
            Protos.SlaveID slaveId,
            int status
    ) {
    }

    public void error(SchedulerDriver driver, String message) {
        System.out.println("Error: " + message);
    }

}

Thanks!

Answer 1

It is not possible to access stdout/stderr of a Mesos task in the way you describe. The Mesos agent by default will write to two files named stdout and stderr within the agent sandbox path of a given task. This content is only be accessible by calling the agent API.

The Mesos master UI works by making an Ajax call to the agent with a URL such as: http://agent:5051/files/browse?path=/var/run/mesos/agents/0/slaves/ab49f68a-b73f-4200-a523-2886353d6450-S0/frameworks/ab49f68a-b73f-4200-a523-2886353d6450-0000/executors/53c98219-f794-4fb6-815d-8b800fc30d9a/runs/0ae1795a-e80f-4f37-ab21-8f78c1286dae/tasks/2981d2db-a218-4c86-9bb2-4371f9be151c/stdout. That path can be inferred with properties that are available to your scheduler.

The new Mesos HTTP Operator (not yet considered stable) can accomplish this via the READ_FILE call. I have some sample code here from a tool I developed (apologies it is not written in Java) which reads from the operator API.

Depending on what kind of scheduling decisions you are trying to make, it might be better to propagate a message back to your scheduler loop via some external channel.