Mesos DCOS doesn't install Kafka

Question

I'm trying to install Kafka on Mesos. Installation seems to have succeeded.

vagrant@DevNode:/dcos$ dcos package install kafka
This will install Apache Kafka DCOS Service.
Continue installing? [yes/no] yes
Installing Marathon app for package [kafka] version [0.9.4.0]
Installing CLI subcommand for package [kafka] version [0.9.4.0]
New command available: dcos kafka
The Apache Kafka DCOS Service is installed:
  docs   - https://github.com/mesos/kafka
  issues - https://github.com/mesos/kafka/issues

vagrant@DevNode:/dcos$ dcos package list
NAME   VERSION  APP     COMMAND  DESCRIPTION
kafka  0.9.4.0  /kafka  kafka    Apache Kafka running on top of Apache Mesos

But kafka task is not started.

vagrant@DevNode:/dcos$ dcos kafka
Error: Kafka is not running
vagrant@DevNode:/dcos$

Marathon UI says service is waiting. Looks like it is not accepting resource that was allocated to it. More logs here.

Mar 23 03:52:59 ip-10-0-4-194.ec2.internal java[1425]: [2016-03-23 03:52:59,335] INFO Offer ID: [54f71504-b37a-4954-b082-e1f2a04b7fa4-O77]. Considered resources with roles: [*]. Not all basic resources satisfied: cpu not in offer, disk SATISFIED (0.0 <= 0.0), mem not in offer (mesosphere.mesos.ResourceMatcher$:marathon-akka.actor.default-dispatcher-11)
Mar 23 03:52:59 ip-10-0-4-194.ec2.internal java[1425]: [2016-03-23 03:52:59,370] INFO Offer [54f71504-b37a-4954-b082-e1f2a04b7fa4-O77]. Insufficient resources for [/kafka] (need cpus=0.5, mem=307.0, disk=0.0, ports=(1 dynamic), available in offer: [id { value: "54f71504-b37a-4954-b082-e1f2a04b7fa4-O77" } framework_id { value: "54f71504-b37a-4954-b082-e1f2a04b7fa4-0000" } slave_id { value: "54f71504-b37a-4954-b082-e1f2a04b7fa4-S1" } hostname: "10.0.4.190" resources { name: "ports" type: RANGES ranges { range { begin: 1 end: 21 } range { begin: 23 end: 5050 } range { begin: 5052 end: 32000 } } role: "slave_public" } resources { name: "cpus" type: SCALAR scalar { value: 4.0 } role: "slave_public" } resources { name: "mem" type: SCALAR scalar { value: 14019.0 } role: "slave_public" } resources { name: "disk" type: SCALAR scalar { value: 32541.0 } role: "slave_public" } attributes { name: "public_ip" type: TEXT text { value: "true" } } url { scheme: "http" address { hostname: "10.0.4.190" ip: "10.0.4.190" port: 5051 } path: "/slave(1)" }] (mesosphere.mesos.TaskBuilder:marathon-akka.actor.default-dispatcher-11)

Mesos master logs..

Mar 23 15:38:22 ip-10-0-4-194.ec2.internal mesos-master[1371]: I0323 15:38:22.339759  1376 master.cpp:5350] Sending 2 offers to framework 54f71504-b37a-4954-b082-e1f2a04b7fa4-0000 (marathon) at scheduler-f86b567c-de59-4891-916b-fb00c7959a09@10.0.4.194:60450
Mar 23 15:38:22 ip-10-0-4-194.ec2.internal mesos-master[1371]: I0323 15:38:22.341790  1381 master.cpp:3673] Processing DECLINE call for offers: [ 54f71504-b37a-4954-b082-e1f2a04b7fa4-O373 ] for framework 54f71504-b37a-4954-b082-e1f2a04b7fa4-0000 (marathon) at scheduler-f86b567c-de59-4891-916b-fb00c7959a09@10.0.4.194:60450
Mar 23 15:38:22 ip-10-0-4-194.ec2.internal mesos-master[1371]: I0323 15:38:22.342041  1381 master.cpp:3673] Processing DECLINE call for offers: [ 54f71504-b37a-4954-b082-e1f2a04b7fa4-O374 ] for framework 54f71504-b37a-4954-b082-e1f2a04b7fa4-0000 (marathon) at scheduler-f86b567c-de59-4891-916b-fb00c7959a09@10.0.4.194:60450

No sure why marathon didn't like that offer. I'm fairly sure there is enough resource.

Answer 1

Marathon is waiting for an Offer with enough resources for the Kafka scheduler. The offers it is rejecting appear not to have any cpu or memory. The Offer you see which does have sufficient resources is already statically reserved for the Role "slave_public".

The Kafka scheduler will be running with the Role *. Your cluster lacks sufficient resources in the default Role *. This is the role associated with private Slaves.

You should go look at mesos/state and look at the available Resources on slaves with the "*" role.