How do you configure Vagrant so host can communica

Question

I have kafka-docker running on a Vagrant Ubuntu VM. I can produce and consume messages to it using NodeJs programs running on the VM.

I am unable however to use the same NodeJs programs from the Windows host, the programs seem to reach but no answer is received.

Here is my docker-compose.yml file:

 version: '2'
 services:
   zookeeper:
     image: "wurstmeister/zookeeper:latest"
     network_mode: "host"
     ports:
       - "2181:2181"
   kafka:
     image: "wurstmeister/kafka:latest"
     network_mode: "host"
     ports:
       - 9092:9092
     environment:
       KAFKA_LISTENERS: PLAINTEXT://:9092
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.17.0.1:9092
       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
       KAFKA_CREATE_TOPICS: "BertTopic:3:1"

Here is my Vagrantfile:

SRCMOUNT = "/hyperledger"
LOCALDEV = "/local-dev"

$script = <<SCRIPT
set -x

echo "127.0.0.1 couchdb" | tee -a /etc/hosts

cd #{SRCMOUNT}/fabric/devenv
./setup.sh

SCRIPT

Vagrant.require_version ">= 1.7.4"
Vagrant.configure('2') do |config|
  config.vm.box = "ubuntu/xenial64"
  config.vm.network :forwarded_port, guest: 2181, host: 2181, id: "zookeeper", host_ip: "localhost", auto_correct: true # zookeeper 
  config.vm.network :forwarded_port, guest: 9092, host: 9092, id: "kafka", host_ip: "localhost", auto_correct: true # kafka
  config.vm.network :forwarded_port, guest: 7050, host: 7050, id: "orderer", host_ip: "localhost", auto_correct: true # fabric orderer service
  config.vm.network :forwarded_port, guest: 7051, host: 7051, id: "peer", host_ip: "localhost", auto_correct: true # fabric peer service
  config.vm.network :forwarded_port, guest: 7053, host: 7053, id: "peer_event", host_ip: "localhost", auto_correct: true # fabric peer event service
  config.vm.network :forwarded_port, guest: 7054, host: 7054, id: "ca", host_ip: "localhost", auto_correct: true # fabric-ca service
  config.vm.network :forwarded_port, guest: 5984, host: 15984, id: "couchdb", host_ip: "localhost", auto_correct: true # CouchDB service
  config.vm.synced_folder "../..", "#{SRCMOUNT}"
  config.vm.synced_folder "../..", "/opt/gopath/src/github.com/hyperledger"
  config.vm.synced_folder ENV.fetch('LOCALDEVDIR', "../.."), "#{LOCALDEV}"
  config.vm.provider :virtualbox do |vb|
    vb.name = "kafkadocker"
    vb.customize ['modifyvm', :id, '--memory', '4096']
    vb.cpus = 2

    storage_backend = ENV['DOCKER_STORAGE_BACKEND']
    case storage_backend
    when nil,"","aufs","AUFS"
      # No extra work to be done
    when "btrfs","BTRFS"
      # Add a second disk for the btrfs volume
      IO.popen("VBoxManage list systemproperties") { |f|

        success = false
        while line = f.gets do
          # Find the directory where the machine images are stored
          machine_folder = line.sub(/^Default machine folder:\s*/,"")

          if line != machine_folder
            btrfs_disk = File.join(machine_folder, vb.name, 'btrfs.vdi')

            unless File.exist?(btrfs_disk)
              # Create the disk if it doesn't already exist
              vb.customize ['createhd', '--filename', btrfs_disk, '--format', 'VDI', '--size', 20 * 1024]
            end

            # Add the disk to the VM
            vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', btrfs_disk]
            success = true

            break
          end
        end
        raise Vagrant::Errors::VagrantError.new, "Could not provision btrfs disk" if !success
      }
    else
      raise Vagrant::Errors::VagrantError.new, "Unknown storage backend type: #{storage_backend}"
    end

  end

  config.vm.provision :shell, inline: $script
end

My ifconfig on the Linux VM shows:

docker0   Link encap:Ethernet  HWaddr 02:42:11:89:57:c8
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0

Here is my /etc/hosts file on Ubuntu:

127.0.0.1       localhost
127.0.0.1               zookeeper
127.0.0.1               kafkadocker

Any help greatly appreciated.

Answer 1

Finally figured out how to get past the "Error: connect ETIMEDOUT" when I could see the nodejs program in Windows connecting but failing to follow through.

Here is the simple recipe.

First, in the Vagrantfile, in addition to the forwarded ports lines listed above, provide the Oracle Virtualbox vm a proper name with line:

# after line: config.vm.provider :virtualbox do |vb|
    vb.name = "kafkadocker"

Start the vm with:

vagrant up

Next, in a Windows command prompt, do:

ipconfig

Note the IP address of the Oracle Virtualbox vm, for me the importants lines were:

Ethernet adapter VirtualBox Host-Only Network:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::9900:a662:107f:3420%11
   IPv4 Address. . . . . . . . . . . : 192.168.56.1

Next, we need to configure Windows to understand when it is told to connect to server "kafkadocker" that it needs to go to IP address 192.168.56.1, so open Notepad++ AS AN ADMINISTRATOR and add the following line to file C:\Windows\System32\drivers\etc\hosts :

192.168.56.1    kafkadocker

Next, in the vm, edit docker-compose.yml and change it to:

version: '2'
services:
  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    network_mode: "host"
    ports:
      - "2181:2181"
  kafka:
    image: "wurstmeister/kafka:latest"
    network_mode: "host"
    ports:
      - 9092
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafkadocker:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "BertTopic:3:1"

The important line is KAFKA_ADVERTISED_LISTENERS. On connect, Kafka will tell the client on Windows to passthru to server "kafkadocker". Now thanks to the Windows hosts file, the node program will know which IP to go to.

Now start kafka-docker in the VM with:

 docker-compose up -d

Next in Windows open Visual Studio code:

cd c:\nodecode
code .

And run the nodejs programs. For producer:

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    KeyedMessage = kafka.KeyedMessage,
    client = new kafka.Client(),
    producer = new Producer(client),
    km = new KeyedMessage('key', 'message'),
    payloads = [
        { topic: 'BertTopic', messages: 'first message', partition: 0 },
        { topic: 'BertTopic', messages: 'second message', partition: 0 },
        { topic: 'BertTopic', messages: 'third message', partition: 0 }
    ];
producer.on('ready', function () {
    producer.send(payloads, function (err, data) {
        console.log(data);
        process.exit(0);
    });
});

producer.on('error', function (err) {
console.log('ERROR: ' + err.toString());
});

For consumer (reading all messages from beginning):

var kafka = require('kafka-node'),
    Consumer = kafka.Consumer,
    client = new kafka.Client(),
    consumer = new Consumer(
        client,
        [
              { topic: 'BertTopic', partition: 0, offset: 0 }
        ],
        { fromOffset: true }         
    );

consumer.on('message', function (message) 
{
    console.log(message);
});

consumer.on('error', function (err) 
{
    console.log('ERROR ' + err.toString());
});

Hope this helps someone.