Debezium How do I correctly register the SqlServer

Question

Question:

How do I correctly register the SqlServer connector with Kafka Connect to connect to a standalone SQL Server Instance? Note: I am NOT running SQL Server in Docker.

Error:

Error: Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host 127.0.0.1, port 1433 has failed. Error: "Connection refused (Connection refused). Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.".

Troubleshooting:

1. Enabled CDC on the dbo.Posts table (see below)
2. Verified that TCP/IP is enabled and set all addresses to enabled and active on port 1433 in SQL Server Configuration Manager's SQL Server Network Configuration
3. Disabled Windows Firewall
4. Enabled SQL Server Browser in SQL Server Configuration Manager
5. Verified that I can telnet in on the following: (127.0.0.1 1433; localhost 1433; lt-ls231 1433)

Context:

I am attempting to setup Debezium on Windows 10 using Docker. I can successfully run the following commands in Powershell without problem:

docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.1 

docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.1 

docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e  OFFSET_STORAGE_TOPIC=my_connect_offsets -e  STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka debezium/connect:1.1

I get an error (see above) when I attempt to register the SQL Server connection with Kafka-Connect connect, I run the following command:

Invoke-RestMethod -Method Post -Uri 'http://localhost:8083/connectors/' -Headers @{'Accept' = 'application/json'; 'Content-Type' = 'application/json'} -Body '{"name": "mssqlserver-localhost-testDb-connector",  "config": {"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector", "database.hostname": "127.0.0.1", "database.port": "1433", "database.user": "svc_kafka", "database.password": "password", "database.dbname": "Posts", "database.server.name": "LT-LS231", "table.whitelist": "dbo.Posts", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.LT-LS231" }}'

The formatted JSON is as follows:

Invoke-RestMethod -Method Post -Uri 'http://localhost:8083/connectors/' -Headers @{'Accept' = 'application/json'; 'Content-Type' = 'application/json'} -Body '

{  
    "name": "mssqlserver-localhost-testDb-connector",  
    "config": {
        "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector", 
        "database.hostname": "127.0.0.1", 
        "database.port": "1433", 
        "database.user": "svc_kafka", 
        "database.password": "password", 
        "database.dbname": "Posts", 
        "database.server.name": "LT-LS231", 
        "table.whitelist": "dbo.Posts", 
        "database.history.kafka.bootstrap.servers": "kafka:9092", 
        "database.history.kafka.topic": "dbhistory.LT-LS231" 
  }
}'

This is modeled after the JSON provided on Debezium's site: https://debezium.io/documentation/reference/1.1/connectors/sqlserver.html#sqlserver-deploying-a-connector

Answer 1

The solution for a non-production environment is to use host.docker.internal. This is found on Docker's website here: https://docs.docker.com/docker-for-windows/networking/. See section titled I WANT TO CONNECT FROM A CONTAINER TO A SERVICE ON THE HOST. I will need to look into the Docker0 that is recommended in Jiri's answer.

The correct JSON is as follows:

{  
    "name": "LT-LS231-mssqlserver-connector",  
    "config": {
        "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector", 
        "database.server.name": "staging", 
        "database.hostname": "host.docker.internal", 
        "database.port": "1433", 
        "database.user": "svc_kafka", 
        "database.password": "*********", 
        "database.dbname": "Test", 
        "table.whitelist": "dbo.Posts", 
        "database.history.kafka.bootstrap.servers": "kafka:9092", 
        "database.history.kafka.topic": "dbhistory.Posts" 
  }
}

Answer 2

The problem is in the reference to the localhost. If you run the Connect in Docker container then the localhost is the localhost of the container not of the machine. Try exposing the SQL Server on docker0 interface or all intefaces and then use docker0 IP in the registration request.