When I start a spark master locally on my windows 7 machine for some reason to WebUI displays the URL and REST URL with the wrong ip address.
Below is my batch script I use to start master where %IP%
is my local IP address.
set SPARK_MASTER_IP=%IP%
set SPARK_PUBLIC_DNS=%IP%
set SPARK_MASTER_PORT=7077
set SPARK_MASTER_WEBUI_PORT=8080
set SPARK_MASTER_REST_PORT=6066
START "spark-master" /MIN "C:/spark-1.6.0-bin-hadoop2.6/bin/spark-class.cmd" ^
org.apache.spark.deploy.master.Master
The master Web UI displays the wrong IP. For some reason it is picking up the IP address of the IPv4 Address belonging to the VirtualBox installed on the same machine. (Note I'm am not using Virutalbox in any way to run Spark)
URL: spark://169.254.237.14:7077
REST URL: spark://169.254.237.14:6066 (cluster mode)
I think the issue is caused by hostname to ip resolution order after VirtualBox is installed. I solved the problem by added my hostname and IP the my
hosts
file.c:\Windows\System32\Drivers\etc\hosts
MY_HOST_NAME
MY_IP