Spark/Gradle — Getting IP Address in build.gradle

问题:

I understand at a basic level the various moving parts of build.gradle build scripts but am having trouble tying it all together.

In Apache Spark standalone mode, just trying to start a master and worker on the same box from build.gradle. (Later will extend with call with $SPARK_HOME/sbin/start-slaves with the proper argument for masterIP.)

Question: How can I assign my IP address to a variable in Groovy/build.gradle so I can pass it to a command in an Exec task? We want this to run on a couple different development machines.

We have a (I think fairly standard) /etc/hosts config with the FQDN and hostname assigned to 127.0.1.1. The driver gets around this OK but starting master and slaves with hostnames is not an option, I need the ip address.

I am trying:

task getMasterIP (type: Exec){
    // declare script scope variable using no def or
    executable "hostname"
    args += "-I"

    // need results of hostname call assigned to script scope variable
    sparkMasterIP = <resultsOfHostnameCall>
}

// added this because startSlave stops if Master is already running
task startSlaveOnly(dependsOn:'getMasterIP', type: Exec){
    executable "/usr/local/spark/sbin/start-slave.sh"
    args += "spark://$sparkMasterIP:7077"
    doLast {
        println "enslaved"
    }
}

// now make startSlave call startSlaveOnly after the initial startMaster
task startSlave(dependsOn:'startMaster', type: Exec) {
    finalizedBy 'startSlaveOnly'
}

When I try something like suggested in the docs for Exec for Groovy calls:

task getMasterIP (type: Exec){
    // declare script scope variable using no def or
    sparkMasterIP = executable "hostname"
    args += "-I"
}

I get a warning that executable is not recognized.

---

The " for a little more background on what I am thinking" section, not the main question.

Googling "build.gradle script scope variables" and looking at the first two results, in the basic docs I only see one type of variable and ext properties to be used.

16.4. Declaring variables -- There are two kinds of variables that can be declared in a build script: local variables and extra properties.

But in this other Gradle doc Appendix B. Potential Traps I am seeing two kinds of variables scopes aside from the ext properties:

For Gradle users it is important to understand how Groovy deals with script variables. Groovy has two types of script variables. One with a local scope and one with a script-wide scope.

With this example usage:

String localScope1 = 'localScope1'
def localScope2 = 'localScope2'
scriptScope = 'scriptScope'

I am assuming I should be using script-scope variables with no "def" or type declaration. Totally open to completely different approaches, thanks.

回答1:

To fetch local IPs:

//return all v4 addresses 
def getLocalIPv4() {
    def ip4s = []
    NetworkInterface.getNetworkInterfaces()
            .findAll { it.isUp() && !it.isLoopback() && !it.isVirtual() }
            .each {
        it.getInetAddresses()
                .findAll { !it.isLoopbackAddress() && it instanceof Inet4Address }
                .each { ip4s << it }
    }
    return ip4s
}

//optionally, return all ipv6 addresses
def getLocalIPv6() {
    def ip6s = []
    NetworkInterface.getNetworkInterfaces()
            .findAll { it.isUp() && !it.isLoopback() && !it.isVirtual() }
            .each {
        it.getInetAddresses()
                .findAll { !it.isLoopbackAddress() && it instanceof Inet6Address }
                .each { ip6s << it }
    }
    return ip6s
}



task printIP()<<{
    println getLocalIPv4()
    println getLocalIPv6()
}

The two functions above return a list of ipv4 or ipv6 addresses respectively. You might notice that I'm skipping all localhosts, interfaces that are not up, all loopbacks and virtual interfaces. If you want to use the first ipv4 address, you can use it elsewhere as:

getLocalIPv4()[0]

or in your case:

args += "spark:/"+ getLocalIPv4()[0] + ":7077"

回答2:

I found this post that appears to be a more straightforward way of doing this but it limited to Linux platforms, hostname -I doesn't work in Windows and maybe not all Linux distros?

• getting hostname
• assigning it to variable
• using in a build.gradle task

Here's the task I built as a result, the accepted answer is much better and more universal, this is just for another way of looking at it

task getMasterIP{
        doLast {
            new ByteArrayOutputStream().withStream { os ->
                def result = exec {
                    executable = 'hostname'
                    args += '-I'
                }
                ext.ipAddress = os.toString()
            }
        }
}

RaGe's answer does a better job of looking at all interfaces on all platforms