Building and linking dynamically from a go binary

Question

My problem is the following:

1. I have a go binary on a machine
2. From that binary I need to compile an external .go file
3. Once compiled, I need to link the compiled go file into the current binary so I can use the just-compiled go code.

Do you think that's possible ?

I did a few researches and it does not seem to be possible, but I might have overlooked something.

Thanks :)

The first go binary would contain something like

func main() {
    // Here I need to compile an external go file (or package) which contains
    // The definition of runFoo()

    // Once the file/package is compiled and linked I need to call the compiled code
    runFoo()

    // Continue the execution process normally here
}

Answer 1

Feature promiced since 1.5 :) http://talks.golang.org/2015/state-of-go-may.slide#23

Answer 2

Update: It is now possible to do this in mainline Go, see Go Execution Modes

From the Go 1.5 release notes:

For the amd64 architecture only, the compiler has a new option, -dynlink, that assists dynamic linking by supporting references to Go symbols defined in external shared libraries.

Old Answer (useful discussion of other options):

It is not currently possible to create dynamically linked libraries* in main line Go. There has been some talk about this, so you may see support in the future. However, there is a 3rd party go project called goandriod that needed the same functionality you need, so they maintain patches that should allow you to patch the official Go code base to support the dynamic linked support you are requesting.

If you want to use a the standard Go run-time, I would recommend the one of the following. Invoke your Go program from your other program, and communicate using:

1. Pipes to communicate
2. A UNIX domain socket
3. A mmaped region of shared memory.
   1. That is create a file on /dev/shm and have both programs mmap it.
   2. Go mmap library: https://github.com/edsrzf/mmap-go

Each consecutive option will take more effort to setup, be more platform specific, but potentially be more powerful than the previous one.

*Note: That is DLLs in the Windows world, and .so files in the UNIX/Linux world.

Answer 3

I think go plugins could be also related to this question, they are supported from go version 1.8. It allows you to dynamically link go binaries implementing required interfaces at runtime.

For example your code has a dependency for a logging backend, but you'd like to support several of them and resolve it at runtime, elasticsearch and splunk could fit here. You might need to have 2 files: es.go and splunk.go which should both contain a struct of type LoggingBackend implementing a method Write(log string).

To create plugins you need to use buildmode plugin during compilation:

go build -buildmode=plugin -o es.so es.go

go build -buildmode=plugin -o splunk.so splunk.go

After that you could pass the needed plugin via command line arguments and load it:

package main

import "plugin"
import "flag"


type LoggingBackend interface {
    Write(log string)
}
var (
    backend = flag.String("backend", "elasticsearch", "Default logging backend is elasticsearch")
)

func main() {
    flag.Parse()
    var mode string
    switch backend {
    case "elasticsearch":
        mode = "./es.so"
    case "splunk":
        mode = "./splunk.so"
    default:
        fmt.Println("Didn't recognise your backend")
        os.Exit(1)
    plug, _ := plugin.Open(mod)
    loggingBackend, _ := plug.Lookup("LoggingBackend")
    logWriter, _ := loggingBackend.(LoggingBackend)
    logWriter.Write("Hello world")
}

Answer 4

This is very possible, you can even compile it as a native shared library

go build -buildmode=c-shared goc.go 

# file goc
goc: ELF 32-bit LSB  shared object, ARM, EABI5 version 1 (SYSV),
dynamically linked, 
BuildID[sha1]=f841e63ee8e916d7848ac8ee50d9980642b3ad86, 
not stripped

nm -D --defined-only ./goc | grep "T"

0004ebe8 T _cgoexp_f88ec80374ab_PrintInt
000a6178 T _cgo_panic
0004e954 T _cgo_sys_thread_start
000a48c8 T _cgo_topofstack
0004e88c T _cgo_wait_runtime_init_done
000a61a4 T crosscall2
0004ebc8 T crosscall_arm1
0004e7b0 T fatalf
00102648 T _fini
0004e544 T _init
0004e76c T PrintInt
0004ebe4 T __stack_chk_fail_local
0004eb5c T x_cgo_free
0004ea60 T x_cgo_init
0004eb24 T x_cgo_malloc
0004e8e0 T x_cgo_notify_runtime_init_done
0004eb14 T x_cgo_setenv
0004e820 T x_cgo_sys_thread_create
0004eb64 T x_cgo_thread_start
0004eb20 T x_cgo_unsetenv

like so (tested on go 1.5.1 linux/arm)

goc.go:

package main

import (
    "C"
    "fmt"
)

//export PrintInt
func PrintInt(x int) {
    fmt.Println(x)
}

// http://stackoverflow.com/questions/32215509/using-go-code-in-an-existing-c-project
// go build -buildmode=c-archive goc.go
// go build -buildmode=c-shared goc.go 

// https://groups.google.com/forum/#!topic/golang-nuts/1oELh6joLQg
// Trying it on windows/amd64, looks like it isn't supported yet.  Is this planned for the 1.5 release? 
// It will not be in the 1.5 release.
// It would be nice if somebody worked on it for 1.6.
// https://golang.org/s/execmodes

// http://stackoverflow.com/questions/19431296/building-and-linking-dynamically-from-a-go-binary
// go build -linkshared hello.g
// go install -buildmode=shared std



func main() {
    fmt.Println("Hello world")
}

Answer 5

The ability to create shared libraries will be in Go 1.5 in August 2015¹.

From "The State of Go" talk by Andrew Gerrand:

Shared libraries

Go 1.5 can produce Go shared libraries that can be consumed by Go programs.

Build the standard library as shared libraries:

$ go install -buildmode=shared std

Build a "Hello, world" program that links against the shared libraries:

$ go build -linkshared hello.go
$ ls -l hello
-rwxr-xr-x 1 adg adg 13926 May 26 02:13 hello

Go 1.5 can also build Go programs as C archive files (for static linking) or shared libraries (for dynamic linking) that can be consumed by C programs.

[See:] golang.org/s/execmodes

_{¹ Note, gccgo already had limited support for this for some time, Go 1.5 will be the first time this is supported by the regular go build tools.}