Go Execution Modes Ian Lance Taylor August, 2014
Currently the gc compiler supports three execution modes:
A statically linked Go binary. Default for a Go program that does not import the net or os/user packages and does not use cgo or SWIG. A dynamically linked Go binary, linked with the C library for DNS and user name lookup. Default for a Go program that imports the net or os/user packages and does not otherwise use cgo or SWIG. A Go binary linked with arbitrary non-Go code, either statically or dynamically linked. Default for a Go program that uses cgo or SWIG. The interface between Go and non-Go code is a C style API (SWIG permits calling between C++ and Go, but this is implemented using a C style API). Can be selected with -ldflags -linkmode=external.
In general the default works, and most people do not care about the differences between these modes. However, there are other possibilities. This document discusses additional execution modes, how we would select them, and implementation issues.
This is intended as a roadmap for future development. We aren’t going to implement all of this at once, and we may not implement all or even any of it. The hope is that what we do implement will follow this roadmap.
There is no necessary connection between the new execution modes that will be proposed here. They are being discussed together because they all relate to how Go programs are built and run, and because although they are different in effect there is significant overlap in implementation details.
This document draws on earlier documents by Elias Naur and David Crawshaw. APIs
When discussing execution modes it’s necessary to consider the style of API used between different parts of the program. There are two relevant styles: C style and Go style.
A C style API permits only those functions that may be written in C: no channels or maps, functions may return at most one value, etc. This is what the cgo command implements. There are severe restrictions on passing pointers when using a C style API. While the exact details of those restrictions are undetermined, in general a Go pointer may not be passed to a C style API. As we will see, in some cases Go code may wind up calling Go code via a C style API.
A Go style API permits any Go function. Versioning
Most of these execution modes permit code to be built at different times and linked together at runtime. Nevertheless, we require that all Go code that is combined into a single executable image must be built with the same version of the Go toolchain. We require further that if any Go package appears more than once in the executable image, it must be built from the same source code.
This is a very strict requirement. It means that if a C program has a C API style plugin interface, and it opens multiple plugins written in Go, all of those plugins must be built with the same version of the Go toolchain, and any shared packages must be identical. It may be possible to weaken this restriction in the future, but we’ll start with this. The Go runtime
It follows that all Go code shares a single runtime. All Go code uses the same memory allocator, the same goroutine scheduler, and in general acts as though it were linked into a single Go program. This is true even when multiple shared libraries are involved. . New execution modes
These are the new execution modes we propose:
Go code linked into, and called from, a non-Go program. Go code linked into a shared library loaded as a plugin by a program (Go or non-Go) that supports a C style plugin API. Go code linked into a shared library loaded as a plugin by a Go program that supports a general Go style plugin API. A Go program that uses a plugin interface, either C style or Go style, where plugins are implemented as shared libraries. Building a Go package, or collection of packages, as a shared library that may be linked into a Go program. A Go program built as a PIE--a Position Independent Executable.
The following sections describe these execution modes in more detail.
Go code linked into, and called from, a non-Go program
In this mode Go code acts as a library that may be called by a non-Go program. A single Go library will be an archive, a .a file on Unix, that will act as a unit, providing a C style API. If some of the execution modes below are implemented, then it will also be possible to implement this as a shared library, a .so file on Unix.
This mode supports people who must work with large existing programs, especially in C/C++ but possibly in other languages as well. It permits them to extend those existing programs with new packages written in Go.
Go code as a shared library plugin with a C style API
Many programs implement a plugin interface in which a shared library may be loaded at runtime. The shared library implements one or more well-known functions that are called using a C style API. In this mode a Go program may be built into a shared library that may be loaded as a plugin.
This mode makes it possible to write Go plugins for existing programs with a plugin interface.
Go code as a shared library plugin with a Go style API
This is a natural extension of the preceding execution mode. A Go program may implement a plugin interface that uses a Go style API. In this mode Go code may be built into a shared library that may be loaded as a Go style plugin.
Go code that uses a shared library plugin
This is the converse of the previous two modes. This is the ability for Go code to support a plugin interface itself, a plugin interface that may be satisfied by a shared library. The plugin interface may use a C style API, in which case the plugin may be written in any language, or a Go style API, in which case the plugin must be written in Go. The details of how this is implemented are described below.
This mode makes it possible for Go programs to define a plugin interface that permits them to be extended at run time.
Building Go packages as a shared library
In this mode a Go package, or set of packages, may be built as a shared library. A Go program that imports one or more of those Go packages may be linked against this shared library. The shared library may be changed between the time the Go program is linked and the time it is run; the shared library that is available when the program starts is the one that will be used.
C shared libraries provide a variety of techniques that make it possible to change a shared library ABI without requiring the programs linked against it to change: weak symbols, versioned symbols, SONAMEs, dynamic linker configuration files. There is no intention to support any of these techniques for Go. Updating the Go runtime to a new version requires rebuilding all Go programs that use it.
Similarly, there is no intention to support symbol interposition for Go. (I won’t bother to define that here--if you don’t know what this is, don’t worry about it; it won’t work anyway.)
This mode is mainly intended to support distro builders. They can distribute Go packages or groups of packages as shared libraries, and can thus update all Go programs by updating the shared libraries, without requiring the programs to be relinked.
A Go program built as a PIE
In this mode a Go program is built as usual, but the resulting executable is position-independent, and may be relocated at run time.
This mode supports people who care a great deal about security.
Changes to the go tool
All execution modes are selected by options to the go tool. We propose two new build flags: -buildmode and -linkshared. The -buildmode flag will take an argument indicating the build mode, described below. Also, one new package will be introduced.
As mentioned above this is a proposed roadmap. Not all of the build modes will be implemented immediately or simultaneously. Not all of the build modes will be supported on all hosts. On hosts where a requested build mode can not work, the go tool will fail. The -buildmode flag
-buildmode=archive This is the default build mode for a package that is not main. It means to build the package into a .a file. -buildmode=c-archive Requires a main package, but the main function is ignored (init functions are run as usual). Build the main package, plus all packages that it imports, into a single C archive file. The only callable symbols will be those functions marked as exported (by any package), as described in the cgo documentation. -buildmode=shared Combine all the listed packages into a single shared library that will be used when building with the -linkshared option. -buildmode=c-shared Requires a main package as for -buildmode=c-archive. Build the main package, plus all packages that it imports, into a single C shared library. The only callable symbols will be those functions marked as exported. -buildmode=plugin Requires a main package as for -buildmode=c-archive. Build the main package, plus all packages that it imports, into a single shared library that may be loaded as a runtime plugin. -buildmode=exe This is the default build mode for a package named main. It means to build the package and everything it imports into an executable. -buildmode=pie This is like -buildmode=exe, but it builds a Position Independent Executable. [ Consider also -buildmode=guiexe for a Windows GUI program. ]
The -linkshared flag
The -linkshared flag directs to Go tool to use link against shared libraries when available. When no shared library is available for some imported package, the ordinary archive will be used instead. The -linkshared flag may be used with -buildmode=shared, exe, or pie. It may be possible to use -linkshared with -buildmode=c-shared or plugin; more experience with the implementation is required. It does not make sense to -use -linkshared with -buildmode=archive or c-archive.
A new package
To support Go packages that want to use plugins, a new package will be added to the standard library: plugin. It will define a function and a type.
// A plugin opened at runtime. type Plugin / definition, probably a struct but who knows /
// Open the plugin name. func Open(name string) (Plugin, error)
// Look up a symbol in a Go style plugin. You can only look up functions and global variables. func (p Plugin) Lookup(name string) (interface{}, error)
// Look up a symbol in a C style plugin, passing in a pointer to a value with the type it // is expected to have. The value must be a function type with a C style API, or a // C variable type. func (p Plugin) LookupC(name string, valptr interface{}) (error)
// Close the plugin. func (p Plugin) Close() error
Sample usage:
var fc func() *C.char err := p.LookupC(“version”, &fc) fmt.Println(C.GoString(fc()))
v, err := p.Lookup(“Version”) fmt.Println(v.(func() string)())
The Open call may be used for a given plugin multiple times, but it only loaded and initialized once. A second Open call with the same name returns the previously opened instance of the plugin.
Selecting the execution modes
How to implement the execution modes listed above using these options:
Go code linked into, and called from, a non-Go program. -buildmode=c-archive or -buildmode=c-shared. Go code linked into a shared library loaded as a plugin by a program (Go or non-Go) that supports a C style plugin API. -buildmode=c-shared Go code linked into a shared library loaded as a plugin by a Go program that supports a general Go style plugin API. -buildmode=plugin A Go program that uses a plugin interface, either C style or Go style, where plugins are implemented as shared libraries. Use the new plugin package. Building a Go package, or collection of packages, as a shared library that may be linked into a Go program. To build the shared library: -buildmode=shared. To link against shared libraries previously built and installed: -linkshared. A Go program built as a PIE--a Position Independent Executable. -buildmode=pie
Implementation
Although I hope that the new flags are comprehensible, the implementation is somewhat complex. I will discuss the details of ELF based systems, as those are the ones I know best. I hope that similar techniques can be applied on Darwin and Windows.
Multiples copies of a Go package
There are several different ways that multiple copies of a Go package, including the runtime package, can wind up in a single executable image. For example, a C API plugin must include all Go packages that it imports, since they might not otherwise be available. A Go program might then load that plugin. We might be able to eliminate all the ways that we can have multiple copies of a package, but I think it would be more robust to prepare for the possibility and make it work.
Because we require that all Go packages be built from the same sources, we can assume that all the code is the same. What we must ensure is that we always use exactly one copy of the global variables for each package.
To implement this, we must ensure that when there is a possibility of multiple Go packages, we always include a complete set of variables for a package. That is, we must disable linker garbage collection of variables when -buildmode=c-archive, shared, c-shared, or plugin. We must also disable linker garbage collection of variables when the plugin package is imported.
When linking a Go package into a shared library, we must ensure that all the global variables have have default visibility. Note that this applies even to variables that are not exported.
If we follow those guidelines, all references to a Go variable will resolve to the same symbol in the same runtime implementation, so there should be no confusion.
For safety, when a Go package appears in a shared library, it could verify that the package being used has the same version as the one in the shared library. This can be done using a non-Go global constructor that compares a globally visible version variable with the value that the symbol is expected to have. The value of this version variable could be, for example, a hash of the package file contents, generated when the package is built.
Initialization
When there are multiple copies of a package, it is possible that a package may be initialized more than once. All packages must check whether they have been initialized already, and, if they have been, do nothing.
When building a plugin or a C API archive or shared library, there may not be a Go main function. The code must initialize itself using a global constructor. The runtime package will need a new entry point that may be used to initialize the scheduler, heap, etc., if necessary.
PIC
All code that goes into a shared library must be compiled as Position Independent Code. As Go does not support symbol interposition, this is not quite the same as PIC for C; in fact, it is more like PIE for C. Because we know that all the code is the same, it is not necessary to use a PLT to call a locally defined function. All function calls must be PC-relative, or via a PLT for functions defined in a different shared library. All references to global variables (whether exported or not) must use a GOT.
-buildmode=c-archive Link all code together as for an ordinary Go program that uses the external linker, to produce a single object file. If any non-Go code is involved, link it with the Go object using -r. Include a global constructor to run all initialization code. -buildmode=c-shared Like -buildmode=c-archive, but always use the external linker with -shared. Use PIC versions of all code. This means that we need a separate install tree of Go packages compiled as PIC. This will be similar to the separate install tree of Go packages compiled with race detector support. Compile all C code with -fPIC. C symbols can be selectively hidden by passing a version script to the external linker. Go symbols must remain visible. -buildmode=shared Always use the external linker, with -shared. Use PIC versions of all code, as for -buildmode=c-shared. Add code to initialize all packages at global constructor time. We can build multiple packages into a single shared library. When we do this, we must install the shared library and must also install each package separately. The separate package installs will include just the export information and a pointer to the shared library; that is, they won’t include the code. This will be in yet another package tree. The shared library should include a hash of the export data for each package that it contains. It should also include a hash of the export data for each shared library that it imports. These hashes can be used to determine when packages must be recompiled. These hashes should be accessible to any build tool, not just the go tool. I’m not sure where the shared libraries should be installed. The go tool has to be able to find them, but also distro builders need to be able to update them independently. -buildmode=plugin This is like -buildmode=shared; the only difference is the list of packages included in the shared library. For -buildmode=shared we only include listed packages, for -buildmode=plugin we also include all the packages that they import, transitively. -buildmode=pie Always use the external linker, with -pie. Use PIC versions of all Go code. Compile all C code with -fPIE, or use the -fPIC compiled code if that is simpler.
-linkshared
The implementation of the -linkshared option is of course closely tied to the -buildmode=shared option. The -buildmode=shared option will install packages in a separate package tree that provide the export information and point to the shared library. The -linkshared option will first search that separate package tree. If it finds the package, it will link against the shared library. If it does not find a package in the separate package tree, that means that the package was not built with -buildmode=shared; the linker will search the standard package tree to find the package.
The go tool can look at the hashes of shared libraries and of the libraries that they import to determine when it is necessary to recompile a package that depends on an updated shared library.
The linker will add a runtime library search path as needed to find the shared libraries at runtime. This may be overridden with the linker’s -R option.
Caveats
There are some special caveats that apply to Go code linked into a non-Go program, and to Go code implemented as a C style API plugin when the plugin is opened by a non-Go program. In other words, for -buildmode=c-archive or c-shared. Because the code in these cases will not be running as part of a complete Go program, some limitations apply.
The os.Args variable will start off as nil. It will not reflect the command line of the program. It will only become non-nil if the Go code sets it itself. This is because there is no reliable portable mechanism for getting a copy of the command line from the main program.
A plugin will be initialized when it is loaded, which will normally not be the time that the overall program is started. The Go plugin interface provides no way to unload a plugin. The C plugin interface does: dlclose. Because all Go plugins share a single runtime, it’s not clear that it is possible to safely unload a Go plugin. Where possible plugins will be opened with RTLD_NODELETE and shared libraries will be built with -z nodelete.
If the Go code panics, and the panic is not recovered by Go code, the entire program will be terminated. It would be nice to figure out some more friendly approach, but this is what we will start with.
Fully functional Go code requires that the Go runtime control certain signal handlers. For code built in the c-archive or c-shared modes, the Go runtime will install signal handlers as usual, including setting up an alternate signal stack, but will also save the existing signal handlers. If the Go signal handler receives a signal that did not originate in Go code, the Go runtime will call the existing signal handler. If possible it will jump to it directly so that the stack frame looks the same. If the non-Go code overrides the Go signal handler, then some Go code will not work as expected. There is nothing to be done about that (the same issue can arise today when using cgo).
