Go Busybox
Go Busybox is a set of Go tools that allow you to compile many Go commands into
one binary. The resulting binary uses its invocation arguments (os.Args
) to
determine which command is being called.
Feature |
Support status |
Go version |
1.13+ |
Packaging |
Go modules, Go vendoring, bazel w/ rules_go |
GOOS |
linux (others may work, but untested) |
GOARCH |
amd64, arm, arm64, riscv64 (others may work, but untested) |
CGO |
Not supported |
Other GOARCH
and GOOS
architectures are likely to work as well, but are
untested.
An example:
(cd ./src/cmd/makebb && go install)
makebb ./test/nested/cmd/dmesg ./test/nested/cmd/strace
A binary named bb
should appear. It can be invoked in one of two ways --
either with a symlink or using a second argument.
# Make a symlink dmesg -> bb
ln -s bb dmesg
# Symlink means that argv[0] is the command name.
./dmesg
# Make a symlink strace -> bb
ln -s bb strace
./strace echo "hi"
If symlinks are a hassle, you can also invoke the binary like this:
./bb dmesg
./bb strace echo "hi"
Go Busybox does this by copying all the source for these Go commands and
rewriting it in a temporary directory.
Go Busybox can be used with any Go commands across multiple Go modules:
git clone https://github.com/hugelgupf/p9
git clone https://github.com/gokrazy/gokrazy
makebb ./p9/cmd/* ./gokrazy/cmd/*
For the moment, makebb
is tested with repositories on the local file system.
Using Go import paths is supported, as well, but not as well-tested.
APIs
Besides the makebb CLI command, there is a
Go API at src/pkg/bb
and bazel rules in src/gobb2.bzl.
Using bazel go_busybox rule
Assuming you have rules_go set up, add
the following to your WORKSPACE
:
git_repository(
name = "com_github_u_root_gobusybox",
# We do not have regular releases yet.
#
# We also do not guarantee compatibility yet, so it may be worth choosing a
# commit and setting `commit = "hash"` here instead of the branch.
branch = "main",
remote = "https://github.com/u-root/gobusybox.git",
)
Then, in any BUILD
file, you can create a busybox like this:
load("@com_github_u_root_gobusybox//src:gobb2.bzl", "go_busybox")
go_busybox(
name = "bb",
cmds = [
# These must be absolute labels, for the moment, and each command must
# be listed individually. (No :... or :all target patterns.)
"//cmd/foobar",
"//cmd/otherbar",
# Another repository's go_binarys are totally fine, e.g. if imported
# with gazelle's go_repository rule.
"@com_github_u-root_u-root//cmds/core/ls",
],
)
For the moment, the targets listed on cmds
must be individual, absolute
labels (issue #38).
Shortcomings
-
Any imported packages' init
functions are run for every command.
For example, if some command imports the testing
package, all commands in
the busybox will have testing's flags registered as a side effect, because
testing
's init function runs with every command.
While Go busybox handles every main commands' init functions, it does not
handle dependencies' init functions. Done properly, it would have to rewrite
all non-standard-library packages as well as commands. This has not been
necessary to implement so far. It would likely be necessary if, for example,
two different imported packages register the same flag unconditionally
globally.
-
There are still some issues with Go module dependency resolution. Please
file an issue if you
encounter one, even if it turns out to be your own issue -- our error
messages should be telling users what to fix and why.
People who use this project tend to hang out on the
#u-root-dev channel on the
Open Source Firmware Slack
(Sign Up Link).
Common Dependency Conflicts
If commands from more than one Go module are combined into a busybox, there are
a few common dependency pitfalls to be aware of. Go busybox will do its best to
log actionable suggestions in case of conflicts.
It's important to be aware that not all go.mod
files are equal. The
main module is the module containing the
directory where the go
command is invoked. replace
and exclude
directives
only apply in the main module's go.mod
file and are ignored in other modules.
If Go busybox is asked to combine programs under different main modules, it will
do its best to merge the replace
and exclude
directives from all main module
go.mod
files.
Let's say, for example, that u-root's
cmds/core/*
is being combined into a busybox with
u-bmc's cmd/*
. Each have a main module
go.mod
, one at u-root/go.mod
and one at u-bmc/go.mod
.
$ cat ./u-root/go.mod
...
replace github.com/intel-go/cpuid => /somewhere/cpuid
exclude github.com/insomniacslk/dhcp v1.0.2
$ cat ./u-bmc/go.mod
...
replace github.com/intel-go/cpuid => /somewhere/cpuid
exclude github.com/mdlayher/ethernet v1.0.3
Go busybox generated go.mod
(does not list require
statements):
...
# Because *both* u-root/go.mod and u-bmc/go.mod pointed to a local copy of cpuid
replace github.com/intel-go/cpuid => ./src/github.com/intel-go/cpuid
# From either go.mod file.
exclude github.com/insomniacslk/dhcp v1.0.2
exclude github.com/mdlayher/ethernet v1.0.3
Certain conflicts can come up during this prcess. This section covers each
potential conflict and potential solutions you can enact in your code:
-
Conflicting local commands. E.g. two local copies of u-root
and u-bmc
are being combined into a busybox with makebb ./u-root/cmds/core/* ./u-bmc/cmd/*
. If u-bmc/go.mod
depends on u-root@v3 from GitHub, that
conflicts with the local ./u-root
being requested with makebb.
Solution: u-bmc/go.mod
needs replace github.com/u-root/u-root => ../u-root
.
-
Conflicting local replace
directives. Ex:
> u-root/go.mod
replace github.com/insomniacslk/dhcp => ../local/dhcp
> u-bmc/go.mod
require github.com/insomniacslk/dhcp/v2 v2.0.0.-...
u-root has replaced dhcp, but u-bmc still depends on the remote dhcp/v2.
Solution: u-root drops local replace rule, or u-bmc/go.mod
needs
replace github.com/insomniacslk/dhcp => $samedir/local/dhcp
as well.
-
Two conflicting local replace
directives. Ex:
> u-root/go.mod
replace github.com/insomniacslk/dhcp => /some/dhcp
> u-bmc/go.mod
replace github.com/insomniacslk/dhcp => /other/dhcp
Solution: both go.mod files must point replace github.com/insomniacslk/dhcp
at the same directory.
How It Works
src/pkg/bb implements a Go source-to-source transformation on pure
Go code (no cgo).
This AST transformation does the following:
- Takes a Go command's source files and rewrites them into Go package files
(almost) without global side effects.
- Writes a
main.go
file with a main()
that calls into the appropriate Go
command package based on argv[0]
or argv[1]
.
This allows you to take two Go commands, such as Go implementations of sl
and
cowsay
and compile them into one binary.
Which command is invoked is determined by argv[0]
or argv[1]
if argv[0]
is
not recognized. Let's say bb
is the compiled binary; the following are
equivalent invocations of sl
and cowsay
:
(cd ./src/cmd/makebb && go install)
makebb ./test/nested/cmd/dmesg ./test/nested/cmd/strace
# Make a symlink dmesg -> bb
ln -s bb dmesg
./dmesg
# Make a symlink strace -> bb
ln -s bb strace
./strace echo "hi"
./bb dmesg
./bb strace echo "hi"
Principally, the AST transformation moves all global side-effects into callable
package functions. E.g. main
becomes registeredMain
, each init
becomes
initN
, and global variable assignments are moved into their own initN
. A
registeredInit
calls each initN
function in the correct init order.
Then, these registeredMain
and registeredInit
functions can be registered
with a global map of commands by name and used when called upon.
Let's say a command github.com/org/repo/cmds/sl
contains the following
main.go
:
package main
import (
"flag"
"log"
)
var name = flag.String("name", "", "Gimme name")
func init() {
log.Printf("init %s", *name)
}
func main() {
log.Printf("train")
}
This would be rewritten to be:
package sl // based on the directory name or bazel-rule go_binary name
import (
"flag"
"log"
"../bb/pkg/bbmain" // generated import path
)
// Type has to be inferred through type checking.
var name *string
func init0() {
log.Printf("init %s", *name)
}
func init1() {
name = flag.String("name", "", "Gimme name")
}
func registeredInit() {
// Order is determined by go/types.Info.InitOrder.
init1()
init0()
}
func registeredMain() {
log.Printf("train")
}
func init() {
bbmain.Register("sl", registeredInit, registeredMain)
}
Generated main.go
The main.go file is generated from
./src/pkg/bb/bbmain/cmd/main.go.
package main
import (
"os"
"log"
"path/filepath"
// Side-effect import so init in sl calls bbmain.Register
_ "github.com/org/repo/cmds/sl"
"../bb/pkg/bbmain"
)
func main() {
bbmain.Run(filepath.Base(os.Argv[0]))
}
Directory Structure
All files are written into a temporary directory. All dependencies that can be
found on the local file system are also written there.
The directory structure we generate resembles a $GOPATH-based source tree, even
if we are combining module-based Go commands. This just lends itself to code
reuse within bb: if you remove all the go.mod file, and add in vendored files,
the tree still compiles.
/tmp/bb-$NUM/
└── src
├── bb.u-root.com
│ └── bb
│ ├── go.mod << generated main module go.mod (see below)
│ ├── main.go << ./src/pkg/bb/bbmain/cmd/main.go (with edits)
│ └── pkg
│ └── bbmain
│ └── register.go << ./src/pkg/bb/bbmain/register.go
└── github.com
└── u-root
├── u-bmc
│ ├── cmd
│ │ ├── fan << generated command package
│ │ ├── login << generated command package
│ │ └── socreset << generated command package
│ ├── go.mod << remote dependency manifest copied from u-bmc (if module)
│ └── pkg
│ ├── acme << local dependency copied from u-bmc
│ ├── aspeed << local dependency copied from u-bmc
│ ├── gpiowatcher << local dependency copied from u-bmc
│ └── mtd << local dependency copied from u-bmc
└── u-root
├── cmds
│ └── core
│ ├── cat << generated command package
│ ├── ip << generated command package
│ └── ls << generated command package
├── go.mod << remote dependency manifest copied from u-root (if module)
└── pkg
├── curl << local dependency copied from u-root
├── dhclient << local dependency copied from u-root
├── ip << local dependency copied from u-root
├── ls << local dependency copied from u-root
└── uio << local dependency copied from u-root
Dependency Resolution
There are two kinds of dependencies we care about: remote go.mod dependencies,
and local file system dependencies.
For remote go.mod dependencies, we copy over all go.mod files into the
transformed dependency tree. (See u-root/go.mod and u-bmc/go.mod in the example
above.)
Local dependencies can be many kinds, and they all need some special attention:
- non-module builds: dependencies in $GOPATH need to either be copied into the
new tree, or we need to set our
GOPATH=/tmp/bb-$NUM:$GOPATH
to find these
dependencies.
- non-module builds: dependencies in vendor/ need to be copied into the new
tree.
- module builds: dependencies within a command's own module (e.g.
u-root/cmds/core/ls depends on u-root/pkg/ls) need to be copied into the new
tree.
- module builds:
replace
d modules on the local file system. replace
directives are only respected in
main module go.mod files, which would be
u-root/go.mod
and u-bmc/go.mod
respectively in the above example. The
compiled busybox shall respect all main modules' replace
directives,
so they must be added to the generated main module go.mod.
Generated main module go.mod
The generated main module go.mod refers packages to their local copies:
package bb.u-root.com # some domain that will never exist
replace github.com/u-root/u-root => ./src/github.com/u-root/u-root
replace github.com/u-root/u-bmc => ./src/github.com/u-root/u-bmc
# also, this must have copies of `replace` and `exclude` directives from
# u-root/go.mod and u-bmc/go.mod
#
# if these fundamentally conflict, we cannot build a unified busybox.
If u-root/go.mod
and u-bmc/go.mod
contained any replace
or exclude
directives, they also need to be placed in this go.mod, which is the main module
go.mod for bb/main.go
.