mrnative

README

go-mrnative

A utility to generate native Go language MapReduce applications using gobind.

This program can transform a Go application into a shared library, wrapped in a JAR with Java bindings, ready to use on your Hadoop cluster.

Why?

This is a toy project.

A sane person should use hadoop streaming to write MapReduce programs using a non-JVM language.

Applications generated by go-mrnative will work and probably even be performant, however, because a full Go runtime is embedded in each JVM instance, memory usage is an issue. Additionally, copying data between the JVM and Go runtime can be extremely costly.

How?

This project is under development.

It is already very easy to generate Java bindings to a Go shared library using gobind and gojava.

go-mrnative takes this a step further, creating the boilerplate code necessary to make your Go structs directly available to Hadoop MapReduce.

Installation

The simplest way to install go-mrnative is to use go get.

go get -u github.com/veonik/go-mrnative/...

This will install the go-mrnative libraries as well as a command line utility.

Usage

The go-mrnative tool does two things. First, using the init command, go-mrnative will create a skeleton Mapper/Reducer/Combiner in your go package. Second, using the build command, go-mrnative will generate Java boilerplate and bindings for your mapreduce structs, compile them, and jar them into a file.

Initializing a Go mapreduce project

Create a new directory on your machine, then go inside that directory and run the go-mrnative init command.

mkdir gomapred
cd gomapred
go-mrnative init

go-mrnative will interactively walk through what you want to create. Currently, its possible to create Mapper, Reducer, and Combiners (though Combiners are really just Reducers).

Building a Go mapreduce project

Replace <pkg> in the following example with a valid go package name, such as github.com/veonik/go-mrnative.

go-mrnative build <pkg>
gojava build -s build/java/go <pkg>

This process will be streamlined in upcoming changes.

Documentation

Index

• func Asset(name string) ([]byte, error)
• func AssetDir(name string) ([]string, error)
• func AssetInfo(name string) (os.FileInfo, error)
• func AssetNames() []string
• func GoToHadoopType(gt string) string
• func GoToJavaType(gt string) string
• func GoToValueInHadoopType(gt string) string
• func MustAsset(name string) []byte
• func RestoreAsset(dir, name string) error
• func RestoreAssets(dir, name string) error
• type Func
  • func NewFunc(fn *ast.FuncDecl) *Func
• type Generator
  • func NewGenerator(packages []string) *Generator
  • func (g *Generator) Generate()
• type Initializer
  • func NewInitializer(dir string) *Initializer
  • func (i *Initializer) CollectConfig(out io.Writer, in io.Reader)
  • func (i *Initializer) Initialize()
• type Interface
  • func NewInterface(spec *ast.TypeSpec) *Interface
• type Method
  • func NewInterfaceMethod(recv string, f *ast.Field) *Method
  • func NewMethod(fn *ast.FuncDecl) *Method
• type Package
  • func NewPackage(absPath string, p *build.Package) *Package
• type Param
  • func NewParam(param *ast.Field) *Param
• type Struct
  • func NewStruct(tdef structDecl, methods []*Method) *Struct
  • func (s *Struct) String() string
• type Target
  • func NewTarget(pkg *Package, decl *Struct, typ targetType) *Target
  • func (t *Target) IsMapper() bool
  • func (t *Target) IsReducer() bool

Functions

func Asset

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDir

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy:

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

func AssetInfo

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames

func AssetNames() []string

AssetNames returns the names of the assets.

func GoToHadoopType

func GoToHadoopType(gt string) string

GoToHadoopType converts the given go type into its corresponding MapReduce type.

func GoToJavaType

func GoToJavaType(gt string) string

GoToJavaType converts the given go type into its corresponding Java type.

func GoToValueInHadoopType

func GoToValueInHadoopType(gt string) string

func MustAsset

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func RestoreAsset

func RestoreAsset(dir, name string) error

RestoreAsset restores an asset under the given directory

func RestoreAssets

func RestoreAssets(dir, name string) error

RestoreAssets restores an asset under the given directory recursively

Types

type Func

type Func struct {
	// contains filtered or unexported fields
}

func NewFunc

func NewFunc(fn *ast.FuncDecl) *Func

type Generator

type Generator struct {
	// contains filtered or unexported fields
}

Generator holds the state of the analysis. Primarily used to buffer the output for format.Source.

func NewGenerator

func NewGenerator(packages []string) *Generator

NewGenerator creates a new Generator, ready for use.

func (*Generator) Generate

func (g *Generator) Generate()

type Initializer

type Initializer struct {
	// contains filtered or unexported fields
}

func NewInitializer

func NewInitializer(dir string) *Initializer

func (*Initializer) CollectConfig

func (i *Initializer) CollectConfig(out io.Writer, in io.Reader)

func (*Initializer) Initialize

func (i *Initializer) Initialize()

type Interface

type Interface struct {
	// contains filtered or unexported fields
}

func NewInterface

func NewInterface(spec *ast.TypeSpec) *Interface

type Method

type Method struct {
	*Func
	// contains filtered or unexported fields
}

func NewInterfaceMethod

func NewInterfaceMethod(recv string, f *ast.Field) *Method

func NewMethod

func NewMethod(fn *ast.FuncDecl) *Method

type Package

type Package struct {
	// contains filtered or unexported fields
}

A Package contains information about a given Package.

func NewPackage

func NewPackage(absPath string, p *build.Package) *Package

NewPackage creates a new package, ready for use.

type Param

type Param struct {
	// contains filtered or unexported fields
}

func NewParam

func NewParam(param *ast.Field) *Param

type Struct

type Struct struct {
	// contains filtered or unexported fields
}

func NewStruct

func NewStruct(tdef structDecl, methods []*Method) *Struct

func (*Struct) String

func (s *Struct) String() string

type Target

type Target struct {
	// contains filtered or unexported fields
}

A Target is a declaration we intend to generate a bridge for.

func NewTarget

func NewTarget(pkg *Package, decl *Struct, typ targetType) *Target

func (*Target) IsMapper

func (t *Target) IsMapper() bool

IsMapper returns true if this Target is a Mapper.

func (*Target) IsReducer

func (t *Target) IsReducer() bool

IsReducer returns true if this Target is a Reducer.