README
¶
binappend
Library for packing binary data onto the end of files, targeted at adding adding static assets to executables after they're compiled.
A cli tool for reading and writing this "file format" can be found here.
The following is copied from this issue on github.com/gobuffalo/packr:
Background
For one of my own projects, I ran into the problem that the memory overhead for embedding some of the files involved was unacceptably high over the potential lifetime of the program. I still wanted self contained binaries though, so I did some digging and ran into this on stackoverflow, and as it turns out none of the major operating systems care if you dump crap at the end of an executable. This, combined with the discovery of the osext library for finding the path to the current executable prompted the development of the following scheme for embedding files, which I have working as far as I need for my own purposes at this point.
High level description
- Compile the binary normally
- Write each file to embed to the end of the binary through a
gzip.writer, noting the start and end point on the file. - Write the gathered start and end points, along with their names (paths) out to a json-encoded metadata block at the end of binary
- Write the start position of said json data out as a magic number for the last 8 bytes of the file.
The self-extraction process looks like this, then:
- Find the path to the source of the current process
- Open the file
- Read the last 8 bytes
- Open a json.Decoder at the position written to the last 8 bytes
- Decode the json into a struct in memory
- Close the file
Then, when the process wants to access some of the data in the file, a file reader is opened on it that is simply started at $start_ptr and is limited to reading $size bytes
Visualization:
0 400 500 7000
+------------------------+-------------------------+----------------+---------------------+--------------+
| Executable Binary Data | Embeded File ./assets/A | Embeded Data B | Json metadata block | Magic Number |
+------------------------+-------------------------+----------------+---------------------+--------------+
In this case the json would look something like this:
{
"Version": "0.1",
"Data":
{
"./assets/A": { "start_ptr": 400, "size": 100 },
"B": { "start_ptr": 500, "size": 6500 }
}
}
And the magic number would be 7000
Documentation
¶
Index ¶
- Constants
- type Appender
- func (appender *Appender) AppendByteArray(name string, data []byte, compressed bool) error
- func (appender *Appender) AppendFile(source string, compressed bool) error
- func (appender *Appender) AppendStreamReader(name string, source io.Reader, compressed bool) error
- func (appender *Appender) Close() error
- func (appender *Appender) FileName() string
- type Extractor
- type Reader
Constants ¶
const METADATA_VERSION string = "0.2"
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Appender ¶
type Appender struct {
// contains filtered or unexported fields
}
func MakeAppender ¶
Procedure:
MakeAppender
Purpose:
To create an Appender
Parameters:
The name of the file to append to: filename string A function that wraps an io.Writer: writeWrapper This can be used to pre-process data before insertion Note: this function will be called every time a file/stream is added
Produces:
A pointer to a new Appender: output *Appender Any filesystem errors that occur in opening $filename: err error
Preconditions:
The file at filename exists and can be written to
Postconditions:
An appender is created that will append to filename through writeWrapper The caller of this function closes the created Appender
func (*Appender) AppendByteArray ¶
Procedure:
*Appender.AppendByteArray
Purpose:
To append a bytearray to a file
Parameters:
The name of the data: name string The data to append: data []byte Whether to compress the data: compressed bool
Produces:
Side effects: filesystem internal state changes Any errors in writing to the filesystem: err error
Preconditions:
No additional, although you should probably make sure that data is not empty
Postconditions:
AppendSteamReader is called with a reader wrapped around data.
func (*Appender) AppendFile ¶
Procedure:
*Appender.AppendFile
Purpose:
To gzip and pack a file onto the end of the Appender's file
Parameters:
The calling Appender: appender Appender The file to append: source string Whether or not to compress the file: compressed bool
Produces:
Side effects: filesystem internal state changes Any errors in writing to the filesystem: err error
Preconditions:
$source exists and is readable in the file system $source has not been appended already nor has $appender.AppendStreamReader(name, _) been called with name == $source $appender.Close() has not been called
Postconditions:
A reader stream from $source will be passed to $appender.AppendStreamReader, with the name parameter as source
func (*Appender) AppendStreamReader ¶
Procedure:
*Appender.AppendStreamReader
Purpose:
To append the entirety of a stream in an appended file block
Parameters:
The parent *Appender: appender The unique name of the stream: name string The reader to pull data out of: source io.Reader Whether to compress the stream: compressed bool
Produces:
Side effects Any errors in writing to the filesystem: err error
Preconditions:
reader has a finite amount of data to read $appender.Close() has not been called
Postconditions:
All of the data that reader can read has been written to appender's internal writer at the end of its file appender's internal metadata has been updated to reflect the addition Errors will be filesystem related if $compressed: bash equivalent is executed: $source | gzip >> $appender.file else $source >> $appender.file $appender.file.ByteArray()[$appender.metadata[$name].StartFilePtr[:$appender].metadata[$name].[BlockSize].gunzip() == $source.ByteArray[]
func (*Appender) Close ¶
Procedure:
*Appender.Close()
Purpose:
To finish writing to the file being appended to and free it for use elsewhere
Parameters:
The Appender being acted upon: appender
Produces:
Any filesystem errors: err error
Preconditions:
$appender.Close() has not been called
Postconditions:
The json-encoded metadata about the appended files has been written out to the end of file being appended to The start of said json block is encoded in the final 8 bytes of the file being appended to as a little endian int64 The internal file handle for the file being appended to has been closed
type Extractor ¶
type Extractor struct {
// contains filtered or unexported fields
}
Type:
Extractor
Purpose:
To provide an concurrent interface for reading files tacked on the end of a binary
func MakeExtractor ¶
Procedure:
MakeExtractor
Purpose:
To create a Extractor for a given file
Parameters:
The file to open: filename string
Produces:
A pointer to a Extractor: reader *Extractor Any errors that occur: err error
Preconditions:
filename exists on the filesystem filename was appended to with by a er
Postconditions:
reader is initialized to grab the files from files
func (*Extractor) AvalibleData ¶
Procedure:
*Extractor.AvalibleData
Purpose:
To return all avalible names to read
Parameters:
The *Reader being acted upon: reader
Produces:
A list of all names of data that can be read: names []string
Preconditions:
No additional
Postconditions:
All the name keys of avalible data are in $names
func (*Extractor) ByteArray ¶
Procedure:
*Extractor.ByteArray
Purpose:
To read all of a block of appended data to a byte array
Parameters:
The parent *Extractor: extractor The name of the data to retrieve: dataName string
Produces:
The data named dataName: data []byte Any errors raised: err error
Preconditions:
The extractor is has some data named $dataName
Postconditions:
data contains all the data named $dataName in the extractor err will be a file system error, gzip error, or due to $dataName not existing
func (*Extractor) GetReader ¶
Procedure:
*Extractor.GetDataReader
Purpose:
To return provide a reader matching a data name appended to reader's file
Parameters:
The *Extractor being called: reader The name of the data given: dataName string
Produces:
A reader for the data by the same name: reader *Reader Errors produced: err error
Preconditions:
dataName is a name that exists and has data associated with it
Postconditions:
The returned reader will decompress and read back the data with $dataName err will only exist: - When any filesystem errors in opening and seeking in the underlying binary - When $dataName does not match any names in the file
type Reader ¶
type Reader struct {
//The name of the data as inputed by the Writer
Name string
// contains filtered or unexported fields
}
Type:
Reader
Purpose:
Generated by Extractor to provide an interface to read appended data
Explicitly implements:
io.Reader io.Closer io.ReadCloser
Postconditions:
Must be closed so the underlying *os.File can be freed
func (*Reader) Close ¶
Procedure:
*Reader.Close
Purpose:
To clean up the file handle for a Reader
Parameters:
The *Reader being acted upon: reader None
Produces:
Any errors in closing the underlying filesystem handle: err error
Preconditions:
No additonal
Postconditions:
All resources for the Reader have been closed
func (*Reader) Read ¶
Procedure:
*Reader.Read
Purpose:
To read bytes out of a Reader
Parameters:
The *Reader being acted upon: reader The byte array to place read bytes into: p []byte
Produces:
The number of bytes read: n int Any errors in reading: err error
Preconditions:
No additional
Postconditions:
See the documentation for io.Reader
Source Files