go-osi

README

go-osi

A Golang package for dealing with ASAM OSI (Open Simulation Interface) binary trace files and their data.

[!IMPORTANT] This repository is currently under construction. Functionality is limited.

OSI trace files are typically binary files containing multiple messages of the same type. They are separated with uint32s that define their length.

The default protoc toolchain does not provide functionality for this type of message chaining. Instead, wrappers separating the messages from each other are needed.

This package aims at providing the following functionalities:

• Read OSI binary trace files (see reader package)
• Write OSI binary trace files (see writer package)
• Convert OSI data into other structs and formats
• Filter OSI data in a convenient way

The repository currently contains the ASAM OSI release version 3.7.0.
For updating the structs to a newer version, take a look at the generation scripts described in the scripts section.

Usage

To install the package, run:

go get github.com/Shieldine/go-osi

For local development, install dependencies with:

go mod tidy

Reader

This package provides two ways to read OSI trace files containing protobuf messages: a simple function-based reader and a more flexible struct-like reader.

Simple Reader

Use the ReadOSIFile function to read all messages of a specified type from a file at once.

package main

import (
	"github.com/Shieldine/go-osi/pkg/reader"
	"github.com/Shieldine/go-osi/pkg/structs"
	"log"
)

func main() {
	messages, err := reader.ReadOSIFile("trace.osi", reader.GroundTruth)
	if err != nil {
		log.Fatalf("Failed to read OSI file: %v", err)
	}

	for _, msg := range messages {
		// Use the message (e.g., cast to *structs.GroundTruth)
		gt := msg.(*structs.GroundTruth)
		// Process gt...
	}
}

Struct-like Reader

Use OSIFileReader to open a file and read messages incrementally or in batches. This is useful for large files or streaming processing.

package main

import (
	"github.com/Shieldine/go-osi/pkg/reader"
	"github.com/Shieldine/go-osi/pkg/structs"
	"io"
	"log"
)

func main() {
	r, err := reader.NewOSIFileReader("trace.osi", reader.GroundTruth)
	if err != nil {
		log.Fatalf("Failed to open OSI file: %v", err)
	}
	defer r.Close()

	// Read messages one by one
	for {
		msg, err := r.ReadNext()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Fatalf("ReadNext error: %v", err)
		}
		gt := msg.(*structs.GroundTruth)
		// Process gt...
	}

	// Read all remaining messages
	messages, err := r.ReadRest()
	if err != nil {
		log.Fatalf("ReadRest error: %v", err)
	}
	for _, msg := range messages {
		gt := msg.(*structs.GroundTruth)
		// Process gt...
	}

	// Reset to start of file
	if err := r.Reset(); err != nil {
		log.Fatalf("Reset error: %v", err)
	}

	// You can also close and reopen the reader
	if err := r.Close(); err != nil {
		log.Fatalf("Close error: %v", err)
	}
	if err := r.Open(); err != nil {
		log.Fatalf("Open error: %v", err)
	}

	// Read all messages from the start
	allMessages, err := r.ReadAll()
	if err != nil {
		log.Fatalf("ReadAll error: %v", err)
	}
	for _, msg := range allMessages {
		gt := msg.(*structs.GroundTruth)
		// Process gt...
	}
}

This struct-like reader provides methods to ReadNext(), ReadRest(), ReadAll(), Reset(), Close(), and Open() for flexible file reading and lifecycle management. It also provides Size() to check the file size.

Reader Features

• Read OSI trace files containing chained protobuf messages using a simple function or a struct with advanced features.
• Support for reading all messages at once or incrementally.
• Buffering on incremental read.

Writer

The writer package provides functionality to write OSI protobuf messages to files with optional buffering and auto-flush.

Basic Usage Example

package main

import (
	"log"

	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/types/known/anypb"

	"github.com/Shieldine/go-osi/pkg/writer"
)

func main() {
	// Create a new OSIWriter with buffering enabled and autoFlush disabled
	w, err := writer.New("example.osi", true, false)
	if err != nil {
		log.Fatalf("Failed to create OSIWriter: %v", err)
	}
	defer func() {
		if err := w.Close(); err != nil {
			log.Fatalf("Failed to close OSIWriter: %v", err)
		}
	}()

	// Create some protobuf messages (using Any as an example)
	msg1 := &anypb.Any{TypeUrl: "type1", Value: []byte("message1")}
	msg2 := &anypb.Any{TypeUrl: "type2", Value: []byte("message2")}

	// Add a single message
	if err := w.Add(msg1); err != nil {
		log.Fatalf("Failed to add message: %v", err)
	}

	// Add a batch of messages
	if err := w.AddBatch([]proto.Message{msg2}); err != nil {
		log.Fatalf("Failed to add batch: %v", err)
	}

	// Optionally flush buffered data to disk
	if err := w.Flush(); err != nil {
		log.Fatalf("Failed to flush data: %v", err)
	}
}

Features

• Write individual or batch protobuf messages to OSI files.
• Optional in-memory buffering for performance.
• Auto-flush option to flush after every write or batch.
• File truncation and reset with Clean().
• File duplication with Copy().
• File size check with Size().
• Safe closing with flushing.

Scripts

The scripts directory contains scripts with the following purposes:

• download_latest downloads the latest ASAM OSI proto files from the latest release tarball. It will automagically fill the osi_version.proto.in with the needed values.
• generate_structs generates Go structs from the downloaded proto files. Do not modify the generated structs.
• create_osi_sample generates sample OSI traces using the betterosi Python library for testing purposes.

Run the shell scripts with:

sh scripts/download_latest.sh

sh scripts/generate_structs.sh

For the Python script, you will need a working Python interpreter. Install the script's dependencies with:

pip install -r requirements.txt

Then run the script:

python scripts/create_osi_sample.py

The generated traces will be placed in a new directory called testdata. Modify this script to satisfy your needs.

Contributing

If you spot any errors or have ideas for enhancements, please open an issue.

Likewise, pull requests are always welcome.

License

This project is licensed under the MIT License (see LICENSE) for all code written and maintained by this repository’s authors. Generated Go files that derive from ASAM OSI .proto files (in the structs directory) remain subject to ASAM’s license terms. This repository does not redistribute ASAM OSI .proto files.

Third-Party Tools and Code

This project makes use of several third-party tools and specifications:

Protocol Buffers (protoc)
Used for generating Go code from .proto files via the Go codegen plugin.
Protocol Buffers is licensed under the BSD 3-Clause License.

ASAM Open Simulation Interface (OSI)
We use ASAM OSI .proto files for generating Go structs used in this project.
Note: These .proto files are not included in this repository.
Instead, a script is provided to download the original files directly from the official ASAM OSI GitHub repository.
Please review the ASAM license terms for usage and redistribution restrictions.

betterosi Python package
This package is used in a script to generate sample data.
betterosi is licensed under the Mozilla Public License 2.0 (MPL 2.0).