streamer

README

streamer

streamer is a Go library for streaming binary data (such as images, scientific data, etc.) via HTTP endpoints. It supports chunked binary streaming, WebSocket streaming, and zip packaging. The library is designed to be extensible for machine learning, data visualization, or large dataset delivery scenarios.

---

Features

• Stream binary data over HTTP in customizable chunk sizes
• WebSocket support for real-time binary streaming
• ZIP archive generation on-the-fly
• Pluggable readers for different data sources (e.g., image directories, .npy files)
• Gin-based HTTP handlers
• Simple interface (BinaryReader) for implementing custom data readers
• Includes unit tests for all core components

---

Installation

go get github.com/CHESSComputing/golib/streamer

Make sure to import it in your code:

import "github.com/CHESSComputing/golib/streamer"

---

Usage

Define a Reader

To use the library, implement the BinaryReader interface:

type BinaryReader interface {
    ReadChunk(chunkSize int) (*Chunk, error)
    Reset() error
}

A Chunk includes:

type Chunk struct {
    ContentType string
    Data        []byte
}

Built-in Readers

• ImageReader reads images from a directory.
• NPYReader reads NumPy .npy files.

Example:

reader, err := streamer.NewImageReader("/path/to/images")

---

HTTP Streaming Handlers

Chunked Binary Stream

router.GET("/stream", streamer.GinBinaryStreamHandler(reader))

Query parameter chunk can control the chunk size (default: 1).

ZIP Archive Download

router.GET("/zip", func(c *gin.Context) {
    streamer.StreamAsZip(c, reader, "bundle.zip")
})

Each chunk will be a separate file in the resulting ZIP archive.

WebSocket Streaming

router.GET("/ws", func(c *gin.Context) {
    streamer.WebSocketStreamer(c.Writer, c.Request, reader)
})

Clients will receive binary messages per chunk.

---

License

MIT License. See LICENSE for details.

---

Examples

Here is an example how to use streamer library with Gin web framework:

package main

import (
	"log"

	"github.com/gin-gonic/gin"

	"github.com/chesscomputing/golib/streamer"
)

func main() {
	r := gin.Default()

	imgReader, err := streamer.NewImageReader("images")
	if err != nil {
		log.Fatal(err)
	}
	npyReader, err := streamer.NewNPYReader("npy")
	if err != nil {
		log.Fatal(err)
	}

	//	r.GET("/stream/images", streamer.GinBinaryStreamHandler(imgReader))
	r.GET("/stream/images", streamer.MakeImageReaderHandler("images"))
	r.GET("/image/:index", streamer.MakeOneImageReaderHandler("images"))
	r.GET("/stream/numpy", streamer.GinBinaryStreamHandler(npyReader))
	r.GET("/stream/images.zip", func(c *gin.Context) {
		reader, _ := streamer.NewImageReader("images")
		streamer.StreamAsZip(c, reader, "images_bundle.zip")
	})

	// New WebSocket Endpoints
	r.GET("/ws/images", streamer.WebSocketStreamHandler(imgReader))
	r.GET("/ws/numpy", streamer.WebSocketStreamHandler(npyReader))

	log.Println("Listening on http://localhost:8080")
	r.Run(":8080")
}

Documentation

Index

• func GinBinaryStreamHandler(reader BinaryReader) gin.HandlerFunc
• func GinMultipartImageStreamHandler(reader *ImageReader) gin.HandlerFunc
• func MakeImageReaderHandler(dir string) gin.HandlerFunc
• func MakeOneImageReaderHandler(dir string) gin.HandlerFunc
• func StreamAsZip(c *gin.Context, reader BinaryReader, zipName string)
• func WebSocketStreamHandler(reader BinaryReader) gin.HandlerFunc
• type BinaryReader
• type Chunk
• type ImageChunk
• type ImageReader
  • func NewImageReader(dir string) (*ImageReader, error)
  • func (r *ImageReader) ReadChunk(idx int) (*Chunk, error)
  • func (r *ImageReader) Reset() error
• type NPYReader
  • func NewNPYReader(dir string) (*NPYReader, error)
  • func (r *NPYReader) ReadChunk(_ int) (*Chunk, error)
  • func (r *NPYReader) Reset() error

Functions

func GinBinaryStreamHandler

func GinBinaryStreamHandler(reader BinaryReader) gin.HandlerFunc

GinBinaryStreamHandler provides binary streamer handler function for gin web framework

func GinMultipartImageStreamHandler

func GinMultipartImageStreamHandler(reader *ImageReader) gin.HandlerFunc

GinMultipartImageStreamHandler provides multipart image streamer handler for dealing with multiple images in a stream

func MakeImageReaderHandler

func MakeImageReaderHandler(dir string) gin.HandlerFunc

MakeImageReaderHandler provides image handler based on multipart image streamer

func MakeOneImageReaderHandler

func MakeOneImageReaderHandler(dir string) gin.HandlerFunc

MakeOneImageReaderHandler provides one image reader handler, i.e. it reads single image from images area

func StreamAsZip

func StreamAsZip(c *gin.Context, reader BinaryReader, zipName string)

StreamAsZip provides zip streamer

func WebSocketStreamHandler

func WebSocketStreamHandler(reader BinaryReader) gin.HandlerFunc

WebSocketStreamHandler streams binary data chunks over WebSocket

Types

type BinaryReader

type BinaryReader interface {
	ReadChunk(chunkSize int) (*Chunk, error)
	Reset() error
}

BinaryReader provides binary reader

type Chunk

type Chunk struct {
	ContentType string
	Data        []byte
}

Chunk defines chunk data-structure

type ImageChunk

type ImageChunk struct {
	Data     []byte
	MIMEType string
	Name     string
	Width    int
	Height   int
	Format   string
}

ImageChunk defines structure for image chunks

type ImageReader

type ImageReader struct {
	// contains filtered or unexported fields
}

ImageReader defines image reader

func NewImageReader

func NewImageReader(dir string) (*ImageReader, error)

NewImageReader creates new image reader

func (*ImageReader) ReadChunk

func (r *ImageReader) ReadChunk(idx int) (*Chunk, error)

ReadChunk provides read chunk capabilities of image reader

func (*ImageReader) Reset

func (r *ImageReader) Reset() error

Reset provides reset of image reader

type NPYReader

type NPYReader struct {
	// contains filtered or unexported fields
}

NPYReader provides structure for numpy files

func NewNPYReader

func NewNPYReader(dir string) (*NPYReader, error)

NewNPYReader provides numpy reader

func (*NPYReader) ReadChunk

func (r *NPYReader) ReadChunk(_ int) (*Chunk, error)

ReadChunk provides ability to read specific chunk of numpy data file

func (*NPYReader) Reset

func (r *NPYReader) Reset() error

Reset resets numpy reader index