blm

README

Go BLM Processor

A simple Rightmove v3 BLM file processor written in Go.

This library streams each data row until EOF, allowing you to process rows incrementally (for example, validating or transforming data) before batch-inserting into a database.
It is designed to be memory-efficient and suitable for very large BLM files.

---

Features

• Streams BLM #DATA# rows one at a time
• Does not load the entire file into memory
• Header-based value lookup
• Graceful EOF handling
• Suitable for batch inserts and pipelines

---

Installation

go get gitlab.com/matchouse-oss/go-blm-processor

Usage

package main

import (
	"fmt"
	"io"
	"log"

	"gitlab.com/matchouse-oss/go-blm-processor"
)

func main() {
	br, err := blm.NewReader("./sample.blm")
    if err != nil {
        panic(err)
    }
    defer br.Close()

	for {
		data, err := br.Next()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Fatal(err)
		}

		// Raw data row ([]string)
		fmt.Println(data)

		// Get a value by header name
		isPublished, ok := br.GetValueByHeader(data, "PUBLISHED_FLAG")
		if !ok {
			fmt.Println("PUBLISHED_FLAG does not exist")
		} else {
			fmt.Println(isPublished)
		}

		// Get multiple values by header names
		// nil  -> key does not exist
		// ""   -> key exists but value is empty
		selectedCols := br.GetValues(
			data,
			[]string{"PUBLISHED_FLAG", "STATUS_ID", "INVALID_KEY"},
		)

		fmt.Println(selectedCols)
	}
}

When to Use This

• Processing large Rightmove BLM feeds
• Streaming ingestion pipelines
• Batch inserts into PostgreSQL / ClickHouse
• Avoiding full-file memory loads

Documentation

Index

• Variables
• type BLMReader
  • func NewFileReader(path string) (*BLMReader, error)
  • func NewStreamReader(s io.ReadCloser) (*BLMReader, error)
  • func (br *BLMReader) Close() error
  • func (br *BLMReader) DataHeaders() []string
  • func (br *BLMReader) Err() error
  • func (br *BLMReader) FileHeader(key string) string
  • func (br *BLMReader) HeadersByPattern(pattern string) ([]string, error)
  • func (br *BLMReader) Next() error
  • func (br *BLMReader) Row() []string
  • func (br *BLMReader) SetRow(data []string)
  • func (br *BLMReader) TotalRows() int64
  • func (br *BLMReader) Value(key string) string
  • func (br *BLMReader) Values(keys []string) []string

Variables

var (
	InvalidBLM        = errors.New("invalid BLM file")
	InvalidHeader     = errors.New("invalid BLM headers")
	InvalidDefinition = errors.New("invalid BLM definitions")
	InvalidDataRow    = errors.New("invalid data row. either some field missing or EOR appeared in middle")
)

Types

type BLMReader

type BLMReader struct {
	// contains filtered or unexported fields
}

func NewFileReader

func NewFileReader(path string) (*BLMReader, error)

func NewStreamReader

func NewStreamReader(s io.ReadCloser) (*BLMReader, error)

func (*BLMReader) Close

func (br *BLMReader) Close() error

func (*BLMReader) DataHeaders

func (br *BLMReader) DataHeaders() []string

DataHeaders returns the slice of column headers for the data section.

func (*BLMReader) Err

func (br *BLMReader) Err() error

Err returns error if blm file is invalid. call it after Next() loop

func (*BLMReader) FileHeader

func (br *BLMReader) FileHeader(key string) string

FileHeader returns the value associated with the specified key from the file-level header map (e.g., 'VERSION', 'DATE').

func (*BLMReader) HeadersByPattern

func (br *BLMReader) HeadersByPattern(pattern string) ([]string, error)

HeadersByPattern returns a slice of data headers that match the given regex pattern.

func (*BLMReader) Next

func (br *BLMReader) Next() error

Next returns the next data row as a []string. It returns io.EOF when no more rows are available. It returns InvalidDataRow if the row length does not match the header length.

func (*BLMReader) Row

func (br *BLMReader) Row() []string

func (*BLMReader) SetRow

func (br *BLMReader) SetRow(data []string)

SetRow manually injects data into the reader's current row buffer.

This is primarily intended for testing or in-place data manipulation before or after calling processing functions.

Once SetRow is called, the reader's state is updated, allowing immediate use of accessor methods like Row(), Value(), and Values().

IMPORTANT: Do NOT call this function inside the Next() loop, as it will overwrite the data fetched from the underlying stream.

func (*BLMReader) TotalRows

func (br *BLMReader) TotalRows() int64

func (*BLMReader) Value

func (br *BLMReader) Value(key string) string

Value returns the value for the given header key from the data row.

func (*BLMReader) Values

func (br *BLMReader) Values(keys []string) []string

Values returns values for the given header keys from the latest row. Each element in the returned slice corresponds to the key at the same index.

Example:

[]any{"value_1", ""}