csminify

README

cs-demo-minifier

This tool and library aims to provide a way of converting CS:GO demos into a more easily digestible format while decreasing the data size (up to 99.7%) and retaining all important information. It is based on the demo parser demoinfocs-golang.

The project is still under development and the data formats may change in backwards-incompatible ways without notice.

Discussions / Chat

Use gitter to ask questions and discuss ideas about this project.
There are also some other rooms available around the topic of CS:GO demos.

Download

Binaries

See the releases page for pre-compiled executables for Linux, MacOS & Windows.

Go Get

# Library
go get -u github.com/markus-wa/cs-demo-minifier

# Command line tool
go get -u github.com/markus-wa/cs-demo-minifier/cmd/csminify

Usage

Command Line

The following command takes one snapshot of a demo every two seconds (-freq 0.5) and saves the resulting replay in the MessagePack format to demo.mp.

csminify -demo /path/to/demo.dem -format msgpack -freq 0.5 -out demo.mp

Options

$ csminify -help
Usage of csminify:
  -demo path
        Demo file path (default stdin)
  -format string
        Format into which the demo should me minified [json, msgpack, protobuf] (default "json")
  -freq float
        Snapshot frequency - per second (default 0.5)
  -out path
        Output file path (default stdout)

May exit with code 3 if a demo ends unexpectedly, but the minified data may still be usable if this happens

Direct bug reports and feature requests to https://github.com/markus-wa/cs-demo-minifier

Supported Formats

Format	Command Line (-format) Flag	Structure	Minimal Example	Full Example
JSON	json	schema.json	minimal.json	see releases page
MessagePack	msgpack, mp	schema.json	minimal.mp	see releases page
Protocol Buffers	protobuf, proto, pb	replay.proto	minimal.pb	see releases page

The minimal examples contain an extract of a demo with each event being included at least once. Events and attributes are also are documented in events.md.

More formats can be added programmatically by implementing the ReplayMarshaller interface.

If you would like to see additional formats supported please open a feature request (issue) or a pull request if you already have an implementation ready.

Unix pipes and jq

As the CLI supports Unix pipes, you can combine it with other tools such as jq.

In this section you can find a few examples.

Retreiving the map name

$ csminfy < demo.dem | jq -r '.header.map'
de_cache

Selecting the first three kills

$ csminify < test/cs-demos/default.dem | jq -r '[ .ticks[] as $parent |
  $parent.events[] | select(.name=="kill") as $kill |
  $kill.attrs[] | select(.key=="victim") as $victim |
  $kill.attrs[] | select(.key=="killer") as $killer |
  $kill.attrs[] | select(.key=="weapon") as $weapon |
  {
    tick: $parent.nr,
    kill: { victim: $victim.numVal, killer: $killer.numVal, weapon: $weapon.numVal }
  }] | .[0:3]'

[
  {
    "tick": 43,
    "kill": {
      "victim": 9,
      "killer": 2,
      "weapon": 303
    }
  },
  {
    "tick": 1029,
    "kill": {
      "victim": 7,
      "killer": 4,
      "weapon": 9
    }
  },
  {
    "tick": 1057,
    "kill": {
      "victim": 11,
      "killer": 4,
      "weapon": 9
    }
  }
]

Compressing the converted demo

$ du -sk demo.dem
67696   demo.dem
# original demo is 67 MB

$ csminfy < demo.dem | gzip > demo.json.gz

$ du -sk demo.json.gz
160 demo.json.gz
# gzipped JSON is 160 KB
# -> reduced size by ~99.7%

Library

This is an example on how to minify a demo to JSON and decode it to a replay.Replay again.

package main

import (
	"bytes"
	"encoding/json"
	"log"
	"os"

	csminify "github.com/markus-wa/cs-demo-minifier"
	rep "github.com/markus-wa/cs-demo-minifier/replay"
)

func main() {
	// Open the demo file
	f, err := os.Open("/path/to/demo.dem")
	defer f.Close()
	if err != nil {
		log.Fatal(err)
	}

	// Minify the replay to a byte buffer - or any other io.Writer (JSON)
	// Take 0.5 snapshots per second (=one every two seconds)
	freq := 0.5
	buf := new(bytes.Buffer)
	err = csminify.MinifyTo(f, freq, marshalJSON, buf)
	if err != nil {
		log.Fatal(err)
	}

	// Decoding it again is just as easy
	var r rep.Replay
	err = json.NewDecoder(buf).Decode(&r)
	if err != nil {
		log.Fatal(err)
	}
}

// JSON marshaller
func marshalJSON(r rep.Replay, w io.Writer) error {
	return json.NewEncoder(w).Encode(r)
}

MessagePack marshalling works pretty much the same way as JSON.
For Protobuf use protobuf.Unmarshal() (in the sub-package).

Development

Tests

Running tests

To run tests Git LFS is required.

git submodule init
git submodule update
pushd test/cs-demos && git lfs pull -I '*' && popd
go test ./...

Updating .golden files

There are golden files that are used to make sure no unintended changes are introduced. This will cause tests to fail when adding new data to the output.

To update these files when you inteded to make such a change (such as adding new events etc.) you will need to run the following command

go test -updateGolden

Generating Protobuf Code

Should you need to re-generate the protobuf generated code in the protobuf package, you will need the following tools:

• The latest protobuf generator (protoc) from your package manager or https://github.com/google/protobuf/releases

• And protoc-gen-gogofaster from gogoprotobuf to generate code for go.

    go get -u github.com/gogo/protobuf/protoc-gen-gogofaster
  

Make sure both are inside your PATH variable.

After installing these use go generate ./protobuf to generate the protobuf code.

License

This project is licensed under the MIT license.

Documentation

Overview

Package csminify provides functions for parsing CS:GO demos and minifying them into various formats.

Index

• Variables
• func Minify(r io.Reader, snapFreq float64, marshal ReplayMarshaller) ([]byte, error)
• func MinifyTo(r io.Reader, snapFreq float64, marshal ReplayMarshaller, w io.Writer) error
• func MinifyToWithConfig(r io.Reader, cfg ReplayConfig, marshal ReplayMarshaller, w io.Writer) error
• func MinifyWithConfig(r io.Reader, cfg ReplayConfig, marshal ReplayMarshaller) ([]byte, error)
• func ToReplay(r io.Reader, snapFreq float64) (rep.Replay, error)
• func ToReplayWithConfig(r io.Reader, cfg ReplayConfig) (rep.Replay, error)
• type EventCollector
  • func (ec *EventCollector) AddEvent(event rep.Event)
  • func (ec *EventCollector) AddHandler(handler interface{})
  • func (ec *EventCollector) Parser() dem.Parser
• type ExtraEventHandlers
  • func (ExtraEventHandlers) RegisterAll(ec *EventCollector)
  • func (ExtraEventHandlers) RegisterFootstep(ec *EventCollector)
• type ReplayConfig
• type ReplayMarshaller

Variables

var DefaultReplayConfig = func(snapFreq float64) ReplayConfig {
	ec := new(EventCollector)
	EventHandlers.Default.RegisterAll(ec)
	return ReplayConfig{
		SnapshotFrequency: snapFreq,
		EventCollector:    ec,
	}
}

DefaultReplayConfig returns the default configuration with a given snapshot frequency. May be overridden.

var EventHandlers eventHandlers

EventHandlers provides functions for registering the out-of-the-box provided handlers on EventCollectors. The handlers are divided into two groups: Default and Extra. Default contains the handlers that are used if no custom EventCollector is specified. Extra contains other handlers that are usually not required (i.e. a 'footstep' handler).

Functions

func Minify

func Minify(r io.Reader, snapFreq float64, marshal ReplayMarshaller) ([]byte, error)

Minify wraps MinifyTo with a bytes.Buffer and returns the written bytes.

func MinifyTo

func MinifyTo(r io.Reader, snapFreq float64, marshal ReplayMarshaller, w io.Writer) error

MinifyTo reads a demo from r, creates a replay and marshals it to w. See also: ToReplay

func MinifyToWithConfig

func MinifyToWithConfig(r io.Reader, cfg ReplayConfig, marshal ReplayMarshaller, w io.Writer) error

MinifyToWithConfig reads a demo from r, creates a replay and marshals it to w. See also: ToReplayWithConfig

func MinifyWithConfig

func MinifyWithConfig(r io.Reader, cfg ReplayConfig, marshal ReplayMarshaller) ([]byte, error)

MinifyWithConfig wraps MinifyToWithConfig with a bytes.Buffer and returns the written bytes.

func ToReplay

func ToReplay(r io.Reader, snapFreq float64) (rep.Replay, error)

ToReplay reads a demo from r, takes snapshots (snapFreq/sec) and records events into a Replay.

func ToReplayWithConfig

func ToReplayWithConfig(r io.Reader, cfg ReplayConfig) (rep.Replay, error)

ToReplayWithConfig reads a demo from r, takes snapshots and records events into a Replay with a custom configuration.

Types

type EventCollector

type EventCollector struct {
	// contains filtered or unexported fields
}

EventCollector provides the possibility of adding custom events to replays. First all demo-event handlers must be registered via AddHandler(). The registered handlers can add replay-events to the collector via AddEvent(). The handlers can access game-state information via Parser(). After a tick ends all events that were added to the collector during the tick will be stored into the replay.

func (*EventCollector) AddEvent

func (ec *EventCollector) AddEvent(event rep.Event)

AddEvent adds an event to the collector. The event will be added to the replay after the current tick ends.

func (*EventCollector) AddHandler

func (ec *EventCollector) AddHandler(handler interface{})

AddHandler adds a handler which will be registered on the Parser to the collector. The handler should use EventCollector.AddEvent() and be of the type func(<EventType>) where EventType is the type of the event to be handled. The handler parameter is of type interface because lolnogenerics. See: github.com/markus-wa/demoinfocs-golang/v2/pkg/demoinfocs.Parser.RegisterEventHandler() GoDoc: https://pkg.go.dev/github.com/markus-wa/demoinfocs-golang/v2/pkg/demoinfocs?tab=doc#Parser

func (*EventCollector) Parser

func (ec *EventCollector) Parser() dem.Parser

Parser returns the demo-parser through which custom handlers can access game-state information. Returns nil before minification has started - so don't call this before you need it.

type ExtraEventHandlers

type ExtraEventHandlers struct{}

func (ExtraEventHandlers) RegisterAll

func (ExtraEventHandlers) RegisterAll(ec *EventCollector)

func (ExtraEventHandlers) RegisterFootstep

func (ExtraEventHandlers) RegisterFootstep(ec *EventCollector)

type ReplayConfig

type ReplayConfig struct {
	SnapshotFrequency float64
	EventCollector    *EventCollector
}

ReplayConfig contains the configuration for generating a replay.

type ReplayMarshaller

type ReplayMarshaller func(rep.Replay, io.Writer) error

ReplayMarshaller is the signature for functions that serialize replay.Replay structs to an io.Writer