railreader

package module
v0.0.0-...-b365dc5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 7, 2025 License: AGPL-3.0 Imports: 2 Imported by: 0

README

Rail Reader logo banner

railreader

Self-hostable middleman between UK rail datafeeds and your project.

RailReader consumes realtime and static data from multiple railway datasources, and stores accumulated data in a database. This data can then be queried or subscribed to via multiple outputs.

The aim of RailReader is to provide useful and modern APIs for handling data from the UK's railway network by taking existing data feeds and transforming them into formats that are easier to work with for developers who want to save time.

[!WARNING] This project is very much in alpha. Data produced may not be fully accurate yet!

The database schema will continue to change without migrations until a first version is released. Be prepared to drop your database when updating.

Inputs

  • (work-in-progress) Darwin Real Time Train Information XML Push Port (Rail Delivery Group)
  • Darwin Timetable Files (Rail Delivery Group)

Outputs

See it in action

https://www.youtube.com/watch?v=FNzfIL_Gy-U

Resources used

Input name Schemas Documentation
Darwin XML Schema Definition P75301004 Issue 24, CIF specification version 29
Software to display XML schemas

For dealing with very large XML schemas with lots of types split accross multiple files, I've found Altova XMLSpy to do an extremly good job of exploring the whole schema visually. It is paid software, and is Windows only, but a 30-day free trial is availiable without payment details and if you plan to work with any of the XML schemas, it's almost certainly worth the effort to set up.

Setup instructions

PostgreSQL

RailReader requires a minimum of PostgreSQL version 15.

Docker Compose

The included Docker Compose file will run a PostgreSQL server on port 5432 with the username postgres and the password change_me.

sudo docker compose up

You must then create a database on the server yourself using the PostgreSQL CLI:

sudo docker run -it --rm --network host postgres psql -h localhost -U postgres

CREATE DATABASE railreader;

Use this database URL for Docker: postgres://postgres:change_me@localhost:5432/railreader?sslmode=disable.

NixOS

There is a provided NixOS module to configure and host both PostgreSQL and RailReader, availiable from this repository's flake at under nixosModules.railreader. For configuration options, read service.nix.

Your own

Any PostgreSQL database that is at least version 15 will work.

Running RailReader

Use Nix to run RailReader anywhere:

nix run github:headblockhead/railreader -- ingest --help

Or, if developing locally, use Go:

go run ./cmd/railreader ingest --help

How to get data in:

How to get data out:

  • PostgreSQL

    Write SQL queries to select from the database.

    Example queries

    • Destinations of trains originating from Leeds:

      SELECT name FROM locations l JOIN
    (
        SELECT location_id
            FROM (
                SELECT s.schedule_id FROM schedules s
                JOIN schedule_locations sl ON s.schedule_id = sl.schedule_id
                WHERE
                    sl.location_id = 'LEEDS'
                    AND sl.type = 'OR' -- Originates from Leeds
                    AND sl.working_departure_time >= to_char(now(), 'HH24:MI') -- Departs after now
                    AND s.scheduled_start_date >= now()::date
                ORDER BY sl.working_departure_time -- List in order of departure
            ) as ssli
        JOIN schedule_locations sl ON ssli.schedule_id = sl.schedule_id
            WHERE
                sl.type = 'DT' -- Get the destination location
    ) as ssli2
ON ssli2.location_id = l.location_id; -- Use display names over location_ids

AI declaration

Github Copilot was used to autocomplete predictable or repetative lines of code. All logic and documentation (including this file) was written without the assistance of AI in order to ensure accuracy.

Tasks

code-gen

Directory: egesters/grpc/proto

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative *.proto

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ActivityCodeStrings = map[ActivityCode]string{
	ActivityTrainBegins:                                   "begins",
	ActivityTrainFinishes:                                 "finishes",
	ActivityStopsToTakeUpAndSetDownPassengers:             "stops",
	ActivityStopsToTakeUpPassengers:                       "takes up passengers",
	ActivityStopsToSetDownPassengers:                      "sets down passengers",
	ActivityStopsOrShuntsForOtherTrainsToPass:             "stops/shunts for other trains to pass",
	ActivityStopsToAttachOrDetachAssistingLocomotive:      "stops to attach/detach an assisting locomotive",
	ActivityStopsForBankingLocomotive:                     "stops for a banking locomotive",
	ActivityStopsToChangeTrainCrew:                        "stops to change train crew",
	ActivityStopsToDetatchVehicles:                        "stops to detach vehicles",
	ActivityStopsForExamination:                           "stops for examination",
	ActivityPassengerCountPoint:                           "passengers counted",
	ActivityTicketCollectionAndExaminationPoint:           "tickets collected and examined",
	ActivityTicketExaminationPoint:                        "tickets examined",
	ActivityTicketExaminationPointFirstClassOnly:          "tickets for first class examined",
	ActivityTicketExaminationPointSelective:               "tickets possibly examined",
	ActivityStopsToChangeLocomotives:                      "stops to change locomotives",
	ActivityStopNotAdvertised:                             "stops (unadvertised)",
	ActivityStopsForOtherOperatingReasons:                 "stops for operating reasons",
	ActivityPropellingBetweenPointsShown:                  "propells between points shown",
	ActivityStopsWhenRequired:                             "stops when required",
	ActivityStopsForReversingMovementOrDriverEndChange:    "stops to reverse",
	ActivityStopsForLocomotiveToRunRound:                  "stops for locomotive to run round",
	ActivityStopsForRailwayPersonellOnly:                  "stops (railway personnel only)",
	ActivityStopsToAttachAndDetachVehicles:                "stops to attach/detach vehicles",
	ActivityStopsOrPassesForTabletStaffOrToken:            "stops/passes for tablet staff/token",
	ActivityStopsToAttachVehicles:                         "stops to attach vehicles",
	ActivityStopsForWateringOfCoaches:                     "stops for watering of coaches",
	ActivityPassesAnotherTrainAtCrossingPointOnSingleLine: "passes another train at crossing point on a single line",
}
View Source 
var ServiceCategoryStrings = map[ServiceCategory]string{

	CategoryUndergroundOrMetro:    "Underground/Metro",
	CategoryUnadvertisedPassenger: "(unadvertised) Passenger",
	CategoryPassenger:             "Passenger",
	CategoryStaff:                 "Staff",
	CategoryMixed:                 "Mixed",

	CategoryChannelTunnel:       "Channel Tunnel",
	CategorySleeper:             "Sleeper",
	CategoryInternational:       "International",
	CategoryMotorail:            "Motorail",
	CategoryUnadvertisedExpress: "(unadvertised) Express",
	CategoryExpress:             "Express",
	CategorySleeperDomestic:     "Sleeper",

	CategoryBusReplacement: "Rail Replacement",
	CategoryBusService:     "",

	CategoryShip: "Ship",

	CategoryEmptyCoachingStock:                   "Empty",
	CategoryEmptyCoachingStockUndergroundOrMetro: "Empty Underground/Metro",
	CategoryEmptyCoachingStockOrStaff:            "Empty/Staff",

	CategoryPostal:                                 "Postal",
	CategoryPostOfficeControlledParcels:            "Post Office Controlled Parcel",
	CategoryParcels:                                "Parcel",
	CategoryEmptyNonPassengerCarryingCoachingStock: "Empty Non-Passenger",

	CategoryDepartmental:                          "Departmental",
	CategoryCivilEngineering:                      "Civil Engineering",
	CategoryMechanicalOrElectricalEngineering:     "Mechanical/Electrical Engineering",
	CategoryDepartmentalStores:                    "Departmental Store",
	CategoryTest:                                  "Test",
	CategorySignalOrTelecommunicationsEngineering: "Signal/Telecommunications Engineering",

	CategoryLocomotiveOrBrakeVan: "Brake Van/Locomotive",
	CategoryLightLocomotive:      "Light Locomotive",

	CategoryRfDAutomotiveComponents:  "Automotive Components Freight",
	CategoryRfDAutomotives:           "Automotive Freight",
	CategoryRfDEdibleProducts:        "Edible Product Freight",
	CategoryRfDIndustrialMinerals:    "Industrial Mineral Freight",
	CategoryRfDChemicals:             "Chemical Freight",
	CategoryRfDBuildingMaterials:     "Building Material Freight",
	CategoryRfDGeneralMerchandise:    "General Merchandise Freight",
	CategoryRfDEuropean:              "European Freight",
	CategoryRfDFreightlinerContracts: "Contract Freightliner",
	CategoryRfDFreightlinerOther:     "Other Freightliner",

	CategoryTLFCoalDistribution:          "Coal Distribution Trainload Freight",
	CategoryTLFCoalForEletricity:         "Coal (for Electricity) Trainload Freight",
	CategoryTLFCoalOrNuclear:             "Coal/Nuclear Trainload Freight",
	CategoryTLFMetals:                    "Metal Trainload Freight",
	CategoryTLFAggregates:                "Aggregate Trainload Freight",
	CategoryTLFDomesticOrIndustrialWaste: "Domestic/Industrial Waste Trainload Freight",
	CategoryTLFBuildingMaterials:         "Building Material Trainload Freight",
	CategoryTLFPetroleumProducts:         "Petroleum Product Trainload Freight",

	CategoryRfDChannelMixedBusiness:    "Mixed Business Channel Tunnel Freight",
	CategoryRfDChannelIntermodal:       "Intermodal Channel Tunnel Freight",
	CategoryRfDChannelAutomotive:       "Automotive Channel Tunnel Freight",
	CategoryRfDChannelContractServices: "Contract Services Channel Tunnel Freight",
	CategoryRfDChannelHaulmark:         "Haulmark Channel Tunnel Freight",
	CategoryRfDChannelJointVenture:     "Joint Venture Channel Tunnel Freight",
}
View Source 
var ServiceTypeStrings = map[ServiceType]string{
	ServicePassengerOrParcelTrain:     "Train",
	ServiceBus:                        "Bus",
	ServiceShip:                       "Ship",
	ServiceTrip:                       "Trip",
	ServiceFreight:                    "Freight",
	ServicePassengerOrParcelShortTerm: "Train",
	ServiceBusShortTerm:               "Bus",
	ServiceShipShortTerm:              "Ship",
	ServiceTripShortTerm:              "Trip",
	ServiceFreightShortTerm:           "Freight",
}

Functions

This section is empty.

Types

type ActivityCode 

type ActivityCode string
const (
	ActivityNone ActivityCode = "  "

	ActivityTrainBegins   ActivityCode = "TB"
	ActivityTrainFinishes ActivityCode = "TF"

	ActivityStopsToTakeUpAndSetDownPassengers ActivityCode = "T "
	ActivityStopsToTakeUpPassengers           ActivityCode = "U "
	ActivityStopsToSetDownPassengers          ActivityCode = "D "

	ActivityStopsOrShuntsForOtherTrainsToPass        ActivityCode = "A "
	ActivityStopsToAttachOrDetachAssistingLocomotive ActivityCode = "AE"
	//ActivityShowsAsXOnArrival                                   ActivityCode = "AX"
	ActivityStopsForBankingLocomotive ActivityCode = "BL"
	ActivityStopsToChangeTrainCrew    ActivityCode = "C "
	ActivityStopsToDetatchVehicles    ActivityCode = "-D"
	ActivityStopsForExamination       ActivityCode = "E "
	//ActivityNationalRailTimetableDataToAdd                      ActivityCode = "G "
	//ActivityNotionalActivityToPreventWTTTimingColumnsMerge      ActivityCode = "H "
	//ActivityNotionalActivityToPreventWTTTimingColumnsMergeTwice ActivityCode = "HH"
	ActivityPassengerCountPoint                  ActivityCode = "K "
	ActivityTicketCollectionAndExaminationPoint  ActivityCode = "KC"
	ActivityTicketExaminationPoint               ActivityCode = "KE"
	ActivityTicketExaminationPointFirstClassOnly ActivityCode = "KF"
	ActivityTicketExaminationPointSelective      ActivityCode = "KS"
	ActivityStopsToChangeLocomotives             ActivityCode = "L "
	ActivityStopNotAdvertised                    ActivityCode = "N "
	ActivityStopsForOtherOperatingReasons        ActivityCode = "OP"
	//ActivityTrainLocomotiveOnRear                         ActivityCode = "OR"
	ActivityPropellingBetweenPointsShown               ActivityCode = "PR"
	ActivityStopsWhenRequired                          ActivityCode = "R "
	ActivityStopsForReversingMovementOrDriverEndChange ActivityCode = "RM"
	ActivityStopsForLocomotiveToRunRound               ActivityCode = "RR"
	ActivityStopsForRailwayPersonellOnly               ActivityCode = "S "
	ActivityStopsToAttachAndDetachVehicles             ActivityCode = "-T"
	//ActivityDetailConsistForTOPSDirect                    ActivityCode = "TS"
	ActivityStopsOrPassesForTabletStaffOrToken            ActivityCode = "TW"
	ActivityStopsToAttachVehicles                         ActivityCode = "-U"
	ActivityStopsForWateringOfCoaches                     ActivityCode = "W "
	ActivityPassesAnotherTrainAtCrossingPointOnSingleLine ActivityCode = "X "
)

Some of the meaning of these codes are unknown/unclear, and are commented out.

func (ActivityCode) String 

func (ac ActivityCode) String() string

type AssociationCategory 

type AssociationCategory string
const (
	// AssociationJoin indicates that two services join together into a single train.
	AssociationJoin AssociationCategory = "JJ"
	// AssociationDivide indicates a single train divides into two services, and one of them terminates.
	AssociationDivide AssociationCategory = "VV"
	// AssociationLink indicates two services are linked together into a single service.
	// For example, a train that terminates halfway through its schedule, and a bus replacement that continues the service.
	// Services are not necessarily linked at their termini, and links may create branching paths.
	// This is different from a join/divide as links may involve a change of service type.
	// In Darwin, passengers transfer from the MainService to the AssociatedService.
	AssociationLink AssociationCategory = "LK"
	// AssociationNext indicates the next service to be run using the same rolling stock.
	AssociationNext AssociationCategory = "NP"
)

type Egester 

type Egester interface {
	Serve(net.Listener) error
	Close() error
}

type Ingester 

type Ingester[T any] interface {
	Fetch(context.Context) (T, error)
	ProcessAndCommit(T) error
	Close() error
}

type ServiceCategory 

type ServiceCategory string

ServiceCategory describes the function of a service. The String() form can be prepended to the ServiceType when describing a service to give additional information about it. 

const (
	// O - Ordinary
	CategoryUndergroundOrMetro    ServiceCategory = "OL"
	CategoryUnadvertisedPassenger ServiceCategory = "OU"
	CategoryPassenger             ServiceCategory = "OO"
	CategoryStaff                 ServiceCategory = "OS"
	CategoryMixed                 ServiceCategory = "OW"
	// X - Express
	CategoryChannelTunnel       ServiceCategory = "XC"
	CategorySleeper             ServiceCategory = "XD"
	CategoryInternational       ServiceCategory = "XI"
	CategoryMotorail            ServiceCategory = "XR"
	CategoryUnadvertisedExpress ServiceCategory = "XU"
	CategoryExpress             ServiceCategory = "XX"
	CategorySleeperDomestic     ServiceCategory = "XZ"
	// B - Bus
	CategoryBusReplacement ServiceCategory = "BR"
	CategoryBusService     ServiceCategory = "BS"
	// S - Ship
	CategoryShip ServiceCategory = "SS"
	// E - Empty Coaching Stock
	CategoryEmptyCoachingStock                   ServiceCategory = "EE"
	CategoryEmptyCoachingStockUndergroundOrMetro ServiceCategory = "EL"
	CategoryEmptyCoachingStockOrStaff            ServiceCategory = "ES"
	// Parcels and Postal trains
	CategoryPostal                                 ServiceCategory = "JJ"
	CategoryPostOfficeControlledParcels            ServiceCategory = "PM"
	CategoryParcels                                ServiceCategory = "PP"
	CategoryEmptyNonPassengerCarryingCoachingStock ServiceCategory = "PV"
	// D - Departmental trains
	CategoryDepartmental                          ServiceCategory = "DD"
	CategoryCivilEngineering                      ServiceCategory = "DH"
	CategoryMechanicalOrElectricalEngineering     ServiceCategory = "DI"
	CategoryDepartmentalStores                    ServiceCategory = "DQ"
	CategoryTest                                  ServiceCategory = "DT"
	CategorySignalOrTelecommunicationsEngineering ServiceCategory = "DY"
	// Z - Light locomotives
	CategoryLocomotiveOrBrakeVan ServiceCategory = "ZB"
	CategoryLightLocomotive      ServiceCategory = "ZZ"
	// Railfreight distribution
	CategoryRfDAutomotiveComponents  ServiceCategory = "J2"
	CategoryRfDAutomotives           ServiceCategory = "H2"
	CategoryRfDEdibleProducts        ServiceCategory = "J3"
	CategoryRfDIndustrialMinerals    ServiceCategory = "J4"
	CategoryRfDChemicals             ServiceCategory = "J5"
	CategoryRfDBuildingMaterials     ServiceCategory = "J6"
	CategoryRfDGeneralMerchandise    ServiceCategory = "J8"
	CategoryRfDEuropean              ServiceCategory = "H8"
	CategoryRfDFreightlinerContracts ServiceCategory = "J9"
	CategoryRfDFreightlinerOther     ServiceCategory = "H9"
	// Trainload Freight
	CategoryTLFCoalDistribution          ServiceCategory = "A0"
	CategoryTLFCoalForEletricity         ServiceCategory = "E0"
	CategoryTLFCoalOrNuclear             ServiceCategory = "B0"
	CategoryTLFMetals                    ServiceCategory = "B1"
	CategoryTLFAggregates                ServiceCategory = "B4"
	CategoryTLFDomesticOrIndustrialWaste ServiceCategory = "B5"
	CategoryTLFBuildingMaterials         ServiceCategory = "B6"
	CategoryTLFPetroleumProducts         ServiceCategory = "B7"
	// Railfreight distribution through the Channel Tunnel
	CategoryRfDChannelMixedBusiness    ServiceCategory = "H0"
	CategoryRfDChannelIntermodal       ServiceCategory = "H1"
	CategoryRfDChannelAutomotive       ServiceCategory = "H3"
	CategoryRfDChannelContractServices ServiceCategory = "H4"
	CategoryRfDChannelHaulmark         ServiceCategory = "H5"
	CategoryRfDChannelJointVenture     ServiceCategory = "H6"
)

func (ServiceCategory) String 

func (sc ServiceCategory) String() string

type ServiceType 

type ServiceType string

ServiceType represents what mode of transport a service is. 

const (
	ServicePassengerOrParcelTrain     ServiceType = "P"
	ServiceBus                        ServiceType = "B"
	ServiceShip                       ServiceType = "S"
	ServiceTrip                       ServiceType = "T"
	ServiceFreight                    ServiceType = "F"
	ServicePassengerOrParcelShortTerm ServiceType = "1"
	ServiceBusShortTerm               ServiceType = "5"
	ServiceShipShortTerm              ServiceType = "4"
	ServiceTripShortTerm              ServiceType = "3"
	ServiceFreightShortTerm           ServiceType = "2"
)

func (ServiceType) String 

func (tt ServiceType) String() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
railreader command
egesters
grpc
ingesters
darwin
darwin/inserter
darwin/unmarshaller

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.