binlog

package
v0.0.0-...-436d200 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 23, 2023 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Overview

The binlog package provides functionality for reading and parsing MySQL binary / relay log events.

This package is a work in progress. For now, it can handle a large subset of row based replication logs generated by MySQL 5.5 / 5.6 (It will probably work for query based replication logs as well).

Additional documentation can be found here:

Index

Constants

View Source
const (
	FDEFixedLengthDataSizeFor55 = 2 + 50 + 4 + 1 + 27
	FDEFixedLengthDataSizeFor56 = 2 + 50 + 4 + 1 + 35
)
View Source
const MaxDbsInEventMts = 254
View Source
const NullLength = uint64(^uint32(0))

Variables

View Source
var BigEndian = bigEndian{binary.BigEndian}

Mysql extensions to binary.BigEndian. This is mainly used for decoding MyISAM values (as defined in include/myisampack.h).

View Source
var LittleEndian = littleEndian{binary.LittleEndian}

Mysql extensions to binary.LittleEndian.

Functions

func CreateEventBytes

func CreateEventBytes(
	timestamp uint32,
	eventType uint8,
	serverId uint32,
	nextPosition uint32,
	flags uint16,
	data []byte) ([]byte, error)

This constructs a raw binlog event and returns its payload.

func IsRetryableError

func IsRetryableError(err error) bool

This returns true if the error returned by the event parser is retryable.

Types

type BaseRowsEvent

type BaseRowsEvent struct {
	Event
	// contains filtered or unexported fields
}

BaseRowsEvent is the representation common to all v1/v2 write/update/delete rows events

TODO(patrick): figure out what's inside the extra_row_info bytes (so far, my searches lead to dead ends; looks almost like it's not used.)

Common to both 5.5 and 5.6:
    19 bytes for common v4 event header
    6 bytes (uint64) for table id
    2 bytes (uint16) for flags
V2 rows events specific (5.6 only):
    2 bytes (uint16), X, for 2 + the length of variable-sized header
    X bytes for variable-sized header:
        (optional) extra info tag:
            1 byte for RW_V_EXTRAINFO_TAG (=0)
            1 byte (uint8), Y, for extra row info length
            Y bytes for extra row info data
Common to both 5.5 and 5.6:
    1 to 9 bytes (net_store_length variable encoded uint64), Z, for total
            number of columns (XXX: should be same as # of columns in table
            map event?).
    ceil(Z / 8) bytes for bitmap indicating which columns are used.  (NOTE:
            for update events, this bitmap is used for the before image).
V1/v2 update events specific:
    ceil(Z / 8) bytes for bitmap indicating which columns are used in the
            after image.
Common to both 5.5 and 5.6:
    The remaining body contains the row data (row values are decoded based
            on current table context):
        V1/v2 write/delete events specific:
            List of rows
        V1/v2 update events specific:
            List of pairs of (before image row, after image row)
        Each row image is compose of:
            bit field indicating whether each field in the row is NULL.
            list of non-NULL encoded values.
5.6 Specific:
    (optional) 4 bytes footer for checksum

func (*BaseRowsEvent) Context

func (e *BaseRowsEvent) Context() TableContext

Context returns the event's table context.

func (*BaseRowsEvent) ExtraRowInfoBytes

func (e *BaseRowsEvent) ExtraRowInfoBytes() []byte

ExtraRowInfoBytes returns the uninterpreted extra row info bytes. NOTE: When the event's encoding version is v1, this always returns nil.

func (*BaseRowsEvent) NumColumns

func (e *BaseRowsEvent) NumColumns() int

NumColumns returns the table's width.

func (*BaseRowsEvent) RowDataBytes

func (e *BaseRowsEvent) RowDataBytes() []byte

RowDataBytes returns the uninterpreted row data bytes. NOTE: This does not include the used columns bit map.

func (*BaseRowsEvent) RowsFlags

func (e *BaseRowsEvent) RowsFlags() uint16

RowsFlags returns the rows event's flags.

func (*BaseRowsEvent) TableId

func (e *BaseRowsEvent) TableId() uint64

TableId returns which table the event should act on.

func (*BaseRowsEvent) Version

Version returns the event's encoding version.

type ColumnDescriptor

type ColumnDescriptor interface {
	FieldDescriptor

	// IndexPosition returns the column's table index position.
	IndexPosition() int
}

func NewColumnDescriptor

func NewColumnDescriptor(fd FieldDescriptor, pos int) ColumnDescriptor

type DeleteRowsEvent

type DeleteRowsEvent struct {
	BaseRowsEvent
	// contains filtered or unexported fields
}

A representation of the v1 / v2 delete rows event.

func (*DeleteRowsEvent) DeletedRows

func (e *DeleteRowsEvent) DeletedRows() []RowValues

DeletedRows returns the rows removed from the table.

func (*DeleteRowsEvent) UsedColumns

func (e *DeleteRowsEvent) UsedColumns() []ColumnDescriptor

UsedColumns returns the column descriptors that are used by the event.

type DeleteRowsEventParser

type DeleteRowsEventParser struct {
	// contains filtered or unexported fields
}

func (*DeleteRowsEventParser) EventType

func (p *DeleteRowsEventParser) EventType() mysql_proto.LogEventType_Type

func (*DeleteRowsEventParser) FixedLengthDataSize

func (p *DeleteRowsEventParser) FixedLengthDataSize() int

func (*DeleteRowsEventParser) Parse

func (p *DeleteRowsEventParser) Parse(raw *RawV4Event) (Event, error)

func (*DeleteRowsEventParser) SetTableContext

func (p *DeleteRowsEventParser) SetTableContext(context TableContext)

type Event

type Event interface {
	// SourceName returns the name of the event's source stream.
	SourceName() string

	// SourcePosition returns the position relative to the beginning of the
	// source stream.  Unlike the next position stored in the event header,
	// this position is always correct.
	SourcePosition() int64

	// Timestamp returns the event's timestamp.
	Timestamp() uint32

	// EventType returns the event's type.
	EventType() mysql_proto.LogEventType_Type

	// ServerId returns the id of the server which generated this event.
	ServerId() uint32

	// EventLength returns the event's total (header + body + footer) length.
	EventLength() uint32

	// NextPosition returns the next position stored in the event entry.
	// NOTE: This value is independent of the entry real position within the
	// source stream.  Use SourcePosition() to get the correct absolute
	// position relative to the beginning of the source stream.
	NextPosition() uint32

	// Flags returns the event's flags.
	Flags() uint16

	// Bytes returns the event payload (header + data)
	Bytes() []byte

	// BasicHeader returns the fixed length portion of the header bytes.
	BasicHeader() []byte

	// ExtraHeaders returns the extra header bytes associated to the event.
	ExtraHeaders() []byte

	// FixedLengthData returns the fixed length data associated to the event.
	// NOTE: in mysql source code, the length of this data is referred to as
	// the event's header length (i.e., FORMAT_DESCRIPTION_HEADER_LEN)
	FixedLengthData() []byte

	// VariableLengthData returns the variable length data associated to the
	// event.
	VariableLengthData() []byte

	// Checksum returns the checksum bytes (which may be empty).  NOTE:
	// checksum is an optional field introduced in 5.6.  The checksum
	// algorithm used is defined in the format description event.
	Checksum() []byte
}

Event is the common interface for all mysql v4 binlog format events.

type EventReader

type EventReader interface {
	// NextEvent returns the next available event from the event stream.
	NextEvent() (Event, error)

	// Close closes the reader.  Subsequent calls to NextEvent will return
	// an error.
	Close() error
	// contains filtered or unexported methods
}

EventReader is the common reader interface for all mysql binary log event streams. NOTE: The EventReader interface purposely does not support jumping to a specific log position offset because skipping is very error prone. WaitForEvent is threadsafe; none of the other methods are threadsafe.

func NewLogFileV4EventReader

func NewLogFileV4EventReader(
	src io.Reader,
	srcName string,
	parsers V4EventParserMap,
	logger Logger) EventReader

This returns an EventReader which read events from a single (bin / relay) log file, with appropriate parser applied on each event. If no parser is available for the event, or if an error occurs during parsing, then the reader will return the original event along with the error. NOTE: this reader is responsible for checking the log file magic marker, the binlog format version and all format description events within the stream. It is also responsible for setting the checksum size for non-FDE events.

func NewLogStreamV4EventReader

func NewLogStreamV4EventReader(
	logDirectory string,
	logPrefix string,
	startingLogFileNum uint,
	isRelayLog bool,
	logger Logger) EventReader

This returns an EventReader which read and parses events from a (bin /relay) log stream composed of multiple log files. If no parser is available for the event, or if an error occurs during parsing, then the reader will return the original event along with the error. NOTE: this reader will transparently switch log files on rotate events (relay log wrapper events are not returned). When the reader fails to open a log file, it will return a *FailedToOpenFileError; it is safe to retry reading, assuming the filename is valid. When the reader encounters an invalid rotate event, it will return both the rotate event and an *InvalidRotationError.

func NewLogStreamV4EventReaderWithLogFileReaderCreator

func NewLogStreamV4EventReaderWithLogFileReaderCreator(
	logDirectory string,
	logPrefix string,
	startingLogFileNum uint,
	isRelayLog bool,
	logger Logger,
	newLogFileReader LogFileReaderCreator) EventReader

func NewParsedV4EventReader

func NewParsedV4EventReader(
	reader EventReader,
	parsers V4EventParserMap) EventReader

This returns an EventReader which applies the appropriate parser on each raw v4 event in the stream. If no parser is available for the event, or if an error occurs during parsing, then the reader will return the original event along with the error.

func NewRawV4EventReader

func NewRawV4EventReader(src io.Reader, srcName string) EventReader

This returns an EventReader which extracts entries from the src event stream and returns the entries as RawV4Event objects. The src stream can be a binary log event stream or a relay log event stream. NOTE: This reader assumes there is no binlog magic marker at the beginning of the stream. It also assumes the event entries are serialized using v4 binlog format. Finally, this reader does not set sizes for extra headers, fixed length data, and checksum (i.e. event.VariableLengthData() will return the entire event payload).

type FailedToOpenFileError

type FailedToOpenFileError struct {
	errors.DropboxError
	LogFileNum uint
}

When tailing logs on a mysql box, there's a potential race conditions where the rotate event is written, but a new log file is not created yet. It's probably safe to retry when this occurs (before quitting).

type FieldDescriptor

type FieldDescriptor interface {
	// Type returns the descriptor's field type.
	Type() mysql_proto.FieldType_Type

	// IsNullable returns whether or not the field is nullable.
	IsNullable() bool

	// ParseValue extracts a single mysql value from the data array.  The value
	// must an uint64 for int fields (NOTE that sign is uninterpreted), double
	// for floating point fields, []byte for string fields, and time.Time
	// (in UTC) for temporal fields.
	ParseValue(data []byte) (value interface{}, remaining []byte, err error)
}

FieldDescriptor defines the common interface for interpreting all mysql field types.

func NewBitFieldDescriptor

func NewBitFieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_BIT (i.e., Field_bit_as_char)

func NewBlobFieldDescriptor

func NewBlobFieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_BLOB (i.e., Field_blob)

func NewDateTime2FieldDescriptor

func NewDateTime2FieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_DATETIME2 (i.e., Field_datetimef). See TIME_from_longlong_datetime_packed ( in sql-common/my_time.c) for encoding detail.

func NewDateTimeFieldDescriptor

func NewDateTimeFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a fields descriptor for FieldType_DATETIME (i.e., Field_datetime). See number_to_datetime (in sql-common/my_time.c) for encoding detail.

func NewDecimalFieldDescriptor

func NewDecimalFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_DECIMAL (i.e., Field_decimal)

func NewDoubleFieldDescriptor

func NewDoubleFieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_DOUBLE (i.e., Field_double)

func NewFloatFieldDescriptor

func NewFloatFieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_FLOAT (i.e., Field_float)

func NewInt24FieldDescriptor

func NewInt24FieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_INT24 (i.e., Field_medium)

func NewLongFieldDescriptor

func NewLongFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_LONG (i.e., Field_long)

func NewLongLongFieldDescriptor

func NewLongLongFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_LONGLONG (i.e., Field_longlong)

func NewNewDecimalFieldDescriptor

func NewNewDecimalFieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_NEWDECIMAL (i.e., Field_newdecimal)

func NewNullFieldDescriptor

func NewNullFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_NULL (i.e., Field_null)

func NewShortFieldDescriptor

func NewShortFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_SHORT (i.e., Field_shart)

func NewStringFieldDescriptor

func NewStringFieldDescriptor(
	fieldType mysql_proto.FieldType_Type,
	nullable NullableColumn,
	maxLen int) FieldDescriptor

func NewTimestamp2FieldDescriptor

func NewTimestamp2FieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_TIMESTAMP2 (i.e., Field_timestampf). See my_timestamp_from_binary (in sql-common/my_time.c) for encoding detail.

func NewTimestampFieldDescriptor

func NewTimestampFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a fields descriptor for FieldType_TIMESTAMP (i.e., Field_timestamp)

func NewTinyFieldDescriptor

func NewTinyFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_TINY (i.e., Field_tiny).

func NewVarcharFieldDescriptor

func NewVarcharFieldDescriptor(nullable NullableColumn, metadata []byte) (
	fd FieldDescriptor,
	remaining []byte,
	err error)

This returns a field descriptor for FieldType_VARCHAR (i.e., Field_varstring)

func NewYearFieldDescriptor

func NewYearFieldDescriptor(nullable NullableColumn) FieldDescriptor

This returns a field descriptor for FieldType_YEAR (i.e., Field_year)

type FormatDescriptionEvent

type FormatDescriptionEvent struct {
	Event
	// contains filtered or unexported fields
}

A representation of the format description event.

FDE binlog payload is structured as follow:

Common to both 5.5 and 5.6:
    19 bytes for common v4 event headers
    2 bytes (uint16) for binlog version
    50 bytes for server version string (padded with '\0's)
    4 bytes (uint32) for created timestamp.  Note that this value may be
        unpopulated.
    1 byte (uint8) for total header size, where total header size =
        common header size + extra headers size
    1 byte per event type for event's fixed length data size.  Note that
        unknown events does not have an entry.
5.5 Specific:
    27 bytes for events' fixed size length (one uint8 entry per event
        type, except unknown events)
5.6 Specific:
    35 bytes for events' fixed size length (one uint8 entry per event
        type, except unknown events)
    1 byte (uint8) for checksum algorithm
    4 bytes for checksum

func (*FormatDescriptionEvent) BinlogVersion

func (e *FormatDescriptionEvent) BinlogVersion() uint16

BinlogVersion returns the binlog version (which should always be 4)

func (*FormatDescriptionEvent) ChecksumAlgorithm

ChecksumAlgorithm returns the algorithm used for checksumming non-FDE events

func (*FormatDescriptionEvent) CreatedTimestamp

func (e *FormatDescriptionEvent) CreatedTimestamp() uint32

CreatedTimestamp returns the fde's creation timestamp. NOTE: mysql log writer may leave the timestamp undefined.

func (*FormatDescriptionEvent) ExtraHeadersSize

func (e *FormatDescriptionEvent) ExtraHeadersSize() int

ExtraHeaderSize returns the extra header size for non-FDE events. For both mysql 5.5 and mysql 5.6, this should be 0.

func (*FormatDescriptionEvent) FixedLengthDataSizeForType

func (e *FormatDescriptionEvent) FixedLengthDataSizeForType(
	eventType mysql_proto.LogEventType_Type) int

FixedLengthDataSizeForType returns the size of fixed length data for each event type.

func (*FormatDescriptionEvent) NumKnownEventTypes

func (e *FormatDescriptionEvent) NumKnownEventTypes() int

NumKnownEventTypes returns the number of event types that is potentially in the stream.

func (*FormatDescriptionEvent) ServerVersion

func (e *FormatDescriptionEvent) ServerVersion() []byte

ServerVersion returns the server version from which the events were emitted.

type FormatDescriptionEventParser

type FormatDescriptionEventParser struct {
	// contains filtered or unexported fields
}

func (*FormatDescriptionEventParser) EventType

FormatDescriptionEventParser's EventType always returns mysql_proto.LogEventType_FORMAT_DESCRIPTION_EVENT.

func (*FormatDescriptionEventParser) FixedLengthDataSize

func (p *FormatDescriptionEventParser) FixedLengthDataSize() int

FormatDescriptionEventParser's FixedLengthDataSize always returns 0 (i.e., we pretend FDE does not have fixed length data). NOTE: In 5.6, the real "fixed" number is FDEFixedLengthDataSizeFor56. In 5.5, the real "fixed" number is FDEFixedLengthDataSizeFor55. The difference is due to increased number of event types.

func (*FormatDescriptionEventParser) Parse

FormatDecriptionEventParser's Parse processes a raw FDE event into a FormatDescriptionEvent.

func (*FormatDescriptionEventParser) SetTableContext

func (*FormatDescriptionEventParser) SetTableContext(context TableContext)

type GtidLogEvent

type GtidLogEvent struct {
	Event
	// contains filtered or unexported fields
}

func (*GtidLogEvent) Gno

func (e *GtidLogEvent) Gno() uint64

func (*GtidLogEvent) IsCommit

func (e *GtidLogEvent) IsCommit() bool

func (*GtidLogEvent) Sid

func (e *GtidLogEvent) Sid() []byte

type GtidLogEventParser

type GtidLogEventParser struct {
	// contains filtered or unexported fields
}

func (*GtidLogEventParser) EventType

GtidLogEventParser's EventType always returns mysql_proto.LogEventType_GTID_LOG_EVENT

func (*GtidLogEventParser) FixedLengthDataSize

func (p *GtidLogEventParser) FixedLengthDataSize() int

GtidLogEventParser's FixedLengthDataSize always returns 25.

func (*GtidLogEventParser) Parse

func (p *GtidLogEventParser) Parse(raw *RawV4Event) (Event, error)

GtidLogEventParser's Parse processes a raw gtid log event into a GtidLogEvent.

func (*GtidLogEventParser) SetTableContext

func (*GtidLogEventParser) SetTableContext(context TableContext)

type GtidRange

type GtidRange struct {
	Start, End uint64 // NOTE: End is EXCLUSIVE
}

type GtidSet

type GtidSet map[string][]GtidRange

type InvalidRotationError

type InvalidRotationError struct {
	errors.DropboxError
}

type LogFileReaderCreator

type LogFileReaderCreator func(
	dir string,
	file string,
	parsers V4EventParserMap) (
	EventReader,
	error)

type Logger

type Logger struct {
	Fatalf       func(pattern string, values ...interface{})
	Infof        func(pattern string, values ...interface{})
	VerboseInfof func(pattern string, values ...interface{})
}

type MockFileFetcher

type MockFileFetcher func() []*MockLogFile

Assumes that all files returned by successive calls to MockFileFetcher are the same, except perhaps the last MockLogFile may have additional content in a later call, and later calls may return more mock files. These assumptions should be reasonable for an append-only log model.

type MockLogFile

type MockLogFile struct {
	// contains filtered or unexported fields
}

MockLogFile is thread-safe.

func NewMockLogFile

func NewMockLogFile() *MockLogFile

func (*MockLogFile) Copy

func (mlf *MockLogFile) Copy() *MockLogFile

func (*MockLogFile) GetReader

func (mlf *MockLogFile) GetReader() *MockLogFileReader

func (*MockLogFile) Write

func (mlf *MockLogFile) Write(contents []byte)

Every function for writing into the MockLogFile should acquire the lock via either Write() or writeWithHeader().

func (*MockLogFile) WriteBegin

func (mlf *MockLogFile) WriteBegin()

func (*MockLogFile) WriteDelete

func (mlf *MockLogFile) WriteDelete(value int)

func (*MockLogFile) WriteDeleteWithParam

func (mlf *MockLogFile) WriteDeleteWithParam(value int, tableId int8)

func (*MockLogFile) WriteFDE

func (mlf *MockLogFile) WriteFDE()

func (*MockLogFile) WriteGtid

func (mlf *MockLogFile) WriteGtid(sid []byte, gno uint64)

func (*MockLogFile) WriteInsert

func (mlf *MockLogFile) WriteInsert(value int)

func (*MockLogFile) WriteInsertWithParam

func (mlf *MockLogFile) WriteInsertWithParam(value int, tableId int8)

func (*MockLogFile) WriteLogFileMagic

func (mlf *MockLogFile) WriteLogFileMagic()

func (*MockLogFile) WritePGLE

func (mlf *MockLogFile) WritePGLE(set GtidSet)

func (*MockLogFile) WriteQuery

func (mlf *MockLogFile) WriteQuery(query string)

func (*MockLogFile) WriteQueryWithParam

func (mlf *MockLogFile) WriteQueryWithParam(query string, dbName string)

func (*MockLogFile) WriteRotate

func (mlf *MockLogFile) WriteRotate(prefix string, num int)

func (*MockLogFile) WriteRowsQuery

func (mlf *MockLogFile) WriteRowsQuery(query string)

func (*MockLogFile) WriteStop

func (mlf *MockLogFile) WriteStop()

func (*MockLogFile) WriteTableMap

func (mlf *MockLogFile) WriteTableMap()

func (*MockLogFile) WriteTableMapWithParams

func (mlf *MockLogFile) WriteTableMapWithParams(
	tableId int8,
	dbName string,
	tableName string)

func (*MockLogFile) WriteUpdate

func (mlf *MockLogFile) WriteUpdate(before int, after int)

func (*MockLogFile) WriteUpdateWithParam

func (mlf *MockLogFile) WriteUpdateWithParam(
	before int,
	after int,
	tableId int8)

func (*MockLogFile) WriteXid

func (mlf *MockLogFile) WriteXid(id uint64)

type MockLogFileReader

type MockLogFileReader struct {
	// contains filtered or unexported fields
}

func (*MockLogFileReader) Read

func (reader *MockLogFileReader) Read(p []byte) (n int, err error)

type MockMultifileReader

type MockMultifileReader struct {
	// contains filtered or unexported fields
}

func NewMockMultifileReader

func NewMockMultifileReader(fetchFiles MockFileFetcher) *MockMultifileReader

There may be more files over time, so the MockFileFetcher is used instead of a static slice of *MockLogFile.

func (*MockMultifileReader) Close

func (r *MockMultifileReader) Close() error

func (*MockMultifileReader) NextEvent

func (r *MockMultifileReader) NextEvent() (Event, error)

type NullableColumn

type NullableColumn bool
const (
	Nullable    NullableColumn = true
	NotNullable NullableColumn = false
)

type PreviousGtidsLogEvent

type PreviousGtidsLogEvent struct {
	Event
	// contains filtered or unexported fields
}

func (*PreviousGtidsLogEvent) GtidSet

func (p *PreviousGtidsLogEvent) GtidSet() GtidSet

type PreviousGtidsLogEventParser

type PreviousGtidsLogEventParser struct {
	// contains filtered or unexported fields
}

func (*PreviousGtidsLogEventParser) EventType

PreviousGtidLogEventParser's EventType always returns mysql_proto.LogEventType_PREVIOUS_GTIDS_LOG_EVENT

func (*PreviousGtidsLogEventParser) FixedLengthDataSize

func (p *PreviousGtidsLogEventParser) FixedLengthDataSize() int

PreviousGtidLogEventParser's FixedLengthDataSize always returns 0.

func (*PreviousGtidsLogEventParser) Parse

PreviousGtidLogEventParser's Parse processes a raw gtid log event into a PreviousGtidLogEvent.

func (*PreviousGtidsLogEventParser) SetTableContext

func (*PreviousGtidsLogEventParser) SetTableContext(context TableContext)

type QueryEvent

type QueryEvent struct {
	Event
	// contains filtered or unexported fields
}

A representation of the query event.

Query event's binlog payload is structured as follow:

Common to both 5.5 and 5.6:
    19 bytes for common v4 event header
    4 bytes (uint32) for thread id which executed the query
    4 bytes (uint32) for query executation duration (in seconds)
    1 byte (uint8) for, X, the length of database name.  Note that the
        length does not include the null terminator character.
    2 bytes (uint16) for error code resulting from the query execution
    2 bytes (uint16) for, Y, the length of the variable status block
    Y bytes for the variable status block (all status are optional):
        flags2:
            1 byte for Q_FLAGS2_CODE (= 0)
            4 bytes (uint32) for flags2
        sql mode:
            1 byte for Q_SQL_MODE_CODE (= 1)
            8 bytes (uint64) for sql mode
        catalog:
            1 byte for Q_CATALOG_NZ_CODE (= 6)
            1 byte for length, Z
            Z bytes for catalog data (NOTE: As of 5.6, this value should
                always be "std")
        auto increment:
            1 byte for Q_AUTO_INCREMENT (= 3)
            2 bytes (uint16) for increment
            2 bytes (uint16) for offset
        charset:
            1 byte for Q_CHARSET_CODE (= 4)
            6 bytes for charset
        time zone:
            1 byte for Q_TIME_ZONE_CODE (= 5)
            1 byte for length, R
            R bytes for time zone
        lc time:
            1 byte for Q_LC_TIME_NAMES_CODE (= 7)
            2 bytes (uint16) for lc time names number
        charset database:
            1 byte for Q_CHARSET_DATABASE_CODE (= 8)
            2 bytes (uint16) fro charset database number
        table map for update:
            1 byte for Q_TABLE_MAP_FOR_UPDATE (= 9)
            8 bytes (uint64) for table map for update
        master data written: (not used by v4 events)
            1 byte for Q_MASTER_DATA_WRITTEN (= 10)
            4 bytes (uint32) for master data written
        invoker:
            1 byte for Q_INVOKER (= 11)
            1 byte for user length, S
            S bytes for user string
            1 byte for host length, T
            T bytes for host string
        updated db name:
            1 byte for Q_UPDATED_DB_NAMES (= 12)
            1 byte for number of dbs, N
            if N < MAX_DBS_IN_EVENT_MTS (= 254):
                N zero-terminated db name strings
        microseconds:
            1 byte for Q_MICROSECONDS (= 13)
            3 bytes (uint24) for microseconds
    X bytes for the database name (zero terminated)
    the remaining is for the query (not zero terminated).
5.6 Specific:
    (optional) 4 bytes footer for checksum.

func (*QueryEvent) AutoIncIncrement

func (e *QueryEvent) AutoIncIncrement() *uint16

AutoIncIncrement returns the auto increment status's increment. This returns nil if the status is not set.

func (*QueryEvent) AutoIncOffset

func (e *QueryEvent) AutoIncOffset() *uint16

AutoIncOffset returns the auto increment status's offset. This returns nil if the status is not set.

func (*QueryEvent) Catalog

func (e *QueryEvent) Catalog() []byte

Catalog returns the catalog status. This returns nil if the status is not set.

func (*QueryEvent) Charset

func (e *QueryEvent) Charset() []byte

Charset returns the charset status. This returns nil if the status is not set.

func (*QueryEvent) CharsetDatabaseNumber

func (e *QueryEvent) CharsetDatabaseNumber() *uint16

CharsetDatabaseNumber returns the charset database number status. This returns nil if the status is not set.

func (*QueryEvent) DatabaseName

func (e *QueryEvent) DatabaseName() []byte

DatabaseName returns the database name which was the DEFAULT database when the statement was executed.

func (*QueryEvent) Duration

func (e *QueryEvent) Duration() uint32

Duration returns the amount of time in second the query took to execute.

func (*QueryEvent) ErrorCode

func (e *QueryEvent) ErrorCode() mysql_proto.ErrorCode_Type

ErrorCode returns the error code resulting from executing the query. See https://dev.mysql.com/doc/refman/5.5/en/error-messages-server.html for additional details.

func (*QueryEvent) Flags2

func (e *QueryEvent) Flags2() *uint32

Flags2 returns the flags2 status. This returns nil if the status is not set.

func (*QueryEvent) InvokerHost

func (e *QueryEvent) InvokerHost() []byte

InvokerHost returns the invoker status's host string. This returns nil if the status is not set.

func (*QueryEvent) InvokerUser

func (e *QueryEvent) InvokerUser() []byte

InvokerUser returns the invoker status's user string. This returns nil if the status is not set.

func (*QueryEvent) IsModeEnabled

func (e *QueryEvent) IsModeEnabled(mode mysql_proto.SqlMode_BitPosition) bool

IsModeEnabled returns true iff sql mode status is set and the mode bit is set.

func (*QueryEvent) LcTimeNamesNumber

func (e *QueryEvent) LcTimeNamesNumber() *uint16

LcTimeNamesNumber returns the lc time names number status. This returns nil if the status is not set.

func (*QueryEvent) Microseconds

func (e *QueryEvent) Microseconds() *uint32

Microseconds returns the microseconds status. This returns nil if the status is not set.

func (*QueryEvent) NumUpdatedDbs

func (e *QueryEvent) NumUpdatedDbs() *uint8

NumUpdatedDbs returns the updated db status' size. This return nil if the status is not set.

func (*QueryEvent) Query

func (e *QueryEvent) Query() []byte

Query returns the query string that was executed.

func (*QueryEvent) SqlMode

func (e *QueryEvent) SqlMode() *uint64

SqlMode returns the sql mode status. This returns nil if the status is not set.

func (*QueryEvent) StatusBytes

func (e *QueryEvent) StatusBytes() []byte

StatusBytes returns the uninterpreted status block as bytes.

func (*QueryEvent) TableMapForUpdate

func (e *QueryEvent) TableMapForUpdate() *uint64

TableMapForUpdate returns the table map for update id status. This returns nil if the status is not set.

func (*QueryEvent) ThreadId

func (e *QueryEvent) ThreadId() uint32

ThreadId returns the thread id which executed the query.

func (*QueryEvent) TimeZone

func (e *QueryEvent) TimeZone() []byte

TimeZone returns the time zone status. This returns nil if the status is not set.

func (*QueryEvent) UpdatedDbNames

func (e *QueryEvent) UpdatedDbNames() [][]byte

UpdatedDbNames returns a list of names from the updated db status. This return nil if the status is not set. Also, this returns nil when NumUpdatedDbs >= MaxDbsInEventMts

type QueryEventParser

type QueryEventParser struct {
	// contains filtered or unexported fields
}

func (*QueryEventParser) EventType

QueryEventParser's EventType always returns mysql_proto.LogEventType_QUERY_EVENT.

func (*QueryEventParser) FixedLengthDataSize

func (p *QueryEventParser) FixedLengthDataSize() int

QueryEventParser's FixedLengthDataSize always returns 13.

func (*QueryEventParser) Parse

func (p *QueryEventParser) Parse(raw *RawV4Event) (Event, error)

QueryEventParser's Parse processes a raw query event into a QueryEvent.

func (*QueryEventParser) SetTableContext

func (*QueryEventParser) SetTableContext(context TableContext)

type RawV4Event

type RawV4Event struct {
	// contains filtered or unexported fields
}

A generic v4 Event entry. This event is event type agnostic, i.e., the data (including the extra headers) is not interpreted.

func (*RawV4Event) BasicHeader

func (e *RawV4Event) BasicHeader() []byte

BasicHeader returns the fixed length portion of the header bytes.

func (*RawV4Event) Bytes

func (e *RawV4Event) Bytes() []byte

Bytes returns the event payload (header + data)

func (*RawV4Event) Checksum

func (e *RawV4Event) Checksum() []byte

Checksum returns the checksum bytes (which may be empty). NOTE: by default, the checksum length is not set and this returns an empty byte slice. Use SetChecksumSize to specify the length.

func (*RawV4Event) EventLength

func (e *RawV4Event) EventLength() uint32

EventLength returns the event's total (header + body) length.

func (*RawV4Event) EventType

func (e *RawV4Event) EventType() mysql_proto.LogEventType_Type

EventType returns the event's type.

func (*RawV4Event) ExtraHeaders

func (e *RawV4Event) ExtraHeaders() []byte

RawV4Event's ExtraHeaders returns the extra header bytes associated to the event. NOTE: by default, the extra header length is not set and this returns an empty byte slice. Use SetExtraHeaderSize to specify the length.

func (*RawV4Event) FixedLengthData

func (e *RawV4Event) FixedLengthData() []byte

RawV4Event's FixedLengthData returns the fixed legnth data associated to the event. NOTE: by default, the fixed-length data's size is not set and this returns an empty byte slice. Use SetFixedLengthDataSize to specify the length.

func (*RawV4Event) Flags

func (e *RawV4Event) Flags() uint16

Flags returns the event's flags.

func (*RawV4Event) NextPosition

func (e *RawV4Event) NextPosition() uint32

NextPosition returns the next position stored in the event entry. NOTE: This value is independent of the entry real position within the source stream. Use SourcePosition() to get the correct absolute position relative to the beginning of the source stream.

func (*RawV4Event) ServerId

func (e *RawV4Event) ServerId() uint32

ServerId returns the id of the server which generated this event.

func (*RawV4Event) SetChecksumSize

func (e *RawV4Event) SetChecksumSize(size int) error

Set the checksum's size.

func (*RawV4Event) SetExtraHeadersSize

func (e *RawV4Event) SetExtraHeadersSize(size int) error

Set the extra headers' size.

func (*RawV4Event) SetFixedLengthDataSize

func (e *RawV4Event) SetFixedLengthDataSize(size int) error

Set the fixed length data's size.

func (*RawV4Event) SourceName

func (e *RawV4Event) SourceName() string

SourceName returns the name of the event's source stream.

func (*RawV4Event) SourcePosition

func (e *RawV4Event) SourcePosition() int64

SourcePosition returns the position relative to the beginning of the source stream. Unlike the next position stored in the event header, this position is always correct.

func (*RawV4Event) Timestamp

func (e *RawV4Event) Timestamp() uint32

Timestamp returns the event's timestamp.

func (*RawV4Event) VariableLengthData

func (e *RawV4Event) VariableLengthData() []byte

VariableLengthData returns the variable length data associated to the event. By default, the variable length data also include the extra headers, the fixed length data and the optional checksum footer.

type RotateEvent

type RotateEvent struct {
	Event
	// contains filtered or unexported fields
}

A representation of the rotate event. NOTE: Users should ignore rotate events that originated from relay logs.

Ratate event's binlog payload is structured as follow:

Common to both 5.5 and 5.6:
    19 bytes for common v4 event headers
    8 bytes (uint64) for offset position
    the remaining for the new log name (not zero terminated).
5.6 Specific:
    (optional) 4 bytes footer for checksum.

func (*RotateEvent) NewLogName

func (e *RotateEvent) NewLogName() []byte

NewLogName returns the name of the new log file to read from.

func (*RotateEvent) NewPosition

func (e *RotateEvent) NewPosition() uint64

NewPosition returns the position in the new log file to seek to (In practice, this should always return 4).

type RotateEventParser

type RotateEventParser struct {
	// contains filtered or unexported fields
}

func (*RotateEventParser) EventType

RotateEventParser's EventType always returns mysql_proto.LogEventType_ROTATE_EVENT.

func (*RotateEventParser) FixedLengthDataSize

func (p *RotateEventParser) FixedLengthDataSize() int

RotateEventParser's FixedLengthDataSize always returns 8.

func (*RotateEventParser) Parse

func (p *RotateEventParser) Parse(raw *RawV4Event) (Event, error)

RotateEventParser's Parse processes a raw rotate event into a RotateEvent.

func (*RotateEventParser) SetTableContext

func (*RotateEventParser) SetTableContext(context TableContext)

type RowValues

type RowValues []interface{}

A single row's used columns values.

type RowsQueryEvent

type RowsQueryEvent struct {
	Event
	// contains filtered or unexported fields
}

A representation of the rows-query event. NOTE: Rows-query event is not available in 5.5.

Rows-query event's payload is structured as follow
    19 bytes for common v4 event header.
    1 byte for the (truncated) query length (ignore this since there's a
        bug in mysql's write_str_at_most_255_bytes).
    the remaining is for the (maybe truncated) query (not zero terminated).
    (optional) 4 bytes footer for checksum.

func (*RowsQueryEvent) TruncatedQuery

func (e *RowsQueryEvent) TruncatedQuery() []byte

This returns the (possibly truncated) query string.

type RowsQueryEventParser

type RowsQueryEventParser struct {
	// contains filtered or unexported fields
}

func (*RowsQueryEventParser) EventType

RowsQueryEventParser's EventType always returns mysql_proto.LogEventType_ROWS_QUERY_EVENT.

func (*RowsQueryEventParser) FixedLengthDataSize

func (p *RowsQueryEventParser) FixedLengthDataSize() int

RowsQueryEventParser's FixedLengthDataSize always returns 0.

func (*RowsQueryEventParser) Parse

func (p *RowsQueryEventParser) Parse(raw *RawV4Event) (Event, error)

RowsQueryEventParser's Parse processes a raw query event into a RowsQueryEvent.

func (*RowsQueryEventParser) SetTableContext

func (*RowsQueryEventParser) SetTableContext(context TableContext)

type StopEvent

type StopEvent struct {
	Event
}

A representation of an event generated when mysqld stops.

Common to both 5.5 and 5.6:

The Post-Header and Body of the event are empty.
The StopEvent only contains the Common-Header.

5.6 Specific:

(optional) 4 byte footer for checksum

type TableContext

type TableContext interface {
	// TableId returns which table the context is referring to.
	TableId() uint64

	// TableFlags returns the table's flags.
	TableFlags() uint16

	// DatabaseName returns which database the table belongs to.
	DatabaseName() []byte

	// TableName returns the table's name.
	TableName() []byte

	// NumColumns returns the number of columns in the table.
	NumColumns() int

	// ColumnDescriptors returns the columns' field descriptors.
	ColumnDescriptors() []ColumnDescriptor
}

type TableContextNotSetError

type TableContextNotSetError struct {
	errors.DropboxError
}

This expected error can occur when reading from a new log file.

type TableMapEvent

type TableMapEvent struct {
	Event
	// contains filtered or unexported fields
}

A representation of the table map event.

Common to both 5.5 and 5.6:
    19 bytes for common v4 event header
    6 bytes (uint64) for table id
    2 bytes (uint16) for flags (as of 5.6, this is always 0)
    1 byte (uint8), x, for db name length (WARNING: mysql assumes the db
        name length is always less than 255; the log writer truncates
        the db name length from size_t to uchar without checking)
    x + 1 bytes for db name (zero-terminated)
    1 byte (uint8), y, for table name length (WARNING: mysql assumes the
        table name length is always less than 255; the log writer truncates
        the table name length from size_t to uchar without checking)
    y + 1 bytes for table name (zero-terminated)
    1 to 9 bytes (net_store_length variable encoded uint64), z, for number
        of columns
    z bytes for column types (1 byte per column)
    1 to 9 bytes (net_store_length variable encoded uint64), w, for
        field metadata size
    w bytes for field metadata
    ceil(z / 8) bytes for nullable columns (1 bit per column)
5.6 Specific:
    (optional) 4 bytes footer for checksum
NOTE:
    - old_row_based_repl_4_byte_map_id_master mode is not supported.

func (*TableMapEvent) ColumnDescriptors

func (e *TableMapEvent) ColumnDescriptors() []ColumnDescriptor

ColumnDescriptors returns the columns' field descriptors parsed from ColumnTypesBytes/MetadataBytes/NullColumnsBytes.

func (*TableMapEvent) ColumnTypesBytes

func (e *TableMapEvent) ColumnTypesBytes() []byte

ColumnTypesBytes returns the columns' types as uninterpreted bytes.

func (*TableMapEvent) DatabaseName

func (e *TableMapEvent) DatabaseName() []byte

DatabaseName returns which database the table belongs to.

func (*TableMapEvent) MetadataBytes

func (e *TableMapEvent) MetadataBytes() []byte

MetadataBytes returns the metadata associated to the columns as uninterpreted bytes.

func (*TableMapEvent) NullColumnsBytes

func (e *TableMapEvent) NullColumnsBytes() []byte

NullColumnsBytes returns the null column bit vector as uninterpreted bytes.

func (*TableMapEvent) NumColumns

func (e *TableMapEvent) NumColumns() int

NumColumns returns the number of columns in the table.

func (*TableMapEvent) TableFlags

func (e *TableMapEvent) TableFlags() uint16

TableFlags returns the table's flags. (As of 5.6, this is always 0).

func (*TableMapEvent) TableId

func (e *TableMapEvent) TableId() uint64

TableId returns which table the following row event entries should act on.

func (*TableMapEvent) TableName

func (e *TableMapEvent) TableName() []byte

TableName returns the table's name.

type TableMapEventParser

type TableMapEventParser struct {
	// contains filtered or unexported fields
}

func (*TableMapEventParser) EventType

TableMapEventParser's EventType always returns mysql_proto.LogEventType_TABLE_MAP_EVENT.

func (*TableMapEventParser) FixedLengthDataSize

func (p *TableMapEventParser) FixedLengthDataSize() int

TableMapEventParser's FixedLengthDataSize always returns 8.

func (*TableMapEventParser) Parse

func (p *TableMapEventParser) Parse(raw *RawV4Event) (Event, error)

TableMapEventParser's Parse processes a raw table map event into TableMapEvent.

func (*TableMapEventParser) SetTableContext

func (*TableMapEventParser) SetTableContext(context TableContext)

type UpdateRowValues

type UpdateRowValues struct {
	BeforeImage RowValues
	AfterImage  RowValues
}

A single update row's used columns values.

type UpdateRowsEvent

type UpdateRowsEvent struct {
	BaseRowsEvent
	// contains filtered or unexported fields
}

A representation of the v1 / v2 update rows event.

func (*UpdateRowsEvent) AfterImageUsedColumns

func (e *UpdateRowsEvent) AfterImageUsedColumns() []ColumnDescriptor

AfterImageUsedColumns returns the after image column descriptors that are used by the event.

func (*UpdateRowsEvent) BeforeImageUsedColumns

func (e *UpdateRowsEvent) BeforeImageUsedColumns() []ColumnDescriptor

BeforeImageUsedColumns returns the before image column descriptors that are used by the event.

func (*UpdateRowsEvent) UpdatedRows

func (e *UpdateRowsEvent) UpdatedRows() []UpdateRowValues

UpdatedRows returns the rows in the table that were mutated.

type UpdateRowsEventParser

type UpdateRowsEventParser struct {
	// contains filtered or unexported fields
}

func (*UpdateRowsEventParser) EventType

func (p *UpdateRowsEventParser) EventType() mysql_proto.LogEventType_Type

func (*UpdateRowsEventParser) FixedLengthDataSize

func (p *UpdateRowsEventParser) FixedLengthDataSize() int

func (*UpdateRowsEventParser) Parse

func (p *UpdateRowsEventParser) Parse(raw *RawV4Event) (Event, error)

func (*UpdateRowsEventParser) SetTableContext

func (p *UpdateRowsEventParser) SetTableContext(context TableContext)

type V4EventParser

type V4EventParser interface {
	// EventType returns the type of event that this parser can handle.
	EventType() mysql_proto.LogEventType_Type

	// FixedLengthDataSize returns the event's fixed-length data size.
	FixedLengthDataSize() int

	// SetTableContext provides context for rows events parsing.
	SetTableContext(context TableContext)

	// Parse processes a raw event's data bytes into a more useful
	// representation.  NOTE: the parser may assume the event's extra headers
	// size, fixed-length data and checksum size are coreectly set.  Also, it
	// may assume the raw event is of the correct event type.  When an error
	// occurs, the parser should return the original raw event along with the
	// error.
	Parse(raw *RawV4Event) (Event, error)
}

V4EventParser is the common parser interface for all v4 binlog event types.

func NewNoOpV4EventParser

func NewNoOpV4EventParser(
	eventType mysql_proto.LogEventType_Type,
	fixedLengthDataSize int) V4EventParser

NewNoOpV4EventParser returns are parser which does nothing (except setting the fixed length data size when used in conjunction with ParsedV4EventReader). This is mainly used for testing and FDE validation.

type V4EventParserMap

type V4EventParserMap interface {
	// ExtraHeadersSize returns extra headers size for all events that are not
	// format description events (FDE's extra headers size is always 0)
	ExtraHeadersSize() int

	// Get returns the parser for the specified event type.
	Get(t mysql_proto.LogEventType_Type) V4EventParser

	// ChecksumSize returns the checksum's size for all events that are not
	// format description events (FDE's checksum size is always 0 for
	// mysql 5.5, 4 for mysql 5.6).
	ChecksumSize() int

	// SetChecksumSize is used for specifying the non-FDE events' checksum size.
	SetChecksumSize(size int)

	// SetTableContext sets the table map context for all registered
	// parsers.
	SetTableContext(context TableContext)

	// SetNumSupportedEventTypes sets the number of supported event types.
	// Calls to Get will return nil when the event type is larger than this
	// upper bound.
	SetNumSupportedEventTypes(num int)
}

V4EventParserMap holds a set of V4EventParsers.

func NewV4EventParserMap

func NewV4EventParserMap() V4EventParserMap

NewV4EventParserMap returns an initialize V4EventParserMap with all handled event types' parsers registered.

type WriteRowsEvent

type WriteRowsEvent struct {
	BaseRowsEvent
	// contains filtered or unexported fields
}

A representation of the v1 / v2 write rows event.

func (*WriteRowsEvent) InsertedRows

func (e *WriteRowsEvent) InsertedRows() []RowValues

InsertedRows returns the rows written into the table.

func (*WriteRowsEvent) UsedColumns

func (e *WriteRowsEvent) UsedColumns() []ColumnDescriptor

UsedColumns returns the column descriptors that are used by the event.

type WriteRowsEventParser

type WriteRowsEventParser struct {
	// contains filtered or unexported fields
}

func (*WriteRowsEventParser) EventType

func (p *WriteRowsEventParser) EventType() mysql_proto.LogEventType_Type

func (*WriteRowsEventParser) FixedLengthDataSize

func (p *WriteRowsEventParser) FixedLengthDataSize() int

func (*WriteRowsEventParser) Parse

func (p *WriteRowsEventParser) Parse(raw *RawV4Event) (Event, error)

func (*WriteRowsEventParser) SetTableContext

func (p *WriteRowsEventParser) SetTableContext(context TableContext)

type XidEvent

type XidEvent struct {
	Event
	// contains filtered or unexported fields
}

A representation of the xid event.

Xid event's binlog payload is structured as follow:

Common to both 5.5 and 5.6:
    19 bytes for common v4 event headers
    8 bytes (uint64) for xid.  NOTE: xid is written using the master's
        machine endianness.  The id's value will differ when read on
        different processor platforms; however, replication will function
        correctly since the uniqueness of the id is preserved.
5.6 Specific:
    (optional) 4 bytes footer for checksum

func (*XidEvent) Xid

func (e *XidEvent) Xid() uint64

Xid returns the event's transaction id.

type XidEventParser

type XidEventParser struct {
	// contains filtered or unexported fields
}

func (*XidEventParser) EventType

XidEventParser's EventType always returns mysql_proto.LogEventType_XID_EVENT

func (*XidEventParser) FixedLengthDataSize

func (p *XidEventParser) FixedLengthDataSize() int

XidEventParser's FixedLengthDataSize always return 0.

func (*XidEventParser) Parse

func (p *XidEventParser) Parse(raw *RawV4Event) (Event, error)

XidEventParser's Parse processes a raw xid event into a XidEvent.

func (*XidEventParser) SetTableContext

func (*XidEventParser) SetTableContext(context TableContext)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL