dbf

package module
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 24, 2023 License: MIT Imports: 16 Imported by: 1

README

go-foxpro-dbf

GoDoc Go Report Card codecov

Golang package for reading FoxPro DBF/FPT files.

This package provides a reader for reading FoxPro database files. At this moment it is only tested for Alaska XBase++ DBF/FPT files in FoxPro format and some older FoxPro files, see the included testdbf folder for these files. These files have file flag 0x30 (or 0x31 if autoincrement fields are present).

Since these files are almost always used on Windows platforms the default encoding is from Windows-1250 to UTF8 but a universal encoder will be provided for other code pages.

Features

There are several similar packages but they are not suited for our use case, this package will try to implement:

  • Support for FPT (memo) files
  • Full support for Windows-1250 encoding to UTF8
  • File readers for scanning files (instead of reading the entire file to memory)

The focus is on performance while also trying to keep the code readable and easy to use.

Supported field types

At this moment not all FoxPro field types are supported. When reading field values, the value returned by this package is always interface{}. If you need to cast this to the correct value helper functions are provided.

The supported field types with their return Go types are:

Field Type Field Type Name Golang type
B Double float64
C Character string
D Date time.Time
F Float float64
I Integer int32
L Logical bool
M Memo string
M Memo (Binary) []byte
N Numeric (0 decimals) int64
N Numeric (with decimals) float64
T DateTime time.Time
Y Currency float64

Example

func Test() error {
	// Open file
	testdbf, err := dbf.OpenFile("TEST.DBF", new(dbf.Win1250Decoder))
	if err != nil {
		return err
	}
	defer testdbf.Close()

	// Print all the fieldnames
	for _, name := range testdbf.FieldNames() {
		fmt.Println(name)
	}

	// Get fieldinfo for all fields
	for _, field := range testdbf.Fields() {
		fmt.Println(field.FieldName(), field.FieldType(), field.Decimals /*etc*/)
	}

	// Read the complete second record
	record, err := testdbf.RecordAt(1)
	if err != nil {
		return err
	}
	// Print all the fields in their Go values
	fmt.Println(record.FieldSlice())

	// Loop through all records using recordpointer in DBF struct
	// Reads the complete record
	for !testdbf.EOF() { // or for i := uint32(0); i < testdbf.NumRecords(); i++ {

		// This reads the complete record
		record, err := testdbf.Record()
		if err != nil {
			return err
		}
		testdbf.Skip(1)

		// skip deleted records
		if record.Deleted {
			continue
		}
		// get field by position
		field1, err := record.Field(0)
		if err != nil {
			return err
		}
		// get field by name
		field2, err := record.Field(testdbf.FieldPos("NAAM"))
		if err != nil {
			return err
		}

		fmt.Println(field1, field2)
	}

	// Read only the third field of records 2, 50 and 300
	recnumbers := []uint32{2, 50, 300}
	for _, rec := range recnumbers {
		err := testdbf.GoTo(rec)
		if err != nil {
			return err
		}
		deleted, err := testdbf.Deleted()
		if err != nil {
			return err
		}
		if !deleted {
			field3, err := testdbf.Field(3)
			if err != nil {
				return err
			}
			fmt.Println(field3)
		}
	}

	return nil
}

Example using a byte reader

You can use OpenStream with any ReaderAt and ReadSeeker combo, for example a bytes.Reader. The FPT stream is only required when the DBF header indicates there must be an FPT file.

func TestBytes() error {
	
	dbfbytes, err := ioutil.ReadFile("TEST.DBF")
	if err != nil {
		return err
	}
	dbfreader := bytes.NewReader(dbfbytes)

	fptbytes, err := ioutil.ReadFile("TEST.FPT")
	if err != nil {
		return err
	}
	fptreader := bytes.NewReader(fptbytes)

	test_dbf, err = OpenStream(dbfreader, fptreader, new(Win1250Decoder))
	if err != nil {
		return err
	}
	defer testdbf.Close()

	// Print all the fieldnames
	for _, name := range testdbf.FieldNames() {
		fmt.Println(name)
	}
	
	// ETC...
	
	return nil	
}

Thanks

Documentation

Overview

Package dbf provides code for reading data from FoxPro DBF/FPT files

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrEOF is returned when on end of DBF file (after the last record)
	ErrEOF = errors.New("EOF")

	// ErrBOF is returned when the record pointer is attempted to be moved before the first record
	ErrBOF = errors.New("BOF")

	// ErrIncomplete is returned when the read of a record or field did not complete
	ErrIncomplete = errors.New("incomplete read")

	// ErrInvalidField is returned when an invalid fieldpos is used (<1 or >NumFields)
	ErrInvalidField = errors.New("invalid field pos")

	// ErrNoFPTFile is returned when there should be an FPT file but it is not found on disc
	ErrNoFPTFile = errors.New("no FPT file")

	// ErrNoDBFFile is returned when a file operation is attempted on a DBF but a reader is open
	ErrNoDBFFile = errors.New("no DBF file")

	// ValidFileVersionFunc can be used to override file version checks to open untested files
	ValidFileVersionFunc = validFileVersion
)
View Source 
var ErrInvalidUTF8 = errors.New("invalid UTF-8 data")

Functions

func SetValidFileVersionFunc added in v1.1.2 

func SetValidFileVersionFunc(f func(version byte) error)

SetValidFileVersionFunc sets variable ValidFileVersionFunc to a new function used to verify if the opened file has a valid version flag.

Example 
// create function which checks that only file flag 0x03 is valid
validFileVersionFunc := func(version byte) error {
	if version == 0x03 {
		return nil
	}
	return errors.New("not 0x03")
}

// set the new function as verifier
SetValidFileVersionFunc(validFileVersionFunc)

// open DBF as usual
dbf, err := OpenFile("/var/test.dbf", new(Win1250Decoder))
if err != nil {
	log.Fatal(err)
}
defer dbf.Close()

Output:

func ToBool 

func ToBool(in interface{}) bool

ToBool always returns a boolean

func ToFloat64 

func ToFloat64(in interface{}) float64

ToFloat64 always returns a float64

func ToInt64 

func ToInt64(in interface{}) int64

ToInt64 always returns an int64

func ToString 

func ToString(in interface{}) string

ToString always returns a string

func ToTime 

func ToTime(in interface{}) time.Time

ToTime always returns a time.Time

func ToTrimmedString 

func ToTrimmedString(in interface{}) string

ToTrimmedString always returns a string with spaces trimmed

Types

type DBF 

type DBF struct {
	// contains filtered or unexported fields
}

DBF is the main DBF struct which provides all methods for reading files and embeds the file readers and handlers.

func OpenFile 

func OpenFile(filename string, dec Decoder) (*DBF, error)

OpenFile opens a DBF file (and FPT if needed) from disk. After a successful call to this method (no error is returned), the caller should call DBF.Close() to close the embedded file handle(s). The Decoder is used for charset translation to UTF8, see decoder.go

func OpenStream 

func OpenStream(dbffile, fptfile ReaderAtSeeker, dec Decoder) (*DBF, error)

OpenStream creates a new DBF struct from a bytes stream, for example a bytes.Reader The fptfile parameter is optional, but if the DBF header has the FPT flag set, the fptfile must be provided. The Decoder is used for charset translation to UTF8, see decoder.go

func (*DBF) BOF 

func (dbf *DBF) BOF() bool

BOF returns if the internal recordpointer is at BoF (first record)

func (*DBF) Close 

func (dbf *DBF) Close() error

Close closes the file handlers to the disk files. The caller is responsible for calling Close to close the file handle(s)!

func (*DBF) Deleted 

func (dbf *DBF) Deleted() (bool, error)

Deleted returns if the record at the internal record pos is deleted

func (*DBF) DeletedAt 

func (dbf *DBF) DeletedAt(recordpos uint32) (bool, error)

DeletedAt returns if the record at recordpos is deleted

func (*DBF) EOF 

func (dbf *DBF) EOF() bool

EOF returns if the internal recordpointer is at EoF

func (*DBF) Field 

func (dbf *DBF) Field(fieldpos int) (interface{}, error)

Field reads field number fieldpos at the record number the internal pointer is pointing to and returns its Go value

func (*DBF) FieldNames 

func (dbf *DBF) FieldNames() []string

FieldNames returnes a slice of all the fieldnames

func (*DBF) FieldPos 

func (dbf *DBF) FieldPos(fieldname string) int

FieldPos returns the zero-based field position of a fieldname or -1 if not found.

func (*DBF) Fields 

func (dbf *DBF) Fields() []FieldHeader

Fields returns all the FieldHeaders

func (*DBF) GoTo 

func (dbf *DBF) GoTo(recno uint32) error

GoTo sets the internal record pointer to record recno (zero based). Returns ErrEOF if at EOF and positions the pointer at lastRec+1.

func (*DBF) Header 

func (dbf *DBF) Header() *DBFHeader

Header returns the DBF Header struct for inspecting

func (*DBF) NumFields 

func (dbf *DBF) NumFields() uint16

NumFields returns the number of fields

func (*DBF) NumRecords 

func (dbf *DBF) NumRecords() uint32

NumRecords returns the number of records

func (*DBF) Record 

func (dbf *DBF) Record() (*Record, error)

Record reads the complete record the internal record pointer is pointing to

func (*DBF) RecordAt 

func (dbf *DBF) RecordAt(nrec uint32) (*Record, error)

RecordAt reads the complete record number nrec

func (*DBF) RecordToJSON 

func (dbf *DBF) RecordToJSON(nrec uint32, trimspaces bool) ([]byte, error)

RecordToJSON returns a complete record as a JSON object. If nrec > 0 it returns the record at nrec, if nrec <= 0 it returns the record at dbf.recpointer. If trimspaces is true we trim spaces from string values (this is slower because of an extra reflect operation and all strings in the record map are re-assigned)

func (*DBF) RecordToMap 

func (dbf *DBF) RecordToMap(nrec uint32) (map[string]interface{}, error)

RecordToMap returns a complete record as a map. If nrec > 0 it returns the record at nrec, if nrec <= 0 it returns the record at dbf.recpointer

func (*DBF) Skip 

func (dbf *DBF) Skip(offset int64) error

Skip adds offset to the internal record pointer. Returns ErrEOF if at EOF and positions the pointer at lastRec+1. Returns ErrBOF is recpointer would be become negative and positions the pointer at 0. Does not skip deleted records.

func (*DBF) Stat 

func (dbf *DBF) Stat() (os.FileInfo, error)

Stat returns the os.FileInfo for the DBF file

func (*DBF) StatFPT 

func (dbf *DBF) StatFPT() (os.FileInfo, error)

StatFPT returns the os.FileInfo for the FPT file

type DBFHeader 

type DBFHeader struct {
	FileVersion byte     // File type flag
	ModYear     uint8    // Last update year (0-99)
	ModMonth    uint8    // Last update month
	ModDay      uint8    // Last update day
	NumRec      uint32   // Number of records in file
	FirstRec    uint16   // Position of first data record
	RecLen      uint16   // Length of one data record, including delete flag
	Reserved    [16]byte // Reserved
	TableFlags  byte     // Table flags
	CodePage    byte     // Code page mark
}

DBFHeader is the struct containing all raw DBF header fields. Header info from https://docs.microsoft.com/en-us/previous-versions/visualstudio/foxpro/st4a0s68(v=vs.80)

func (*DBFHeader) FileSize 

func (h *DBFHeader) FileSize() int64

FileSize eturns the calculated file size based on the header info

func (*DBFHeader) Modified 

func (h *DBFHeader) Modified() time.Time

Modified parses the ModYear, ModMonth and ModDay to time.Time. Note: The year is stored in 2 digits, 15 is 2015, we assume here that all files were modified after the year 2000 and always add 2000.

func (*DBFHeader) NumFields 

func (h *DBFHeader) NumFields() uint16

NumFields returns the calculated number of fields from the header info alone (without the need to read the fieldinfo from the header). This is the fastest way to determine the number of records in the file. Note: when OpenFile is used the fields have already been parsed so it is better to call DBF.NumFields in that case.

type Decoder 

type Decoder interface {
	Decode(in []byte) ([]byte, error)
}

Decoder is the interface as passed to OpenFile

type FPTHeader 

type FPTHeader struct {
	NextFree  uint32  // Location of next free block
	Unused    [2]byte // Unused
	BlockSize uint16  // Block size (bytes per block)
}

FPTHeader is the raw header of the Memo file. Header info from https://docs.microsoft.com/en-us/previous-versions/visualstudio/foxpro/8599s21w(v=vs.80)

type FieldHeader 

type FieldHeader struct {
	Name     [11]byte // Field name with a maximum of 10 characters. If less than 10, it is padded with null characters (0x00).
	Type     byte     // Field type
	Pos      uint32   // Displacement of field in record
	Len      uint8    // Length of field (in bytes)
	Decimals uint8    // Number of decimal places
	Flags    byte     // Field flags
	Next     uint32   // Value of autoincrement Next value
	Step     uint16   // Value of autoincrement Step value
	Reserved [8]byte  // Reserved
}

FieldHeader contains the raw field info structure from the DBF header. Header info from https://docs.microsoft.com/en-us/previous-versions/visualstudio/foxpro/st4a0s68(v=vs.80)

func (*FieldHeader) FieldName 

func (f *FieldHeader) FieldName() string

FieldName returns the name of the field as a trimmed string (max length 10)

func (*FieldHeader) FieldType 

func (f *FieldHeader) FieldType() string

FieldType returns the type of the field as string (length 1)

type ReaderAtSeeker 

type ReaderAtSeeker interface {
	io.ReadSeeker
	io.ReaderAt
}

ReaderAtSeeker is used when opening files from memory

type Record 

type Record struct {
	Deleted bool
	// contains filtered or unexported fields
}

Record contains the raw record data and a deleted flag

func (*Record) Field 

func (r *Record) Field(pos int) (interface{}, error)

Field gets a fields value by field pos (index)

func (*Record) FieldSlice 

func (r *Record) FieldSlice() []interface{}

FieldSlice gets all fields as a slice

type UTF8Decoder 

type UTF8Decoder struct{}

UTF8Decoder assumes your DBF is in UTF8 so it does nothing

func (*UTF8Decoder) Decode 

func (d *UTF8Decoder) Decode(in []byte) ([]byte, error)

Decode decodes a UTF8 byte slice to a UTF8 byte slice

type UTF8Validator added in v1.1.1 

type UTF8Validator struct{}

UTF8Validator checks if valid UTF8 is read

func (*UTF8Validator) Decode added in v1.1.1 

func (d *UTF8Validator) Decode(in []byte) ([]byte, error)

Decode decodes a UTF8 byte slice to a UTF8 byte slice

type Win1250Decoder 

type Win1250Decoder struct{}

Win1250Decoder translates a Windows-1250 DBF to UTF8

func (*Win1250Decoder) Decode 

func (d *Win1250Decoder) Decode(in []byte) ([]byte, error)

Decode decodes a Windows1250 byte slice to a UTF8 byte slice

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.