README
¶
serval
serval — Scoped Enquiry over Records, Values And Lists (a wild cat of the tall grass, which finds what it wants in a field by listening and takes it in one leap)
If you use this, please support me on ko-fi: https://ko-fi.com/jeffday
Reading a long sequence of records a screenful at a time, without ever holding
all of it and without asking twice for what you already have. It knows nothing
about terminals, protocols, rendering or documents, and its only dependency
outside the standard library is github.com/phroun/pawscript, for reading PSL.
It's a sibling to the rest of the line: Mew (text editor), PurfecTerm (terminal emulator), PawScript (language), KittyTK (UI toolkit), khatool (Hebrew and Arabic text) and ifitfits (viewport tiling).
Model
- A query is asked once and answered once. You state a sequence — a
source, a filter, a sort — and that naming never changes: a different filter
is a different sequence, opened alongside the first and taking its place.
Then you ask for scopes of it. Each scope is a separate question, and
Completeends the answer rather than the interest. - A scope is not a filter and names no field. It says start past this record, give me this many, stop if you reach that one, and walk backwards if I say so. Both ends are optional and both are identities, never positions — where a record stands is the business of whoever put it there, and a source made of several others hands each of them that one's own identity rather than passing one down.
- An answer says how far it is complete. A
watermarkmeans nothing between where you asked from and this record is missing. Three ways to stop —filled,joined,exhausted— because a scope that ran out of records and one that reached its count look identical from outside and mean opposite things about whether to ask again. - Sources compose.
AmendedSourcelays additions, replacements and deletions over another source without touching it.ComposedSourcereads several as one merged sequence, in the merged order, with each include asked only what it can answer. Both are Sources, so they nest. - Answers are kept. A cached scope is a run of records guaranteed complete between its two ends — the same claim a watermark makes, which is why an answer and a cache entry are the same object. Scrolling extends a run rather than making another; two runs that meet join; a run cut in two by an insert is two runs and not a re-query. Eviction is segmented, so a scan of a million records cannot flush out the handful you were actually using.
- Identity is not a spelling.
KeyandEqualdecide whether two values are the same value, and they owe nothing to any grammar: text and bytes are different kinds, 3 and 3.0 are different numbers, and floats compare by their bits so that a key is reflexive.
Use
import "github.com/phroun/serval"
src := serval.NewPSLSource(node, serval.Whole)
set, err := src.Open(&serval.Spec{
Sort: []serval.SortLevel{{Field: "name"}},
})
if err != nil {
return err
}
defer set.Close()
// One screenful, then the next, from where the first one got to.
err = set.Read(&serval.Scope{Count: 30}, sink)
err = set.Read(&serval.Scope{After: watermark, Count: 30}, sink)
A sink takes the records one at a time and is told what ended them:
type Sink interface {
Ordered() // before the first record, or not at all
Record(id *serval.Value, f serval.Record) error // the record entire
Subset(id *serval.Value, f serval.Record) error // only what was asked for
Done(c serval.Complete)
}
Test
go test ./...
License
MIT — see LICENSE.
Documentation
¶
Index ¶
- Constants
- func AppendKey(b []byte, v *Value) []byte
- func Compare(a, b *Value, collation string) int
- func CompareLevels(a, b []*Value, levels []Level) int
- func Equal(a, b *Value) bool
- func FilterKey(f *Filter) string
- func Key(v *Value) string
- func Match(id *Value, rec Subject, f *Filter) bool
- func Rank(v *Value) int
- func SortKey(levels []SortLevel) string
- type AmendedSource
- type Complete
- type ComposedSource
- type DataSet
- type Field
- type Filter
- type Include
- type Kind
- type Level
- type PSLSource
- type Reading
- type Record
- type Scope
- type Sink
- type SortLevel
- type Source
- type Spec
- type Stop
- type Subject
- type Value
Constants ¶
const ( CollateExact = "exact" // codepoint order, rune by rune CollateFold = "fold" // ASCII case folded, then exact CollateNatural = "natural" // digit runs as numbers, then fold )
The collations text may be compared under. A symbol and a run of bytes take none: a symbol is a name, and bytes are not text.
const ( RankUndefined = iota RankNil RankFalse RankTrue RankNumber // int and float share one rank and interleave by value RankSymbol RankString RankBytes RankUnordered // no order of its own; the sort's last level settles it )
A value's rank, which its kind decides before anything in it is read.
const ( KeyField = "key" ValueField = "value" )
The two names the Whole reading gives a record's own key and value.
These are fields, and only fields. A record read this way carries `key` because the document gave it one and somebody may want to sort or show it -- not because this end works by it. What identifies the record travels beside the bag, and a document is free to hold a member called `key` of its own without the two ever meaning the same thing.
const ( OpAnd = "and" OpOr = "or" OpNot = "not" OpEq = "eq" OpNe = "ne" OpLt = "lt" OpLe = "le" OpGt = "gt" OpGe = "ge" OpIn = "in" OpContains = "contains" OpStarts = "starts" OpEnds = "ends" // OpHas and OpLacks ask whether a record carries a field at all, and take // no value. Every other operator compares one, and a field holding // something with no order of its own -- a nested list -- cannot be // compared, so presence needs an operator that does not try. OpHas = "has" OpLacks = "lacks" // OpID matches a record's identity against a set of them, the way `in` // matches a field against a set of values. It names no field, because an // identity is not one: it travels beside a record's fields rather than // among them, and a field called `key` is a field like any other. OpID = "id" )
The operators a filter is built from.
Variables ¶
This section is empty.
Functions ¶
func AppendKey ¶
AppendKey appends a value's Key to b, for a caller building a key out of more than one thing. A table keyed by a sequence and an identity wants one buffer and one string, not a key per part and a join afterwards.
func Compare ¶
Compare orders two values: -1, 0 or 1. The collation applies to the string rank alone and is CollateExact when empty.
Two values of different ranks are decided by the ranks. Two of the same rank are decided by what is in them, except the ranks that hold one value each (undefined, nil, false, true) and the unordered rank, which compare equal and leave the answer to the sort's last level.
func CompareLevels ¶
CompareLevels walks the levels in order and answers on the first that separates the two runs, so each level settles only what the ones above it left equal. A run shorter than the levels stops where it ends.
The caller appends the record's identity as a final level: two records equal on every level a sort names must still have an order, or "the record after this point" names more than one place.
func Equal ¶
Equal reports whether two values are the same value.
Three things it decides, and each is written down in 0_identity_test.go so that changing one is a deliberate act:
- Text and bytes of the same content are NOT the same value. They are different kinds -- one is characters and one is not -- and they do not even sort together.
- An integer and a float of the same magnitude are not the same value either. 3 and 3.0 are different numbers here, and a boundary that changed type between two holders of the same sequence would be comparing different things.
- Floats compare by their bits and not by ==, which makes NaN equal to itself and 0.0 different from -0.0. That is not a preference: a key must be reflexive, or a record whose identity does not equal itself is filed and never found again, and == on NaN gives exactly that.
func Key ¶
Key is a value's canonical form: two values have the same Key exactly when Equal says they are the same value.
Self-delimiting, so keys can be concatenated -- which is what a table keyed by more than one thing does, and what a list of them does inside one key.
func Match ¶
Match reports whether a record passes a filter. A nil filter passes everything, which is what an unfiltered sequence is.
The identity comes in beside the record rather than as a field of it, because that is what it is: `id` asks about the identity, and every other operator asks about a field. A record whose fields happen to include one called `key` is answering an ordinary question about an ordinary field. Pass nil for id where there is none, and `id` then matches nothing.
func Rank ¶
Rank is where a value sits before its contents matter. A value that is not there at all is undefined, which is the bottom of the order.
The kind decides it outright. Nothing is read out of the value and nothing is spelled: which rank a value has is a fact about what it IS.
Types ¶
type AmendedSource ¶
type AmendedSource struct {
// contains filtered or unexported fields
}
An AmendedSource holds replacements and deletions against a child's records.
func NewAmendedSource ¶
func NewAmendedSource(child Source) *AmendedSource
NewAmendedSource amends the records of a child source. The child is any kind -- records here, records an application's, or another amended source.
func (*AmendedSource) Add ¶
func (a *AmendedSource) Add(key *Value, fields Record)
Add says this source holds a record of its own under this key.
It is not a statement about anything the child holds: the key is the author's to choose, and keeping it clear of the child's is the author's to do. Where it turns out not to be clear, the child's record is the one that stands and this one is dropped for good -- the opposite of Replace, which displaces whatever the child has.
func (*AmendedSource) Delete ¶
func (a *AmendedSource) Delete(key *Value, known Record)
Delete says a record is gone.
Known is what was last seen of it, and may be nil. With it, the scope that record would have fallen in is known before the child is asked; without it, the shortfall is discovered afterwards and costs a second question -- which is also where the fields to remember are learned.
func (*AmendedSource) Forget ¶
func (a *AmendedSource) Forget(key *Value)
Forget drops an amendment, leaving the child's own record to stand.
func (*AmendedSource) Open ¶
func (a *AmendedSource) Open(spec *Spec) (DataSet, error)
Open states a sequence, and opens the same one on the child.
func (*AmendedSource) Replace ¶
func (a *AmendedSource) Replace(key *Value, fields Record)
Replace says a record now carries these fields, whatever the child holds.
These are the record entire, not a correction to some of it: what goes out for this key is exactly what is stated here, and it goes out as a whole record.
type Complete ¶
type Complete struct {
Watermark *Value
Stop Stop
// Error is a refusal, which is an answer: this scope cannot be produced,
// the records are gone, whatever held them is no longer reachable.
// Whoever asked carries on with what it has.
Error string
}
A Complete ends a scope: which of the three ways it ended, and how far the answer is complete.
Watermark says there is nothing between where the scope was asked from and that record that the asker does not now have. StopExhausted carries none, because there is no point past the end to be complete up to.
type ComposedSource ¶
type ComposedSource struct {
// contains filtered or unexported fields
}
A ComposedSource answers out of several sources at once.
func NewComposedSource ¶
func NewComposedSource(includes ...Include) (*ComposedSource, error)
NewComposedSource composes sources under the names their records go out under.
The includes are fixed for the life of the source. A sequence opened over it opens one over each of them, so a set that changed underneath would leave a sequence answering out of children it never opened.
func (*ComposedSource) Includes ¶
func (c *ComposedSource) Includes() []Include
Includes are the sources this one is made of, in the order they were given.
func (*ComposedSource) Open ¶
func (c *ComposedSource) Open(spec *Spec) (DataSet, error)
Open states a sequence, and opens it on every include that could hold anything the filter admits.
Each include gets the filter read in its own terms: an identity this source made says which include it came from, so `filter={ id (left/1) }` asks `left` about its own record 1 and never opens `right` at all.
type DataSet ¶
type DataSet interface {
// Read asks for one scope of the sequence and says where to put it.
//
// It does not wait for the answer. Records reach the sink as they are
// produced -- immediately, for a source whose records are here; as they
// arrive, for one whose records are somewhere else -- and the sink is told
// what ended it when it ends. The error is for a request that could not be
// started at all, never for one that has not finished.
//
// A scope names its ends by identity, and an identity means something only
// to the source that issued it. So a scope handed here was issued here: a
// source made of several others translates rather than relays.
Read(s *Scope, out Sink) error
// Close lets the data set go, and with it whatever it was holding.
Close()
}
A DataSet is one stated sequence -- this source, this sort, this filter -- prepared once and drawn from until it is let go. It is what a query names, seen from the end that holds the records.
Those three name it, and nothing else does: the fields a query asks for change what a scope carries rather than which records are in the sequence, and the direction belongs to the scope. So two queries naming the same three are reading one data set, and whatever was worked out for either of them -- the ordering, where each record stands -- holds for both.
It does not change. A different sort or a different filter is a different data set, opened alongside this one and taking its place -- which is also what keeps the source in use while the reader moves from one to the other.
type Field ¶
type Field struct {
// Name is what the member is called, or "" for a member that has no name of
// its own and stands by its position.
Name string
// Value is what is under the name. Nil is undefined, which is a value a
// field can have: a name present with nothing under it.
Value *Value
}
A Field is one member of a record: a name, and what is under it.
type Filter ¶
type Filter struct {
Op string
Field string // predicates: the field being tested
Values []*Value // what it is tested against; more than one only for `in`
Collate string // text predicates: the collation, "" for the default
Children []*Filter // and, or, not
}
A Filter is a tree: a predicate over one field, a test of a record's identity, or an and/or/not over other nodes.
The top of a built filter is an OpAnd, so there is one shape to walk whether the filter held one predicate or twenty.
func (*Filter) String ¶
String is the filter as readable text, in the same shape Record uses: each node as an operator, the field it names and what it is tested against, and a braced block where a node has children. For an error message, a log line, or a test saying what it expected in one line -- never parsed back.
type Include ¶
An Include is one of the sources a ComposedSource is made of, under the name its records go out prefixed with.
type Kind ¶
type Kind int
Kind is what a value is.
const ( NilValue Kind = iota // nothing, deliberately: a field that has no value BoolValue // true or false NumberValue // an integer or a float; IsInt says which SymbolValue // a name the data uses: an identifier, an enum TextValue // text BytesValue // bytes that are not text ListValue // a record in its own right: a nested thing )
type Level ¶
Level is one level of a sort, as it applies to a pair of values already taken from the two records being ordered.
type PSLSource ¶
type PSLSource struct {
// contains filtered or unexported fields
}
A PSLSource is a data source backed by one parsed PSL list.
It is read-only and its records do not move, so everything computed from them stays true: an ordering is built once per spec and reused for every scope drawn from it.
func NewPSLSource ¶
NewPSLSource presents an already-parsed PSL list as a data source: its items as records keyed by index, its keyed members as records keyed by name.
func ParsePSLSource ¶
ParsePSLSource reads PSL text and presents it as a data source.
type Reading ¶
type Reading int
A Reading is how a record's contents are named. Both are useful and neither is a subset of the other's behaviour, so it is said once, when the source is made, rather than guessed per record.
const ( // Whole exposes the record entire. `key` is the document's own key for it // and `value` is its value, and every member wears a dot: `.size` is the // member called size, `.0` is the item at position 0. // // Nothing can shadow anything, so a member called `key` -- which the bundle // format's own `_bundle: (key: "figaro")` has -- is `.key` and is reachable // like any other. It is the reading for data whose records are not all the // same shape, and for anything that has to survive a round trip. // // `key` here is a field. It is what the document called this record, handy // to sort or show; what this end identifies the record by travels beside // the bag and is nobody's field. Whole Reading = iota // Members exposes the members alone, under their own names: `size`, not // `.size`. It is the shorter reading, and it suits a source whose records // are all lists of named fields -- which is most of them. // // It is deliberately not complete: the record's own key and value have no // name here, and positions have none either, a bare `0` being the number it // spells rather than a name. A member called `key` is a // member like any other and is reached and sent as one -- the point of this // reading is the document at its face value, and nothing in the bag is an // identity for it to collide with. Members )
type Record ¶
type Record []*Field
A Record is a run of fields.
func (Record) Field ¶
Field makes a Record a Subject, which is the common case: a record already in our own shape is tested as it stands.
func (Record) Get ¶
Get is the value under a name, or nil where the record does not name it. A field present with no value reads as nil too: a bare name is a name, not a value of its own.
type Scope ¶
type Scope struct {
// After is the record to start past: the asker holds it already. Nil
// starts at the first record in walk order.
After *Value
// Until is the record to stop before: the asker holds that one too, and
// everything beyond it, so a walk that reaches it has joined two runs the
// asker held separately. Nil walks until the count is reached or the
// records run out.
Until *Value
// Count is how many records are wanted.
Count int
// Reversed walks the sequence from its end rather than its beginning.
//
// Every level turns over, the one the sort does not write included: an
// identity settles what the named levels leave equal, and a sequence read
// backwards settles it backwards too. That is what makes this the exact
// mirror -- `size desc` turns one level over and leaves ties facing the way
// they were, which is a different sequence again.
//
// It belongs to the scope rather than the sequence because it costs
// nothing: one prepared ordering is read either way, where a reversed
// *sequence* would be a second ordering of the same records. And it names
// no field, so it is the one way to turn over a sequence whose records are
// read in a way that cannot name their identity at all.
Reversed bool
}
A Scope is the run of records a query asks for: where to start, which way to walk, how many, and where the asker's own knowledge picks up again.
It is not a filter and it names no field. The sequence is already decided by the spec, and a scope only says which part of it to read -- so a source prepares one ordering and serves every scope of it cheaply, rather than preparing a new one because the reader scrolled.
After and Until are identities, not positions. An identity means something only to the source that issued it, which is why a source made of several others never passes one down: it hands each of them that one's own.
type Sink ¶
type Sink interface {
// Ordered says the records about to arrive are in the sequence's order.
// It comes before the first of them or not at all, which is the only place
// it is worth anything: a sink that learns it afterwards can no longer act
// on the records it has already been given. Not being told means not
// ordered, which is always safe.
Ordered()
// Record takes one record entire: its identity, and every field it has.
//
// Whole is worth saying because it outlives the scope that asked for it.
// A record that arrived entire answers any question about that record, so
// whoever holds it can answer the next query out of it instead of asking
// again.
//
// The identity is beside the fields, not among them. A record may well
// carry a field called `key`, and that field is data: it sorts, it
// filters, it fills a column. What names the record is this.
Record(id *Value, fields Record) error
// Subset takes some of a record: its identity, and the fields that were
// asked for, which are fewer than the record has.
//
// It answers the question that asked for it and no other. A later question
// naming a field this one left out is not answered by what came back here,
// however many of the same records it names.
Subset(id *Value, fields Record) error
Done(c Complete)
}
A Sink takes an answer as it is produced: the records one at a time, and then what ended them.
One at a time because a scope of a million records need not be assembled anywhere before the first of them moves, and because a source whose records are across a connection has them in that shape already.
type SortLevel ¶
A SortLevel is one level of a sort: which field, which way, and -- for text -- under which collation.
type Source ¶
type Source interface {
// Open states a sequence: one filter, one sort. It is refused where the
// sequence cannot be produced exactly -- an op that is not implemented, a
// collation that is not carried -- because an ordering that is quietly a
// little different corrupts every answer after it and looks like data.
Open(spec *Spec) (DataSet, error)
}
A Source is a body of records that sequences can be read out of.
type Spec ¶
type Spec struct {
Source string
Fields Record // the fields asked for; empty means whatever the record has
Exclude Record // the fields not wanted, valued the same way
Filter *Filter
Sort []SortLevel
}
A Spec says what sequence a query names: which records, in which order.
Fields and Exclude are what the asker WANTS. What a source carries back may be any superset of that -- sending more than was asked for is always allowed -- but what it CLAIMS to carry must be true, because an answer kept against a later question is answered from what it claims.
type Stop ¶
type Stop string
A Stop is why a scope ended, which the asker cannot work out for itself.
A scope that filled and one that ran out of records look identical from the far end -- both are a run of records that stopped -- and they mean opposite things about whether there is any point asking again.
const ( // StopFilled: the count was reached. There is more past the watermark. StopFilled Stop = "filled" // StopJoined: the walk reached `until`, so what the asker holds on this // side and what it holds on the other are now one run. StopJoined Stop = "joined" // StopExhausted: there are no more records this way. Nothing past the end // to be complete up to, so there is no watermark either. StopExhausted Stop = "exhausted" )
type Subject ¶
A Subject is anything a filter can take a field out of: the thing being tested, whatever shape it is held in.
An interface rather than a Record so that a source holding its records in its own form can put one to a filter without copying it into ours first -- which for a filter that rejects it would be a copy made for nothing.
A field the subject has not got reads as nil, which is `undefined`: a value with a rank of its own rather than an error. That is what makes a test against undefined a question about presence.
type Value ¶
type Value struct {
Kind Kind
Str string // SymbolValue, TextValue, BytesValue
Int int64 // NumberValue, when IsInt
Num float64 // NumberValue, otherwise
IsInt bool // NumberValue: written with no fractional part, and it fits
Bool bool // BoolValue
List Record // ListValue
}
A Value is one value of a record.
The fields a kind does not use are not read and carry no meaning. Str serves the three kinds that are a run of characters -- a symbol's name, text, and bytes as they stand -- because they differ in what they MEAN and not in how they are held.
func NewNil ¶
func NewNil() *Value
The constructors. A value is built, never parsed: whatever read the text owns the reading of it.
func Val ¶
Val is a Go value as a Value, for the callers that have one of a handful of ordinary types and do not want to say which. Anything it does not know becomes undefined rather than a guess.
func (*Value) String ¶
String is a value as readable text, for an error message, a log line or a test that wants to say what it expected in one line.
Readable and not canonical: it is for a person, it is never parsed back, and nothing turns on it. Key is what tables are keyed by and Equal is what decides whether two values are the same one. The shape -- names and values separated by spaces inside braces -- is shared with Record and Filter so that one glance reads all three.
Source Files