Documentation
¶
Overview ¶
Package fssource is a connector that reads a directory tree.
It is the reference connector, the one the interface was written against and the one to read first when writing another. It also does real work: pointed at a checkout of a documentation repository it produces a corpus with real titles, real bodies, real modification times and, with the right policy, real access control lists.
Permissions come from a policy, not from the walk ¶
The connector maps files to documents. It does not decide who may read them. That split exists because a directory tree says almost nothing about access on its own: the mode bits describe the account the crawler is running as, not the people in the company. Somewhere above the filesystem there is a real answer, in an OWNERS file, in a group export or in a share database, and a Policy is where that answer is plugged in.
There is no permissive default. A source built without a policy quarantines everything, which is loud and safe, rather than publishing a directory tree to everybody, which is quiet and not.
Incremental sync ¶
The cursor is the highest modification time the last run saw, held back to a second before that run started. A later run walks the same tree and skips anything not newer, so the cost of a no change sync is a stat of every file rather than a read of every file. That is the honest limit of what a plain filesystem supports on its own: without a change feed there is no way to avoid the walk, only the read.
The second is not slack in the design, it is the design. A file is stamped from a clock that ticks rather than from the one that runs, so a file created a moment after the walk went past its directory carries the same modification time as one the walk read, and a cursor sitting on that time would leave it behind for good. The cost of holding back is that a file written in the second before a sync is read by the next sync as well, which is a document arriving twice rather than a document not arriving at all.
A Watcher is how the walk goes too. The operating system already knows which files were written, and a source built with WithWatcher asks it instead of asking the tree, which turns the cost of a sync from a function of how large the corpus is into a function of how much of it moved. A watcher that cannot vouch for what it recorded says so, and that sync walks, so the worst a watcher can do is cost what not having one costs.
Index ¶
- Constants
- type Checker
- type OSPolicy
- type Option
- func WithInclude(f func(name string) bool) Option
- func WithMaxDocumentSize(n int64) Option
- func WithMaxFileSize(n int64) Option
- func WithMaxImageSize(n int64) Option
- func WithPace(wait func(context.Context) error) Option
- func WithSkipDir(f func(name string) bool) Option
- func WithSkipped(f func(path string, reason error)) Option
- func WithWatcher(w *Watcher) Option
- type OwnersPolicy
- func (o *OwnersPolicy) ChangedAt(ctx context.Context, relPath string) (time.Time, error)
- func (o *OwnersPolicy) Counts() aclmap.Counts
- func (o *OwnersPolicy) IsRule(relPath string) bool
- func (o *OwnersPolicy) Permissions(ctx context.Context, relPath string) (acl.Permissions, error)
- func (o *OwnersPolicy) Reload()
- func (o *OwnersPolicy) WithFallback(p acl.Permissions) *OwnersPolicy
- type Policy
- type PolicyFunc
- type Reloader
- type Ruled
- type Source
- func (s *Source) Close() error
- func (s *Source) Counters() connector.Counters
- func (s *Source) Enumerate(ctx context.Context, fn func(connector.Item) bool) error
- func (s *Source) Fetch(ctx context.Context, id string) (doc.Document, error)
- func (s *Source) Source() string
- func (s *Source) Sync(ctx context.Context, from connector.Cursor, ...) (connector.Cursor, error)
- type Versioned
- type WatchOption
- type WatchStats
- type Watcher
Constants ¶
const DefaultMaxDocumentSize = 16 << 20
DefaultMaxDocumentSize is the largest PDF or Office file read for extraction.
It is the third limit for the same reason there is a second one. A one megabyte text file is almost certainly not prose and a one megabyte screenshot is a screenshot, and an eight megabyte PDF is an ordinary report with a chart in it. The bytes here are compressed and the text inside is a fraction of them, so the number that actually bounds the work is extract.Options.MaxDecompressed rather than this one.
const DefaultMaxFileSize = 1 << 20
DefaultMaxFileSize is the largest file read into a document body.
Past this the file is almost never prose somebody wants to search, and it is often a checked in binary that would cost far more to index than it is worth.
const DefaultMaxImageSize = 4 << 20
DefaultMaxImageSize is the largest image read into a document's content.
It is separate from the body limit and larger, because the two limits are about different things. A one megabyte text file is almost certainly not prose. A one megabyte screenshot is a screenshot.
const DefaultMaxPending = 100_000
DefaultMaxPending bounds how many changed paths one watcher will hold between syncs.
Past this the record has stopped being a saving. Walking the tree is bounded work and remembering an unbounded list of paths is not, so a tree that is being rewritten wholesale, which is what a build directory nobody excluded looks like, goes back to the walk instead of growing a map until the process is killed.
const DefaultMaxWatches = 4096
DefaultMaxWatches bounds how many directories one watcher holds.
Every backend charges for a watch and they charge differently. Linux keeps a per user limit on inotify watches that a large tree will reach, and the kqueue backend the BSDs and macOS use needs an open file descriptor per watched file, which reaches the process limit a great deal sooner. Rather than find that out as a stream of errors halfway through a tree, a watcher that would need more than this refuses to be built and says how many it wanted, and the caller falls back to walking.
const OwnersFile = "OWNERS"
OwnersFile is the name of the file an OwnersPolicy reads.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Checker ¶
type Checker struct {
// contains filtered or unexported fields
}
Checker answers whether somebody may still read a file, while a response is being written rather than at the next sync.
The permissions in the index were read when the crawler last came past. On a tree people are working in, that copy goes wrong in the direction that matters: an OWNERS file is edited on Monday morning, a mode bit is taken away at lunchtime, and until the next walk the index still says the old thing. A checker closes that window by reading the rule again, per request, for the handful of documents that are about to go on somebody's screen.
It is cheap here and that is why the filesystem is the first source to have one. The answer is on the same machine: the id carries the path, the rule is a file a few directories up, and both are almost always in the page cache. A source reached over somebody else's API is a different conversation about what fits inside a query, and it is not this type.
The shape is what the recheck package asks for, so a Checker can be registered under a source name without either package importing the other.
The policy has to be its own ¶
A checker must not be given the same Policy the running source holds. OwnersPolicy caches its answers for the length of a walk, which is right for a sync and wrong here: a check reading that cache would be answering out of the snapshot it exists to go around, and would agree with the stale index every time. So a checker reloads its policy before every batch, and a policy being reloaded underneath a sync would cost that sync its cache.
Build a second one. They are cheap, and the two have different lifetimes for a reason.
func NewChecker ¶
NewChecker returns a checker for the tree under root.
name is the source name the documents carry, which is what the ids are prefixed with and what the checker is registered under.
A nil policy is refused rather than treated as permissive. A checker without one would answer no to everything, which is safe and is also a server that serves nothing, and the difference between the two is worth finding at startup rather than in a support conversation.
func (*Checker) Allowed ¶
func (c *Checker) Allowed(ctx context.Context, p *acl.Principal, ids []string) (map[string]bool, error)
Allowed reports which of the ids the principal may still read.
An id left out of the answer is not an allow. The caller reads a missing id as a check that did not happen and removes the document, so everything this method cannot answer for is simply not in the map: an id from another source, a path that tried to leave the tree, a file it could not stat, and a rule that stopped resolving. What is in the map with a false is the ordinary answer, that the file is gone or that this person is no longer on it.
The policy is reloaded once per call, not once per id, which is what keeps a page of results at one read of each OWNERS file rather than one per document.
type OSPolicy ¶
type OSPolicy struct {
// contains filtered or unexported fields
}
OSPolicy derives permissions from what the operating system keeps on the files themselves: the owner, the group and the mode bits on Unix, and the discretionary access control list on Windows.
It is the policy for a tree that is the file server, because there the operating system is the access control system and there is nothing better above it. It is the wrong policy for a copy of one. A tree that was rsynced to the crawler carries the permissions the copy has, which are the crawler's own, and indexing those would hand the lot to whoever the crawler runs as.
Unix ¶
The owner and the group become a user and a group reference under the identity source given to NewOSPolicy, and each one is a grant only where its read bit is set. An owner who took away their own read bit could put it back, so there is an argument for granting to them regardless, and the literal reading is used instead: being wrong this way costs somebody a file of their own they cannot find, and being wrong the other way costs a file shown to somebody who was refused it.
The world bit is the one that cannot be mapped without being told something first. It says every account on this host may read the file, and a host's accounts are not a tenant: on a laptop they are one person, on a login server they are the company, and on a machine with a guest account they are more than the company. So it grants nothing at all unless NewOSPolicy was given the domain those accounts belong to.
Where a file carries a POSIX access control list, that list is read in place of the mode bits, because the two disagree in the direction that leaks. The group bits of such a file are the mask rather than the group's own permission, so a group the mask has taken read away from still reads as allowed, and a mapping built on the mode alone offers the file to people who cannot open it.
The extended access control lists macOS and the BSDs keep are a different format in a different place, reachable only through the C library, and they are not read. A file carrying one gets the answer its mode bits give, which is narrower than the truth where the list grants and wider than it where the list refuses. That is the one gap in this policy that goes the wrong way, and it is stated here rather than left to be found: on those systems this is a good answer for an ordinary tree and not a safe one for a tree somebody has been managing with the access control list editor.
Windows ¶
The owner comes from the security descriptor and the grants come from the discretionary access control list, which is the one platform here with refusals in it. A refusal becomes a deny and beats every grant, which is what Windows does too.
Entries that exist only to be inherited by files created later are skipped, because they say nothing about this file. Entries naming Everyone or Authenticated Users go through the same door as the Unix world bit and grant nothing until a domain names them.
Names ¶
Every identifier is resolved to an account name through the password and group databases, or through the account database on Windows, and a file whose owner does not resolve is quarantined rather than indexed under a number. A numeric user id is not an identity: it means one person on one host and somebody else on the next, so a grant written in terms of one would either match nobody or match the wrong person.
func NewOSPolicy ¶
NewOSPolicy returns a policy reading the permissions the operating system keeps on the files under root.
source names the connector. identity names the identity source the account names belong to, for example "unix" for a machine's own accounts or the name of the directory the host is joined to. Getting that one right is what lets somebody who authenticated through the company directory match a list that came out of a password file.
worldDomain is optional and is the domain the accounts on this host belong to. Give it and a world readable file is readable by everybody in the tenant. Leave it out and the world bit grants nothing, which is the safe reading and the reason it is not the caller's job to remember to pass a flag.
func (*OSPolicy) Counts ¶
Counts returns what the mapping has seen, so that a deployment can watch the quarantine numbers rather than hear about them from somebody who cannot find a document.
func (*OSPolicy) Permissions ¶
Permissions returns the access control list the operating system holds for a path.
func (*OSPolicy) Reload ¶
func (p *OSPolicy) Reload()
Reload drops the resolved account names.
The names are the only thing cached here, because the permissions themselves are read from the file every time and there is nothing to go stale about them. A name can still go stale: an account renamed between two syncs would otherwise keep the name it had when the process started, and a grant to a name nobody has any more is a grant to nobody.
type Option ¶
type Option func(*Source)
Option configures a source.
func WithInclude ¶
WithInclude replaces the rule for which file names are read. The argument is the base name.
func WithMaxDocumentSize ¶
WithMaxDocumentSize sets the largest PDF or Office file that will be read for extraction. A value below one selects DefaultMaxDocumentSize.
func WithMaxFileSize ¶
WithMaxFileSize sets the largest file that will be read. A value below one selects DefaultMaxFileSize.
func WithMaxImageSize ¶
WithMaxImageSize sets the largest image that will be read into a document's content. A value below one selects DefaultMaxImageSize.
func WithPace ¶
WithPace makes the source wait before it reads each file's content.
A first read of a large tree is the one moment this connector competes with the server it is feeding. The walk is stat calls and the reads are the disk, and a process doing both flat out is a process whose queries are answered off a disk that is busy with something else. Slowing the read down is the only lever that helps, because the work itself cannot be made smaller: every file has to be read once.
It is a function rather than a number so that what the pace is stays with whoever chose it. A token bucket, a semaphore shared with something else and a plain sleep are all the same shape from here, and the connector has no opinion about which of them a deployment wants.
l := limit.NewLimiter(limit.Limits{Rate: 200, Burst: 1}, nil)
src, err := fssource.New(root, "docs", policy, fssource.WithPace(l.Wait))
The error it returns stops the sync rather than skipping the file, because the only thing that makes a pace fail is the context being done, and a sync that carried on from there would spend a shutdown reading the rest of the tree. A nil function is what passing no option means: read as fast as the disk allows.
func WithSkipDir ¶
WithSkipDir replaces the rule for directories that are not descended into. The argument is the base name.
func WithSkipped ¶
WithSkipped installs a callback for files the walk passed over.
A sync does not abandon a tree because one file in it could not be read. A file whose owner revoked the permission, or that was deleted between the listing and the stat, is a fact about the tree and not a reason to lose the hundred thousand files after it.
What it must not be is silent. An index quietly missing the files nobody could read looks exactly like an index that is complete, and the difference only shows up when somebody cannot find a document they know exists. This is how a caller finds out: it is called once per skipped file with the path and the reason, and the default does nothing.
func WithWatcher ¶
WithWatcher makes the source ask a watcher what changed instead of walking the tree to find out.
The watcher is built separately and owned by the caller, because building one fails for reasons that are a property of the machine rather than of the program, and a caller that cannot have one should carry on with a source that walks rather than fail to start.
w, err := fssource.Watch(root)
if err != nil {
log.Warn("watching the tree, syncs will walk it instead", "err", err)
}
src, err := fssource.New(root, "docs", policy, fssource.WithWatcher(w))
A nil watcher is allowed and means exactly what passing no option means, so the error above does not need a second branch.
The watcher has to be watching the same tree the source is reading, and New refuses the pair if it is not. It also has to skip the same directories, which is what WithWatchSkipDir is for.
type OwnersPolicy ¶
type OwnersPolicy struct {
// contains filtered or unexported fields
}
OwnersPolicy derives permissions from OWNERS files in the tree.
This is the convention Kubernetes and a number of other large repositories use: a file named OWNERS in a directory lists the people who approve and review changes below it, and the nearest one going up the tree wins. It is worth supporting because it is a real access control list, maintained by real people for their own reasons, over a corpus anybody can check out. Developing a permission model against it means developing against the awkward parts, such as a subtree that narrows the list its parent set, which invented test data never has because nobody invents inconvenience.
The mapping is deliberately literal. Approvers and reviewers become allowed users under the identity source given to NewOwnersPolicy, and the first approver becomes the owner. Nothing is inferred beyond that.
A file below a directory with no OWNERS file anywhere above it has no answer, and gets one that does not resolve, so it is quarantined rather than published. That is the case worth getting right: the default for "no rule found" is not "no restriction".
func NewOwnersPolicy ¶
func NewOwnersPolicy(root, source, identity string) (*OwnersPolicy, error)
NewOwnersPolicy returns a policy reading OWNERS files under root.
source names the connector, and identity names the identity source the entries in the file belong to, for example "github". Getting that second one right is what lets a principal who authenticated through one system match a list written in terms of another.
func (*OwnersPolicy) ChangedAt ¶
ChangedAt returns when the OWNERS file governing relPath was last written.
A path no OWNERS file governs has no answer and returns the zero time, even when a fallback is set, because a fallback is a constant and a constant never changed.
func (*OwnersPolicy) Counts ¶
func (o *OwnersPolicy) Counts() aclmap.Counts
Counts returns what the mapping has seen, so that a deployment can watch the numbers rather than find out from somebody who cannot find a document.
func (*OwnersPolicy) IsRule ¶
func (o *OwnersPolicy) IsRule(relPath string) bool
IsRule reports whether a path is an OWNERS file.
A watched sync that sees one of these gives up on its record and walks the tree that round, because this is the one edit whose effect is nowhere near the file that changed. Rewriting the OWNERS file at the root of a repository changes who may read every document in it, and the only event anybody raised was about the OWNERS file.
func (*OwnersPolicy) Permissions ¶
func (o *OwnersPolicy) Permissions(ctx context.Context, relPath string) (acl.Permissions, error)
Permissions returns the access control list governing relPath.
func (*OwnersPolicy) Reload ¶
func (o *OwnersPolicy) Reload()
Reload drops the cache so that the next lookup reads the tree again.
A source calls this at the start of every walk. Without it a process that stays up for a week answers with the OWNERS files as they were when it started, and a revocation made on Tuesday is applied to nothing. Holding the cache for the length of one walk is what keeps the cost at one read per OWNERS file per sync instead of one per document.
func (*OwnersPolicy) WithFallback ¶
func (o *OwnersPolicy) WithFallback(p acl.Permissions) *OwnersPolicy
WithFallback sets the permissions used for a path with no OWNERS file above it.
Without it those paths are quarantined. Set it when the tree genuinely has a default, such as a public documentation repository where the unowned files are as public as the owned ones, and leave it alone otherwise.
type Policy ¶
Policy decides who may read a file.
It is called once per file with the path relative to the root, using forward slashes on every platform so that a policy written against a repository layout does not have to care where it is running.
Returning an error quarantines that one document and does not stop the walk, because one unreadable share should not cost a whole sync.
func PublicToTenant ¶
PublicToTenant is a policy where every file is readable by everybody in the tenant.
It is correct for a public documentation corpus and wrong for almost everything else, so it has to be asked for by name.
type PolicyFunc ¶
PolicyFunc adapts a function to Policy.
func (PolicyFunc) Permissions ¶
func (f PolicyFunc) Permissions(ctx context.Context, relPath string) (acl.Permissions, error)
Permissions calls f.
type Reloader ¶
type Reloader interface {
Reload()
}
Reloader is the optional capability of a Policy that caches.
A source calls it once at the start of every walk. That is the contract the caching in a policy is allowed to assume: answers may be held for the length of one sync and must not be held across two, because the thing a later sync exists to notice is exactly the edit that would invalidate them.
type Ruled ¶
type Ruled interface {
// IsRule reports whether a path, relative to the root and separated by
// forward slashes, is one of the policy's rule files.
IsRule(relPath string) bool
}
Ruled is the optional capability of a Policy whose answers come out of files in the tree it is being asked about.
It exists for the watched sync and for nothing else. A watcher reports the file that changed, and an OWNERS file that changed governs who may read every document in the subtree below it, none of which the watcher will ever mention. A sync that read only the reported paths would apply the edit to the OWNERS file itself and to nothing it rules.
So a policy says which paths are its rules, and a sync that finds one of them among the changes walks the tree that round instead. That is the expensive answer, and it is the right one: an OWNERS edit is rare, and getting it wrong means a revocation that silently did not happen.
OSPolicy needs none of this. Its rule for a file is the file's own mode, and changing that raises an event on the file itself.
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source reads documents out of a directory tree.
func New ¶
New returns a source reading root, naming itself name, and asking policy who may read each file.
A nil policy is allowed and quarantines every document. That is deliberate: it makes "I have not thought about permissions yet" a visible state in the stats rather than an invisible one in the index.
func (*Source) Enumerate ¶
Enumerate calls fn for every file the source would index, with its modification time as the version.
It is the same walk Source.Sync does and it reads nothing. That is the whole difference in price on a filesystem: a hundred thousand stats against a hundred thousand reads, which on a real corpus is a second against a minute.
func (*Source) Fetch ¶
Fetch reads one file by document id.
It returns connector.ErrGone for a file that is no longer there, which is the normal answer on a tree people are working in rather than a failure.
func (*Source) Sync ¶
func (s *Source) Sync(ctx context.Context, from connector.Cursor, emit func(context.Context, connector.Change) error) (connector.Cursor, error)
Sync walks the tree and emits every file modified after the cursor.
The returned cursor is the highest modification time seen in the whole tree, including files that were skipped as unchanged, so that a run which finds nothing new still moves the clock forward and a run interrupted halfway does not lose the files it already passed. It is never later than a second before the sync started, for the reason in [settled].
A source built with WithWatcher reads the paths the watcher recorded and does not walk at all, whenever the watcher can vouch for its record. When it cannot, this is what runs, which is why the two paths have to agree on which files are in the corpus.
type Versioned ¶
type Versioned interface {
// ChangedAt returns when the rule governing relPath last changed. A zero
// time means the policy has no rule for it, or has no idea, and nothing is
// refreshed on the strength of it.
ChangedAt(ctx context.Context, relPath string) (time.Time, error)
}
Versioned is the optional capability of a Policy that can say when its answer for a path last changed.
It is what turns a permission change into a write instead of a recrawl. A sync that only compares modification times cannot see one at all: the file did not change, the rule above it did. A policy that implements this is asked about every file the walk decided to skip, and the ones whose rule moved get a permissions only change.
The answer has to be cheap. It is asked once per unchanged file, which on a large tree is once per file, so anything that costs a read has to be cached for the length of the walk.
type WatchOption ¶
type WatchOption func(*Watcher)
WatchOption configures a watcher.
func WithMaxPending ¶
func WithMaxPending(n int) WatchOption
WithMaxPending sets how many changed paths a watcher will remember between syncs. A value below one selects DefaultMaxPending.
func WithMaxWatches ¶
func WithMaxWatches(n int) WatchOption
WithMaxWatches sets how many directories a watcher will hold. A value below one selects DefaultMaxWatches.
func WithWatchSkipDir ¶
func WithWatchSkipDir(f func(name string) bool) WatchOption
WithWatchSkipDir replaces the rule for directories that are not watched. The argument is the base name.
It should be the same rule the source walks with. A watcher that descended into a directory the source skips would spend its watches on a dependency tree and report changes to files nothing indexes, and one that skipped a directory the source reads would silently miss every change in it.
type WatchStats ¶
WatchStats is what a watcher has done.
Walks is the number worth looking at. It is how many times the record could not be trusted and a sync had to walk the tree, and on a healthy watcher it is one, from the first sync. Anything more says the tree is churning past what the backend will carry, and Reason says which way.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher records what changed in a tree, so that a sync can read those files instead of walking to find them.
It is built separately from the Source and owned by the caller, because building one can fail in ways that are a property of the machine rather than of the program: a tree over the inotify limit, a filesystem the backend does not support, a process near its descriptor limit. A caller that gets an error here logs it and carries on with a source that walks, which is the behaviour it would have had anyway.
func Watch ¶
func Watch(root string, opts ...WatchOption) (*Watcher, error)
Watch starts watching root and everything under it.
The watcher is not trusted until a sync has walked the tree, so the first sync after this is a full walk whatever happens.
func (*Watcher) Close ¶
Close stops the watcher and releases every watch it holds.
A source built with this watcher goes back to walking, which is what it does with no watcher at all, so closing one is safe while a sync is running.
func (*Watcher) Stats ¶
func (w *Watcher) Stats() WatchStats
Stats reports what the watcher has done.
Source Files