disk

package
v0.42.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 8, 2022 License: Apache-2.0 Imports: 25 Imported by: 3

Documentation

Overview

Package disk provides disk-based implementation of the storage.Store interface.

The disk.Store implementation uses an embedded key-value store to persist policies and data. Policy modules are stored as raw byte strings with one module per key. Data is mapped to the underlying key-value store with the assistance of caller-supplied "partitions". Partitions allow the caller to control the portions of the /data namespace that are mapped to individual keys. Operations that span multiple keys (e.g., a read against the entirety of /data) are more expensive than reads that target a specific key because the storage layer has to reconstruct the object from individual key-value pairs and page all of the data into memory. By supplying partitions that align with lookups in the policies, callers can optimize policy evaluation.

Partitions are specified as a set of storage paths (e.g., {/foo/bar} declares a single partition at /foo/bar). Each partition tells the store that values under the partition path should be mapped to individual keys. Values that fall outside of the partitions are stored at adjacent keys without further splitting. For example, given the partition set {/foo/bar}, /foo/bar/abcd and /foo/bar/efgh are be written to separate keys. All other values under /foo are not split any further (e.g., all values under /foo/baz would be written to a single key). Similarly, values that fall outside of partitions are stored under individual keys at the root (e.g., the full extent of the value at /qux would be stored under one key.) There is support for wildcards in partitions: {/foo/*} will cause /foo/bar/abc and /foo/buz/def to be written to separate keys. Multiple wildcards are supported (/tenants/*/users/*/bindings), and they can also appear at the end of a partition (/users/*).

All keys written by the disk.Store implementation are prefixed as follows:

/<schema_version>/<partition_version>/<type>

The <schema_version> value represents the version of the schema understood by this version of OPA. Currently this is always set to 1. The <partition_version> value represents the version of the partition layout supplied by the caller. Currently this is always set to 1. Currently, the disk.Store implementation only supports _additive_ changes to the partitioning layout, i.e., new partitions can be added as long as they do not overlap with existing unpartitioned data. The <type> value is either "data" or "policies" depending on the value being stored.

The disk.Store implementation attempts to be compatible with the inmem.store implementation however there are some minor differences:

* Writes that add partitioned values implicitly create an object hierarchy containing the value (e.g., `add /foo/bar/abcd` implicitly creates the structure `{"foo": {"bar": {"abcd": ...}}}`). This is unavoidable because of how nested /data values are mapped to key-value pairs.

* Trigger events do not include a set of changed paths because the underlying key-value store does not make them available.

Example (Store)
package main

import (
	"context"
	"fmt"
	"io/ioutil"
	"os"

	"github.com/open-policy-agent/opa/logging"
	"github.com/open-policy-agent/opa/storage"
	"github.com/open-policy-agent/opa/storage/disk"
	"github.com/open-policy-agent/opa/util"
)

func check(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {

	ctx := context.Background()

	// Create a temporary directory for store.
	dir, err := ioutil.TempDir("", "opa_disk_example")
	check(err)

	// Cleanup temporary directory after finishing.
	defer os.RemoveAll(dir)

	// Create a new disk-based store.
	store, err := disk.New(ctx, logging.NewNoOpLogger(), nil, disk.Options{
		Dir: dir,
		Partitions: []storage.Path{
			storage.MustParsePath("/authz/tenants"),
		},
	})
	check(err)

	// Insert data into the store. The `storage.WriteOne` function automatically
	// opens a write transaction, applies the operation, and commits the
	// transaction in one-shot.
	err = storage.WriteOne(ctx, store, storage.AddOp, storage.MustParsePath("/"), util.MustUnmarshalJSON([]byte(`{
		"authz": {
			"tenants": {
				"acmecorp.openpolicyagent.org": {
					"tier": "gold"
				},
				"globex.openpolicyagent.org" :{
					"tier": "silver"
				}
			}
		}
	}`)))
	check(err)

	// Close the store so that it can be reopened.
	err = store.Close(ctx)
	check(err)

	// Re-create the disk-based store using the same options.
	store2, err := disk.New(ctx, logging.NewNoOpLogger(), nil, disk.Options{
		Dir: dir,
		Partitions: []storage.Path{
			storage.MustParsePath("/authz/tenants"),
		},
	})
	check(err)

	// Read value persisted above and inspect the result.
	value, err := storage.ReadOne(ctx, store2, storage.MustParsePath("/authz/tenants/acmecorp.openpolicyagent.org"))
	check(err)

	err = store2.Close(ctx)
	check(err)

	fmt.Println(value)

}
Output:


map[tier:gold]

Index

Examples

Constants

This section is empty.

Variables

View Source
var ErrInvalidPartitionPath = errors.New("invalid storage path")

Functions

This section is empty.

Types

type Options

type Options struct {
	Dir        string         // specifies directory to store data inside of
	Partitions []storage.Path // data prefixes that enable efficient layout
	Badger     string         // badger-internal configurables
}

Options contains parameters that configure the disk-based store.

func OptionsFromConfig added in v0.39.0

func OptionsFromConfig(raw []byte, id string) (*Options, error)

OptionsFromConfig parses the passed config, extracts the disk storage settings, validates it, and returns a *Options struct pointer on success.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store provides a disk-based implementation of the storage.Store interface.

func New

func New(ctx context.Context, logger logging.Logger, prom prometheus.Registerer, opts Options) (*Store, error)

New returns a new disk-based store based on the provided options.

func (*Store) Abort

func (db *Store) Abort(ctx context.Context, txn storage.Transaction)

Abort implements the storage.Store interface.

func (*Store) Close

func (db *Store) Close(context.Context) error

Close finishes the DB connection and allows other processes to acquire it.

func (*Store) Commit

func (db *Store) Commit(ctx context.Context, txn storage.Transaction) error

Commit implements the storage.Store interface.

func (*Store) DeletePolicy

func (db *Store) DeletePolicy(ctx context.Context, txn storage.Transaction, id string) error

DeletePolicy implements the storage.Policy interface.

func (*Store) GC added in v0.39.0

func (db *Store) GC(logger logging.Logger)

func (*Store) GetPolicy

func (db *Store) GetPolicy(ctx context.Context, txn storage.Transaction, id string) ([]byte, error)

GetPolicy implements the storage.Policy interface.

func (*Store) ListPolicies

func (db *Store) ListPolicies(ctx context.Context, txn storage.Transaction) ([]string, error)

ListPolicies implements the storage.Policy interface.

func (*Store) MakeDir added in v0.39.0

func (db *Store) MakeDir(_ context.Context, txn storage.Transaction, path storage.Path) error

MakeDir makes Store a storage.MakeDirer, to avoid the superfluous MakeDir steps -- MakeDir is implicit in the disk storage's data layout, since

{"foo": {"bar": {"baz": 10}}}

writes value `10` to key `/foo/bar/baz`.

Here, we only check if it's a write transaction, for consistency with other implementations, and do nothing.

func (*Store) NewTransaction

func (db *Store) NewTransaction(ctx context.Context, params ...storage.TransactionParams) (storage.Transaction, error)

NewTransaction implements the storage.Store interface.

func (*Store) Read

func (db *Store) Read(ctx context.Context, txn storage.Transaction, path storage.Path) (interface{}, error)

Read implements the storage.Store interface.

func (*Store) Register

Register implements the storage.Trigger interface.

func (*Store) Truncate added in v0.42.0

Truncate implements the storage.Store interface. This method must be called within a transaction.

func (*Store) UpsertPolicy

func (db *Store) UpsertPolicy(ctx context.Context, txn storage.Transaction, id string, bs []byte) error

UpsertPolicy implements the storage.Policy interface.

func (*Store) Write

func (db *Store) Write(ctx context.Context, txn storage.Transaction, op storage.PatchOp, path storage.Path, value interface{}) error

Write implements the storage.Store interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL