objecthash

README

(Note: this repo is a fork from Ben Laurie's objecthash (https://github.com/benlaurie/objecthash) with some additional features and fixes, which hopefully can be incorporated back upstream as time permits - README below is the original)

objecthash

A way to cryptographically hash objects (in the JSON-ish sense) that works cross-language. And, therefore, cross-encoding.

Build it with:

$ make get
GOPATH=`pwd` go get golang.org/x/text/unicode/norm
$ make
<lots of test output>

OK (I hope)

You only need to do the make get the first time.

Take a look at objecthash_test.*.

Comments/bugs/patches welcome.

What's it good for?

Signing and verification

Now you can sign objects regardless of the transport encoding and programming language used.

Verifiable append-only logs of objects

Certificate Transparency provides a verifiable append-only log of certificates. The same thing can be done for arbitrary objects using objecthash. And you can serve the objects themselves in whatever format is handy - JSON, XML, RDF ... you name it. (note to self: implement XML and RDF encodings)

Redaction

Sometimes you want to show different people different things - maybe some information is private, or maybe your policy changed about what is shown since you wrote the log (remember that once you've logged to a verifiable append-only log you can't change the hash of the log entry). objecthash allows you to redact parts of objects, yet still verify them. For example, the JSON structure:

["foo", {"bar": ["baz", null, 1.0, 1.5, 0.0001, 1000.0, 2.0, -23.1234, 2.0]}]

has hash 783a423b094307bcb28d005bc2f026ff44204442ef3513585e7e73b66e3c2213. The substructure:

{"bar": ["baz", null, 1.0, 1.5, 0.0001, 1000.0, 2.0, -23.1234, 2.0]}

has hash 96e2aab962831956c80b542f056454be411f870055d37805feb3007c855bd823. The redacted structure:

["foo", "**REDACTED**96e2aab962831956c80b542f056454be411f870055d37805feb3007c855bd823"]

matches the original hash, 783a423b094307bcb28d005bc2f026ff44204442ef3513585e7e73b66e3c2213. So does:

["foo", {"bar": ["**REDACTED**82f70430fa7b78951b3c4634d228756a165634df977aa1fada051d6828e78f30", null, 1.0, 1.5, "**REDACTED**1195afc7f0b70bb9d7960c3615668e072a1cbfbbb001f84871fd2e222a87be1d", 1000.0, 2.0, -23.1234, 2.0]}]

or even:

["foo", {"**REDACTED**e303ce0bd0f4c1fdfe4cc1e837d7391241e2e047df10fa6101733dc120675dfe": ["baz", null, 1.0, 1.5, 0.0001, 1000.0, 2.0, -23.1234, 2.0]}]

Magic!

What You Hash Is What You Get

Most object signing/verifying schemes (e.g. X509v3, JOSE) work by signing or verifying some canonical binary or text form of the object, which you then decode and hope you end up with what was actually signed. Using objecthash you decode first, then verify. If verification works, then the object you have is the object that was signed in the first place.

Redactability

If your data is guessable, then redaction doesn't really help: the data can easily be reconstructed with a brute force attack. So, objecthash offers a way to decorate an object so that everything in it can be redacted and be safe from brute-forcing. redactable(o) turns every basic object (except keys) in o into an array with two entries, the first being a 32 byte random string. Since keys are required to be strings, those are prefixed with the random string.

unredactable(o) reverses the process.

Common vs. Python JSON

Python distinguishes between int and float when parsing JSON, which makes it incompatible with GO - all numbers are float in Go. Common JSON functions convert Python JSON to Common JSON first.

Documentation

Index

• Constants
• Variables
• func CommonJSONHash(j []byte) ([]byte, error)
• func Filtered(o interface{}, allowed string) (interface{}, error)
• func ObjectHash(o interface{}) ([]byte, error)
• func ObjectHashWithRedaction(o interface{}, redPref string) ([]byte, error)
• func ObjectHashWithStdRedaction(o interface{}) ([]byte, error)
• func RedactWithRedaction(i interface{}, redPref string) (string, error)
• func RedactWithStdRedaction(i interface{}) (string, error)
• func Redactable(o interface{}) (interface{}, error)
• func RedactableIt(p interface{}) (interface{}, error)
• func Unredactable(o interface{}, redPref string) (interface{}, error)
• func UnredactableWithStdPrefix(o interface{}) (interface{}, error)
• type Filterer
  • func CreateFilterer(allowed string) *Filterer
  • func (self *Filterer) IsAllowed(path []string) bool
• type Set

Examples

• CommonJSONHash (Common)
• CommonJSONHash (FloatAndInt)
• CommonJSONHash (KeyChange)
• CommonJSONHash (KeyOrderIndependence)
• ObjectHash (ComplexSet)
• ObjectHash (ComplexSetRepeated)
• ObjectHash (Json)
• ObjectHash (Json2)
• ObjectHash (Set)

Constants

const (
	REDACTED_PREFIX = "***REDACTED*** Hash: "
)

Variables

var (
	ErrNormalizingFloat       = errors.New("ErrNormalizingFloat")
	ErrUnrecognizedObjectType = errors.New("ErrUnrecognizedObjectType")
	ErrNotImplementedYet      = errors.New("ErrNotImplementedYet")
)

Functions

func CommonJSONHash

func CommonJSONHash(j []byte) ([]byte, error)

Example (Common)

commonJSON(`["foo", "bar"]`)

Output:

32ae896c413cfdc79eec68be9139c86ded8b279238467c216cf2bec4d5f1e4a2

Example (FloatAndInt)

commonJSON(`["foo", {"bar":["baz", null, 1.0, 1.5, 0.0001, 1000.0, 2.0, -23.1234, 2.0]}]`)
// Integers and floats are the same in common JSON
commonJSON(`["foo", {"bar":["baz", null, 1, 1.5, 0.0001, 1000, 2, -23.1234, 2]}]`)

Output:

783a423b094307bcb28d005bc2f026ff44204442ef3513585e7e73b66e3c2213
783a423b094307bcb28d005bc2f026ff44204442ef3513585e7e73b66e3c2213

Example (KeyChange)

commonJSON(`["foo", {"b4r":["baz", null, 1, 1.5, 0.0001, 1000, 2, -23.1234, 2]}]`)

Output:

7e01f8b45da35386e4f9531ff1678147a215b8d2b1d047e690fd9ade6151e431

Example (KeyOrderIndependence)

commonJSON(`{"k1":"v1","k2":"v2","k3":"v3"}`)
commonJSON(`{"k2":"v2","k1":"v1","k3":"v3"}`)

Output:

ddd65f1f7568269a30df7cafc26044537dc2f02a1a0d830da61762fc3e687057
ddd65f1f7568269a30df7cafc26044537dc2f02a1a0d830da61762fc3e687057

func Filtered

func Filtered(o interface{}, allowed string) (interface{}, error)

Filter a previously redacted object.

Format of allowed is: expr(,expr)* Format of expr is: ident(/ident)* Format if ident is: either * or not a comma or /

func ObjectHash

func ObjectHash(o interface{}) ([]byte, error)

Example (ComplexSet)

o := Set{`foo`, 23.6, Set{Set{}}, Set{Set{1}}}
objectHash(o)

Output:

3773b0a5283f91243a304d2bb0adb653564573bc5301aa8bb63156266ea5d398

Example (ComplexSetRepeated)

o := Set{`foo`, 23.6, Set{Set{}}, Set{Set{1}}, Set{Set{}}}
objectHash(o)

Output:

3773b0a5283f91243a304d2bb0adb653564573bc5301aa8bb63156266ea5d398

Example (Json)

// Same as equivalent JSON object
o := []any{`foo`, `bar`}
objectHash(o)

Output:

32ae896c413cfdc79eec68be9139c86ded8b279238467c216cf2bec4d5f1e4a2

Example (Json2)

// Same as equivalent _Python_ JSON object
o := []any{`foo`, map[string]any{`bar`: []any{`baz`, nil, 1, 1.5, 0.0001, 1000, 2, -23.1234, 2}}}
objectHash(o)
// Same as equivalent Common JSON object
o = []any{`foo`, map[string]any{`bar`: []any{`baz`, nil, 1.0, 1.5, 0.0001, 1000.0, 2.0, -23.1234, 2.0}}}
objectHash(o)

Output:

726e7ae9e3fadf8a2228bf33e505a63df8db1638fa4f21429673d387dbd1c52a
783a423b094307bcb28d005bc2f026ff44204442ef3513585e7e73b66e3c2213

Example (Set)

o := map[string]any{`thing1`: map[string]any{`thing2`: Set{1, 2, `s`}}, `thing3`: 1234.567}
objectHash(o)

Output:

618cf0582d2e716a70e99c2f3079d74892fec335e3982eb926835967cb0c246c

func ObjectHashWithRedaction

func ObjectHashWithRedaction(o interface{}, redPref string) ([]byte, error)

func ObjectHashWithStdRedaction

func ObjectHashWithStdRedaction(o interface{}) ([]byte, error)

func RedactWithRedaction

func RedactWithRedaction(i interface{}, redPref string) (string, error)

func RedactWithStdRedaction

func RedactWithStdRedaction(i interface{}) (string, error)

func Redactable

func Redactable(o interface{}) (interface{}, error)

func RedactableIt

func RedactableIt(p interface{}) (interface{}, error)

func Unredactable

func Unredactable(o interface{}, redPref string) (interface{}, error)

func UnredactableWithStdPrefix

func UnredactableWithStdPrefix(o interface{}) (interface{}, error)

* Unredact stuff

Types

type Filterer

type Filterer map[string]Filterer

func CreateFilterer

func CreateFilterer(allowed string) *Filterer

func (*Filterer) IsAllowed

func (self *Filterer) IsAllowed(path []string) bool

type Set

type Set []interface{}

FIXME: if What You Hash Is What You Get, then this needs to be safe to use as a set. Note: not actually safe to use as a set