ctystrings

Documentation

Overview

Package ctystrings is a collection of string manipulation utilities which intend to help application developers implement string-manipulation functionality in a way that respects the cty model of strings, even when they are working in the realm of Go strings.

cty strings are, internally, NFC-normalized as defined in Unicode Standard Annex #15 and encoded as UTF-8.

When working with [cty.Value] of string type cty manages this automatically as an implementation detail, but when applications call [Value.AsString] they will receive a value that has been subjected to that normalization, and so may need to take that normalization into account when manipulating the resulting string or comparing it with other Go strings that did not originate in a [cty.Value].

Although the core representation of [cty.String] only considers whole strings, it's also conventional in other locations such as the standard library functions to consider strings as being sequences of grapheme clusters as defined by Unicode Standard Annex #29, which adds further rules about combining multiple consecutive codepoints together into a single user-percieved character. Functions that work with substrings should always use grapheme clusters as their smallest unit of splitting strings, and never break strings in the middle of a grapheme cluster. The functions in this package respect that convention unless otherwise stated in their documentation.

Index

• func Normalize(str string) string
• func SafeKnownPrefix(prefix string) string

Functions

func Normalize

func Normalize(str string) string

Normalize applies NFC normalization to the given string, returning the transformed string.

This function achieves the same effect as wrapping a string in a value using [cty.StringVal] and then unwrapping it again using [Value.AsString].

func SafeKnownPrefix

func SafeKnownPrefix(prefix string) string

SafeKnownPrefix takes a string intended to represent a known prefix of another string and modifies it so that it would be safe to use with byte-based prefix matching against another NFC-normalized string. It also takes into account grapheme cluster boundaries and trims off any suffix that could potentially be an incomplete grapheme cluster.

Specifically, SafeKnownPrefix first applies NFC normalization to the prefix and then trims off one or more characters from the end of the string which could potentially be transformed into a different character if another string were appended to it. For example, a trailing latin letter will typically be trimmed because appending a combining diacritic mark would transform it into a different character.

This transformation is important whenever the remainder of the string is arbitrary user input not directly controlled by the application. If an application can guarantee that the remainder of the string will not begin with combining marks then it is safe to instead just normalize the prefix string with Normalize.

Note that this function only takes into account normalization boundaries and does _not_ take into account grapheme cluster boundaries as defined by Unicode Standard Annex #29.