script

package module
v0.24.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 22, 2025 License: MIT Imports: 23 Imported by: 184

README

Go Reference Go Report Card Mentioned in Awesome Go CI Audit

import "github.com/bitfield/script"

Magical gopher logo

What is script?

script is a Go library for doing the kind of tasks that shell scripts are good at: reading files, executing subprocesses, counting lines, matching strings, and so on.

Why shouldn't it be as easy to write system administration programs in Go as it is in a typical shell? script aims to make it just that easy.

Shell scripts often compose a sequence of operations on a stream of data (a pipeline). This is how script works, too.

This is one absolutely superb API design. Taking inspiration from shell pipes and turning it into a Go library with syntax this clean is really impressive.
—Simon Willison

Read more: Scripting with Go

Quick start: Unix equivalents

If you're already familiar with shell scripting and the Unix toolset, here is a rough guide to the equivalent script operation for each listed Unix command.

Unix / shell script equivalent
(any program name) Exec
[ -f FILE ] IfExists
> WriteFile
>> AppendFile
$* Args
base64 DecodeBase64 / EncodeBase64
basename Basename
cat File / Concat
curl Do / Get / Post
cut Column
dirname Dirname
echo Echo
find FindFiles
grep Match / MatchRegexp
grep -v Reject / RejectRegexp
head First
jq JQ
ls ListFiles
sed Replace / ReplaceRegexp
sha256sum Hash / HashSums
tail Last
tee Tee
uniq -c Freq
wc -l CountLines
xargs ExecForEach

Some examples

Let's see some simple examples. Suppose you want to read the contents of a file as a string:

contents, err := script.File("test.txt").String()

That looks straightforward enough, but suppose you now want to count the lines in that file.

numLines, err := script.File("test.txt").CountLines()

For something a bit more challenging, let's try counting the number of lines in the file that match the string Error:

numErrors, err := script.File("test.txt").Match("Error").CountLines()

But what if, instead of reading a specific file, we want to simply pipe input into this program, and have it output only matching lines (like grep)?

script.Stdin().Match("Error").Stdout()

Just for fun, let's filter all the results through some arbitrary Go function:

script.Stdin().Match("Error").FilterLine(strings.ToUpper).Stdout()

That was almost too easy! So let's pass in a list of files on the command line, and have our program read them all in sequence and output the matching lines:

script.Args().Concat().Match("Error").Stdout()

Maybe we're only interested in the first 10 matches. No problem:

script.Args().Concat().Match("Error").First(10).Stdout()

What's that? You want to append that output to a file instead of printing it to the terminal? You've got some attitude, mister. But okay:

script.Args().Concat().Match("Error").First(10).AppendFile("/var/log/errors.txt")

And if we'd like to send the output to the terminal as well as to the file, we can do that:

script.Echo("data").Tee().AppendFile("data.txt")

We're not limited to getting data only from files or standard input. We can get it from HTTP requests too:

script.Get("https://wttr.in/London?format=3").Stdout()
// Output:
// London: 🌦   +13°C

That's great for simple GET requests, but suppose we want to send some data in the body of a POST request, for example. Here's how that works:

script.Echo(data).Post(URL).Stdout()

If we need to customise the HTTP behaviour in some way, such as using our own HTTP client, we can do that:

script.NewPipe().WithHTTPClient(&http.Client{
	Timeout: 10 * time.Second,
}).Get("https://example.com").Stdout()

Or maybe we need to set some custom header on the request. No problem. We can just create the request in the usual way, and set it up however we want. Then we pass it to Do, which will actually perform the request:

req, err := http.NewRequest(http.MethodGet, "http://example.com", nil)
req.Header.Add("Authorization", "Bearer "+token)
script.Do(req).Stdout()

The HTTP server could return some non-okay response, though; for example, “404 Not Found”. So what happens then?

In general, when any pipe stage (such as Do) encounters an error, it produces no output to subsequent stages. And script treats HTTP response status codes outside the range 200-299 as errors. So the answer for the previous example is that we just won't see any output from this program if the server returns an error response.

Instead, the pipe “remembers” any error that occurs, and we can retrieve it later by calling its Error method, or by using a sink method such as String, which returns an error value along with the result.

Stdout also returns an error, plus the number of bytes successfully written (which we don't care about for this particular case). So we can check that error, which is always a good idea in Go:

_, err := script.Do(req).Stdout()
if err != nil {
	log.Fatal(err)
}

If, as is common, the data we get from an HTTP request is in JSON format, we can use JQ queries to interrogate it:

data, err := script.Do(req).JQ(".[0] | {message: .commit.message, name: .commit.committer.name}").String()

We can also run external programs and get their output:

script.Exec("ping 127.0.0.1").Stdout()

Note that Exec runs the command concurrently: it doesn't wait for the command to complete before returning any output. That's good, because this ping command will run forever (or until we get bored).

Instead, when we read from the pipe using Stdout, we see each line of output as it's produced:

PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.056 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.054 ms
...

In the ping example, we knew the exact arguments we wanted to send the command, and we just needed to run it once. But what if we don't know the arguments yet? We might get them from the user, for example.

We might like to be able to run the external command repeatedly, each time passing it the next line of data from the pipe as an argument. No worries:

script.Args().ExecForEach("ping -c 1 {{.}}").Stdout()

That {{.}} is standard Go template syntax; it'll substitute each line of data from the pipe into the command line before it's executed. You can write as fancy a Go template expression as you want here (but this simple example probably covers most use cases).

If there isn't a built-in operation that does what we want, we can just write our own, using Filter:

script.Echo("hello world").Filter(func (r io.Reader, w io.Writer) error {
	n, err := io.Copy(w, r)
	fmt.Fprintf(w, "\nfiltered %d bytes\n", n)
	return err
}).Stdout()
// Output:
// hello world
// filtered 11 bytes

The func we supply to Filter takes just two parameters: a reader to read from, and a writer to write to. The reader reads the previous stages of the pipe, as you might expect, and anything written to the writer goes to the next stage of the pipe.

If our func returns some error, then, just as with the Do example, the pipe's error status is set, and subsequent stages become a no-op.

Filters run concurrently, so the pipeline can start producing output before the input has been fully read, as it did in the ping example. In fact, most built-in pipe methods, including Exec, are implemented using Filter.

If we want to scan input line by line, we could do that with a Filter function that creates a bufio.Scanner on its input, but we don't need to:

script.Echo("a\nb\nc").FilterScan(func(line string, w io.Writer) {
	fmt.Fprintf(w, "scanned line: %q\n", line)
}).Stdout()
// Output:
// scanned line: "a"
// scanned line: "b"
// scanned line: "c"

And there's more. Much more. Read the docs for full details, and more examples.

A realistic use case

Let's use script to write a program that system administrators might actually need. One thing I often find myself doing is counting the most frequent visitors to a website over a given period of time. Given an Apache log in the Common Log Format like this:

212.205.21.11 - - [30/Jun/2019:17:06:15 +0000] "GET / HTTP/1.1" 200 2028 "https://example.com/ "Mozilla/5.0 (Linux; Android 8.0.0; FIG-LX1 Build/HUAWEIFIG-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.156 Mobile Safari/537.36"

we would like to extract the visitor's IP address (the first column in the logfile), and count the number of times this IP address occurs in the file. Finally, we might like to list the top 10 visitors by frequency. In a shell script we might do something like:

cut -d' ' -f 1 access.log |sort |uniq -c |sort -rn |head

There's a lot going on there, and it's pleasing to find that the equivalent script program is quite brief:

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Stdin().Column(1).Freq().First(10).Stdout()
}

Let's try it out with some sample data:

16 176.182.2.191
 7 212.205.21.11
 1 190.253.121.1
 1 90.53.111.17

A script “interpreter”

One of the nice things about shell scripts is that there's no build process: the script file itself is the “executable” (in fact, it's interpreted by the shell). Simon Willison (and GPT-4) contributed this elegant script interpreter, written in bash:

With go-script, you can run script one-liners directly:

cat file.txt | ./goscript.sh -c 'script.Stdin().Column(1).Freq().First(10).Stdout()'

or create .goscript files that you can run using a “shebang” line:

#!/tmp/goscript.sh
script.Stdin().Column(1).Freq().First(10).Stdout()

Documentation

See pkg.go.dev for the full documentation, or read on for a summary.

The script package originated as an exercise in my book The Power of Go: Tools:

Not all software engineering is about writing applications. Developers also need tooling: programs and services to automate everyday tasks like configuring servers and containers, running builds and tests, deploying their applications, and so on. Why shouldn't we be able to use Go for that purpose, too?

script is designed to make it easy to write Go programs that chain together operations into a pipeline, in the same way that shell scripts do, but with the robust type checking and error handling of a real programming language. You can use script to construct the sort of simple one‐off pipelines that would otherwise require the shell, or special‐purpose tools.

So, when plain Go doesn’t provide a convenient way to solve a problem, you yourself can use it to implement a domain-specific “language” that does. In this case, we used Go to provide the language of Unix‐style pipelines. But we could have chosen any architecture we wanted to suit the problem. If Go doesn’t already provide the tool you need, use Go to build that tool, then use it.
—From the book

Sources

These are functions that create a pipe with a given contents:

Source Contents
Args command-line arguments
Do HTTP response
Echo a string
Exec command output
File file contents
FindFiles recursive file listing
Get HTTP response
IfExists do something only if some file exists
ListFiles file listing (including wildcards)
Post HTTP response
Slice slice elements, one per line
Stdin standard input

Modifiers

These are methods on a pipe that change its configuration:

Source Modifies
WithEnv environment for commands
WithError pipe error status
WithHTTPClient client for HTTP requests
WithReader pipe source
WithStderr standard error output stream for command
WithStdout standard output stream for pipe

Filters

Filters are methods on an existing pipe that also return a pipe, allowing you to chain filters indefinitely. The filters modify each line of their input according to the following rules:

Filter Results
Basename removes leading path components from each line, leaving only the filename
Column Nth column of input
Concat contents of multiple files
DecodeBase64 input decoded from base64
Dirname removes filename from each line, leaving only leading path components
Do response to supplied HTTP request
Echo all input replaced by given string
EncodeBase64 input encoded to base64
Exec filtered through external command
ExecForEach execute given command template for each line of input
Filter user-supplied function filtering a reader to a writer
FilterLine user-supplied function filtering each line to a string
FilterScan user-supplied function filtering each line to a writer
First first N lines of input
Freq frequency count of unique input lines, most frequent first
Get response to HTTP GET on supplied URL
HashSums hashes of each listed file
Join replace all newlines with spaces
JQ result of jq query
Last last N lines of input
Match lines matching given string
MatchRegexp lines matching given regexp
Post response to HTTP POST on supplied URL
Reject lines not matching given string
RejectRegexp lines not matching given regexp
Replace matching text replaced with given string
ReplaceRegexp matching text replaced with given string
Tee input copied to supplied writers

Note that filters run concurrently, rather than producing nothing until each stage has fully read its input. This is convenient for executing long-running commands, for example. If you do need to wait for the pipeline to complete, call Wait.

Sinks

Sinks are methods that return some data from a pipe, ending the pipeline and extracting its full contents in a specified way:

Sink Destination Results
AppendFile appended to file, creating if it doesn't exist bytes written, error
Bytes data as []byte, error
Hash hash, error
CountLines number of lines, error
Read given []byte bytes read, error
Slice data as []string, error
Stdout standard output bytes written, error
String data as string, error
Wait error
WriteFile specified file, truncating if it exists bytes written, error

What's new

Version New
0.24.1 JQ accepts JSONLines data
0.24.0 Hash
HashSums
0.23.0 WithEnv
DecodeBase64 / EncodeBase64
Wait returns error
v0.22.0 Tee, WithStderr
v0.21.0 HTTP support: Do, Get, Post
v0.20.0 JQ

Contributing

See the contributor's guide for some helpful tips if you'd like to contribute to the script project.

Gopher image by MariaLetta

Documentation

Overview

Package script aims to make it easy to write shell-type scripts in Go, for general system administration purposes: reading files, counting lines, matching strings, and so on.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Pipe 

type Pipe struct {
	// Reader is the underlying reader.
	Reader ReadAutoCloser
	// contains filtered or unexported fields
}

Pipe represents a pipe object with an associated ReadAutoCloser.

func Args added in v0.7.0 

func Args() *Pipe

Args creates a pipe containing the program's command-line arguments from os.Args, excluding the program name, one per line.

Example 
Output:

func Do added in v0.21.0 

func Do(req *http.Request) *Pipe

Do creates a pipe that makes the HTTP request req and produces the response. See Pipe.Do for how the HTTP response status is interpreted.

Example 
Output:

some data

func Echo added in v0.3.0 

func Echo(s string) *Pipe

Echo creates a pipe containing the string s.

Example 
Output:

Hello, world!

func Exec added in v0.5.0 

func Exec(cmdLine string) *Pipe

Exec creates a pipe that runs cmdLine as an external command and produces its combined output (interleaving standard output and standard error). See Pipe.Exec for error handling details.

Use Pipe.Exec to send the contents of an existing pipe to the command's standard input.

Example (Exit_status_not_zero) 
Output:

1
Example (Exit_status_zero) 
Output:

0
Example (Ok) 
Output:

Hello, world!

func File 

func File(path string) *Pipe

File creates a pipe that reads from the file path.

Example 
Output:

hello world

func FindFiles added in v0.16.0 

func FindFiles(dir string) *Pipe

FindFiles creates a pipe listing all the files in the directory dir and its subdirectories recursively, one per line, like Unix find(1). Errors are ignored unless no files are found (in which case the pipe's error status will be set to the last error encountered).

Each line of the output consists of a slash-separated path, starting with the initial directory. For example, if the directory looks like this: 

test/
        1.txt
        2.txt

the pipe's output will be: 

test/1.txt
test/2.txt
Example 
Output:

testdata/multiple_files_with_subdirectory/1.txt
testdata/multiple_files_with_subdirectory/2.txt
testdata/multiple_files_with_subdirectory/3.tar.zip
testdata/multiple_files_with_subdirectory/dir/.hidden
testdata/multiple_files_with_subdirectory/dir/1.txt
testdata/multiple_files_with_subdirectory/dir/2.txt

func Get added in v0.21.0 

func Get(url string) *Pipe

Get creates a pipe that makes an HTTP GET request to url, and produces the response. See Pipe.Do for how the HTTP response status is interpreted.

Example 
Output:

some data

func IfExists added in v0.14.0 

func IfExists(path string) *Pipe

IfExists tests whether path exists, and creates a pipe whose error status reflects the result. If the file doesn't exist, the pipe's error status will be set, and if the file does exist, the pipe will have no error status. This can be used to do some operation only if a given file exists: 

IfExists("/foo/bar").Exec("/usr/bin/something")
Example (Exec) 
Output:

hello
Example (False) 
Output:
Example (NoExec) 
Output:
Example (True) 
Output:

found it

func ListFiles added in v0.11.0 

func ListFiles(path string) *Pipe

ListFiles creates a pipe containing the files or directories specified by path, one per line. path can be a glob expression, as for filepath.Match. For example: 

ListFiles("/data/*").Stdout()

ListFiles does not recurse into subdirectories; use FindFiles instead.

Example 
Output:

testdata/multiple_files_with_subdirectory/1.txt
testdata/multiple_files_with_subdirectory/2.txt
testdata/multiple_files_with_subdirectory/3.tar.zip
testdata/multiple_files_with_subdirectory/dir

func NewPipe 

func NewPipe() *Pipe

NewPipe creates a new pipe with an empty reader (use Pipe.WithReader to attach another reader to it).

func Post added in v0.21.0 

func Post(url string) *Pipe

Post creates a pipe that makes an HTTP POST request to url, with an empty body, and produces the response. See Pipe.Do for how the HTTP response status is interpreted.

func Slice added in v0.11.0 

func Slice(s []string) *Pipe

Slice creates a pipe containing each element of s, one per line. If s is empty or nil, then the pipe is empty.

Example 
Output:

1
2
3

func Stdin added in v0.6.0 

func Stdin() *Pipe

Stdin creates a pipe that reads from os.Stdin.

func (*Pipe) AppendFile added in v0.4.0 

func (p *Pipe) AppendFile(path string) (int64, error)

AppendFile appends the contents of the pipe to the file path, creating it if necessary, and returns the number of bytes successfully written, or an error.

func (*Pipe) Basename added in v0.14.0 

func (p *Pipe) Basename() *Pipe

Basename reads paths from the pipe, one per line, and removes any leading directory components from each. So, for example, /usr/local/bin/foo would become just foo. This is the complementary operation to Pipe.Dirname.

If any line is empty, Basename will transform it to a single dot. Trailing slashes are removed. The behaviour of Basename is the same as filepath.Base (not by coincidence).

Example 
Output:

.
/
root
example.php
tmp
filters
Program Files

func (*Pipe) Bytes added in v0.8.0 

func (p *Pipe) Bytes() ([]byte, error)

Bytes returns the contents of the pipe as a []byte, or an error.

Example 
Output:

[104 101 108 108 111]

func (*Pipe) Close 

func (p *Pipe) Close() error

Close closes the pipe's associated reader. This is a no-op if the reader is not an io.Closer.

func (*Pipe) Column added in v0.9.0 

func (p *Pipe) Column(col int) *Pipe

Column produces column col of each line of input, where the first column is column 1, and columns are delimited by Unicode whitespace. Lines containing fewer than col columns will be skipped.

Example 
Output:

PID
1
50
51

func (*Pipe) Concat added in v0.8.0 

func (p *Pipe) Concat() *Pipe

Concat reads paths from the pipe, one per line, and produces the contents of all the corresponding files in sequence. If there are any errors (for example, non-existent files), these will be ignored, execution will continue, and the pipe's error status will not be set.

This makes it convenient to write programs that take a list of paths on the command line. For example: 

script.Args().Concat().Stdout()

The list of paths could also come from a file: 

script.File("filelist.txt").Concat()

Or from the output of a command: 

script.Exec("ls /var/app/config/").Concat().Stdout()

Each input file will be closed once it has been fully read. If any of the files can't be opened or read, Concat will simply skip these and carry on, without setting the pipe's error status. This mimics the behaviour of Unix cat(1).

Example 
Output:

This is the first line in the file.
Hello, world.
This is another line in the file.
hello world

func (*Pipe) CountLines 

func (p *Pipe) CountLines() (lines int, err error)

CountLines returns the number of lines of input, or an error.

Example 
Output:

3

func (*Pipe) DecodeBase64 added in v0.23.0 

func (p *Pipe) DecodeBase64() *Pipe

DecodeBase64 produces the string represented by the base64 encoded input.

Example 
Output:

Hello, world!

func (*Pipe) Dirname added in v0.14.0 

func (p *Pipe) Dirname() *Pipe

Dirname reads paths from the pipe, one per line, and produces only the parent directories of each path. For example, /usr/local/bin/foo would become just /usr/local/bin. This is the complementary operation to Pipe.Basename.

If a line is empty, Dirname will transform it to a single dot. Trailing slashes are removed, unless Dirname returns the root folder. Otherwise, the behaviour of Dirname is the same as filepath.Dir (not by coincidence).

Example 
Output:

.
/
/
/tmp
/var
./src
C:

func (*Pipe) Do added in v0.21.0 

func (p *Pipe) Do(req *http.Request) *Pipe

Do performs the HTTP request req using the pipe's configured HTTP client, as set by Pipe.WithHTTPClient, or http.DefaultClient otherwise. The response body is streamed concurrently to the pipe's output. If the response status is anything other than HTTP 200-299, the pipe's error status is set.

Example 
Output:

You said: hello

func (*Pipe) EachLine deprecated added in v0.3.0 

func (p *Pipe) EachLine(process func(string, *strings.Builder)) *Pipe

EachLine calls the function process on each line of input, passing it the line as a string, and a *strings.Builder to write its output to.

Deprecated: use Pipe.FilterLine or Pipe.FilterScan instead, which run concurrently and don't do unnecessary reads on the input.

Example 
Output:

> This is the first line in the file.
> Hello, world.
> This is another line in the file.

func (*Pipe) Echo added in v0.2.0 

func (p *Pipe) Echo(s string) *Pipe

Echo sets the pipe's reader to one that produces the string s, detaching any existing reader without draining or closing it.

Example 
Output:

Hello, world!

func (*Pipe) EncodeBase64 added in v0.23.0 

func (p *Pipe) EncodeBase64() *Pipe

EncodeBase64 produces the base64 encoding of the input.

Example 
Output:

SGVsbG8sIHdvcmxkIQ==

func (*Pipe) Error 

func (p *Pipe) Error() error

Error returns any error present on the pipe, or nil otherwise. Error is not a sink and does not wait until the pipe reaches completion. To wait for completion before returning the error, see Pipe.Wait.

func (*Pipe) Exec added in v0.5.0 

func (p *Pipe) Exec(cmdLine string) *Pipe

Exec runs cmdLine as an external command, sending it the contents of the pipe as input, and produces the command's standard output (see below for error output). The effect of this is to filter the contents of the pipe through the external command.

Environment

The command inherits the current process's environment, optionally modified by Pipe.WithEnv.

Error handling

If the command had a non-zero exit status, the pipe's error status will also be set to the string “exit status X”, where X is the integer exit status. Even in the event of a non-zero exit status, the command's output will still be available in the pipe. This is often helpful for debugging. However, because Pipe.String is a no-op if the pipe's error status is set, if you want output you will need to reset the error status before calling Pipe.String.

If the command writes to its standard error stream, this will also go to the pipe, along with its standard output. However, the standard error text can instead be redirected to a supplied writer, using Pipe.WithStderr.

Example 
Output:

HELLO, WORLD!

func (*Pipe) ExecForEach added in v0.15.0 

func (p *Pipe) ExecForEach(cmdLine string) *Pipe

ExecForEach renders cmdLine as a Go template for each line of input, running the resulting command, and produces the combined output of all these commands in sequence. See Pipe.Exec for details on error handling and environment variables.

This is mostly useful for substituting data into commands using Go template syntax. For example: 

ListFiles("*").ExecForEach("touch {{.}}").Wait()
Example 
Output:

a
b
c

func (*Pipe) ExitStatus added in v0.5.0 

func (p *Pipe) ExitStatus() int

ExitStatus returns the integer exit status of a previous command (for example run by Pipe.Exec). This will be zero unless the pipe's error status is set and the error matches the pattern “exit status %d”.

Example 
Output:

0

func (*Pipe) Filter added in v0.19.0 

func (p *Pipe) Filter(filter func(io.Reader, io.Writer) error) *Pipe

Filter sends the contents of the pipe to the function filter and produces the result. filter takes an io.Reader to read its input from and an io.Writer to write its output to, and returns an error, which will be set on the pipe.

filter runs concurrently, so its goroutine will not exit until the pipe has been fully read. Use Pipe.Wait to wait for all concurrent filters to complete.

Example 
Output:

hello world
filtered 11 bytes

func (*Pipe) FilterLine added in v0.19.0 

func (p *Pipe) FilterLine(filter func(string) string) *Pipe

FilterLine sends the contents of the pipe to the function filter, a line at a time, and produces the result. filter takes each line as a string and returns a string as its output. See Pipe.Filter for concurrency handling.

Example (Stdlib) 
Output:

A
B
C
Example (User) 
Output:

> a
> b
> c

func (*Pipe) FilterScan added in v0.19.0 

func (p *Pipe) FilterScan(filter func(string, io.Writer)) *Pipe

FilterScan sends the contents of the pipe to the function filter, a line at a time, and produces the result. filter takes each line as a string and an io.Writer to write its output to. See Pipe.Filter for concurrency handling.

Example 
Output:

scanned line: "a"
scanned line: "b"
scanned line: "c"

func (*Pipe) First added in v0.9.0 

func (p *Pipe) First(n int) *Pipe

First produces only the first n lines of the pipe's contents, or all the lines if there are less than n. If n is zero or negative, there is no output at all. When n lines have been produced, First stops reading its input and sends EOF to its output.

Example 
Output:

a
b

func (*Pipe) Freq added in v0.9.0 

func (p *Pipe) Freq() *Pipe

Freq produces only the unique lines from the pipe's contents, each prefixed with a frequency count, in descending numerical order (most frequent lines first). Lines with equal frequency will be sorted alphabetically.

For example, we could take a common shell pipeline like this: 

sort input.txt |uniq -c |sort -rn

and replace it with: 

File("input.txt").Freq().Stdout()

Or to get only the ten most common lines: 

File("input.txt").Freq().First(10).Stdout()

Like Unix uniq(1), Freq right-justifies its count values in a column for readability, padding with spaces if necessary.

Example 
Output:

10 apple
 4 banana
 4 orange
 1 kumquat

func (*Pipe) Get added in v0.21.0 

func (p *Pipe) Get(url string) *Pipe

Get makes an HTTP GET request to url, sending the contents of the pipe as the request body, and produces the server's response. See Pipe.Do for how the HTTP response status is interpreted.

Example 
Output:

You said: hello

func (*Pipe) Hash added in v0.24.0 

func (p *Pipe) Hash(hasher hash.Hash) (string, error)

Hash returns the hex-encoded hash of the entire contents of the pipe based on the provided hasher, or an error. To perform hashing on files, see Pipe.HashSums.

Example 
Output:

309ecc489c12d6eb4cc40f50c902f2b4d0ed77ee511a7c7a9bcd3ca86d4cd86f989dd35bc5ff499670da34255b45b0cfd830e81f605dcf7dc5542e93ae9cd76f

func (*Pipe) HashSums added in v0.24.0 

func (p *Pipe) HashSums(hasher hash.Hash) *Pipe

HashSums reads paths from the pipe, one per line, and produces the hex-encoded hash of each corresponding file based on the provided hasher, one per line. Any files that cannot be opened or read will be ignored. To perform hashing on the contents of the pipe, see Pipe.Hash.

Example 
Output:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

func (*Pipe) JQ added in v0.20.0 

func (p *Pipe) JQ(query string) *Pipe

JQ executes query on the pipe's contents (presumed to be valid JSON or JSONLines data), applying the query to each newline-delimited input value and producing results until the first error is encountered. An invalid query or value will set the appropriate error on the pipe.

The exact dialect of JQ supported is that provided by github.com/itchyny/gojq, whose documentation explains the differences between it and standard JQ.

Example 
Output:

"https://github.com/mbarley333/blackjack/releases/download/v0.3.3/blackjack_0.3.3_Darwin_x86_64.tar.gz"

func (*Pipe) Join added in v0.7.0 

func (p *Pipe) Join() *Pipe

Join joins all the lines in the pipe's contents into a single space-separated string, which will always end with a newline.

Example 
Output:

hello world

func (*Pipe) Last added in v0.12.0 

func (p *Pipe) Last(n int) *Pipe

Last produces only the last n lines of the pipe's contents, or all the lines if there are less than n. If n is zero or negative, there is no output at all.

Example 
Output:

b
c

func (*Pipe) Match added in v0.2.0 

func (p *Pipe) Match(s string) *Pipe

Match produces only the input lines that contain the string s.

Example 
Output:

b

func (*Pipe) MatchRegexp added in v0.3.0 

func (p *Pipe) MatchRegexp(re *regexp.Regexp) *Pipe

MatchRegexp produces only the input lines that match the compiled regexp re.

Example 
Output:

world

func (*Pipe) Post added in v0.21.0 

func (p *Pipe) Post(url string) *Pipe

Post makes an HTTP POST request to url, using the contents of the pipe as the request body, and produces the server's response. See Pipe.Do for how the HTTP response status is interpreted.

Example 
Output:

You said: hello

func (*Pipe) Read added in v0.8.1 

func (p *Pipe) Read(b []byte) (int, error)

Read reads up to len(b) bytes from the pipe into b. It returns the number of bytes read and any error encountered. At end of file, or on a nil pipe, Read returns 0, io.EOF.

Example 
Output:

12

func (*Pipe) Reject added in v0.2.0 

func (p *Pipe) Reject(s string) *Pipe

Reject produces only lines that do not contain the string s.

Example 
Output:

a
c

func (*Pipe) RejectRegexp added in v0.3.0 

func (p *Pipe) RejectRegexp(re *regexp.Regexp) *Pipe

RejectRegexp produces only lines that don't match the compiled regexp re.

Example 
Output:

hello

func (*Pipe) Replace added in v0.10.0 

func (p *Pipe) Replace(search, replace string) *Pipe

Replace replaces all occurrences of the string search with the string replace.

Example 
Output:

a
replacement
c

func (*Pipe) ReplaceRegexp added in v0.10.0 

func (p *Pipe) ReplaceRegexp(re *regexp.Regexp, replace string) *Pipe

ReplaceRegexp replaces all matches of the compiled regexp re with the string replace. $x variables in the replace string are interpreted as by [regexp#Regexp.Expand]; for example, $1 represents the text of the first submatch.

Example 
Output:

hello
replacement

func (*Pipe) SHA256Sum added in v0.17.0 

func (p *Pipe) SHA256Sum() (string, error)

SHA256Sum returns the hex-encoded SHA-256 hash of the entire contents of the pipe, or an error. Deprecated: SHA256Sum has been deprecated by Pipe.Hash. To get the SHA-256 hash for the contents of the pipe, call `Hash(sha256.new())`

Example 
Output:

b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9

func (*Pipe) SHA256Sums added in v0.17.0 

func (p *Pipe) SHA256Sums() *Pipe

SHA256Sums reads paths from the pipe, one per line, and produces the hex-encoded SHA-256 hash of each corresponding file, one per line. Any files that cannot be opened or read will be ignored. Deprecated: SHA256Sums has been deprecated by Pipe.HashSums. To get the SHA-256 hash for each file path in the pipe, call `HashSums(sha256.new())`

Example 
Output:

a562c9c95e2ff3403e7ffcd8508c6b54d47d5f251387758d3e63dbaaa8296341

func (*Pipe) SetError 

func (p *Pipe) SetError(err error)

SetError sets the error err on the pipe.

func (*Pipe) Slice added in v0.18.0 

func (p *Pipe) Slice() ([]string, error)

Slice returns the pipe's contents as a slice of strings, one element per line, or an error.

An empty pipe will produce an empty slice. A pipe containing a single empty line (that is, a single \n character) will produce a slice containing the empty string as its single element.

Example 
Output:

[a b c]

func (*Pipe) Stdout added in v0.6.0 

func (p *Pipe) Stdout() (int, error)

Stdout copies the pipe's contents to its configured standard output (using Pipe.WithStdout), or to os.Stdout otherwise, and returns the number of bytes successfully written, together with any error.

Example 
Output:

a
b
c
6

func (*Pipe) String 

func (p *Pipe) String() (string, error)

String returns the pipe's contents as a string, together with any error.

Example 
Output:

hello
world

func (*Pipe) Tee added in v0.22.0 

func (p *Pipe) Tee(writers ...io.Writer) *Pipe

Tee copies the pipe's contents to each of the supplied writers, like Unix tee(1). If no writers are supplied, the default is the pipe's standard output.

Example (Stdout) 
Output:

hello
hello
Example (Writers) 
Output:

hello
hello
hello

func (*Pipe) Wait added in v0.19.0 

func (p *Pipe) Wait() error

Wait reads the pipe to completion and returns any error present on the pipe, or nil otherwise. This is mostly useful for waiting until concurrent filters have completed (see Pipe.Filter).

func (*Pipe) WithEnv added in v0.23.0 

func (p *Pipe) WithEnv(env []string) *Pipe

WithEnv sets the environment for subsequent Pipe.Exec and Pipe.ExecForEach commands to the string slice env, using the same format as os/exec.Cmd.Env. An empty slice unsets all existing environment variables.

func (*Pipe) WithError 

func (p *Pipe) WithError(err error) *Pipe

WithError sets the error err on the pipe.

func (*Pipe) WithHTTPClient added in v0.21.0 

func (p *Pipe) WithHTTPClient(c *http.Client) *Pipe

WithHTTPClient sets the HTTP client c for use with subsequent requests via Pipe.Do, Pipe.Get, or Pipe.Post. For example, to make a request using a client with a timeout: 

NewPipe().WithHTTPClient(&http.Client{
        Timeout: 10 * time.Second,
}).Get("https://example.com").Stdout()

func (*Pipe) WithReader 

func (p *Pipe) WithReader(r io.Reader) *Pipe

WithReader sets the pipe's input reader to r. Once r has been completely read, it will be closed if necessary.

func (*Pipe) WithStderr added in v0.22.0 

func (p *Pipe) WithStderr(w io.Writer) *Pipe

WithStderr sets the standard error output for Pipe.Exec or Pipe.ExecForEach commands to w, instead of the pipe.

Example 
Output:

true

func (*Pipe) WithStdout added in v0.18.2 

func (p *Pipe) WithStdout(w io.Writer) *Pipe

WithStdout sets the pipe's standard output to the writer w, instead of the default os.Stdout.

func (*Pipe) WriteFile added in v0.4.0 

func (p *Pipe) WriteFile(path string) (int64, error)

WriteFile writes the pipe's contents to the file path, truncating it if it exists, and returns the number of bytes successfully written, or an error.

type ReadAutoCloser added in v0.8.0 

type ReadAutoCloser struct {
	// contains filtered or unexported fields
}

ReadAutoCloser wraps an io.ReadCloser so that it will be automatically closed once it has been fully read.

func NewReadAutoCloser added in v0.8.0 

func NewReadAutoCloser(r io.Reader) ReadAutoCloser

NewReadAutoCloser returns a ReadAutoCloser wrapping the reader r.

func (ReadAutoCloser) Close added in v0.8.0 

func (ra ReadAutoCloser) Close() error

Close closes ra's reader, returning any resulting error.

func (ReadAutoCloser) Read added in v0.8.0 

func (ra ReadAutoCloser) Read(b []byte) (n int, err error)

Read reads up to len(b) bytes from ra's reader into b. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF. If end-of-file is reached, the reader will be closed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.