flagscanner

README

Golang CLI Flags Scanner

The flagscanner Go package contains a scanner for lexing and classifying command line arguments. It is a building block that enables building command-line-flags parsers.

For example:

import (
	"os"

	"github.com/bassosimone/flagscanner"
)

// Construct a scanner recognizing GNU style options.
scanner := &flagscanner.Scanner{
	Prefixes:  []string{"-", "--"},
	Separator: "--",
}

// Lex the command line arguments using the given prefixes and separator
tokens := scanner.Scan(os.Args[1:])

The above example configures GNU style options but we support a wide variety of command-line-flags styles including Go, dig, Windows, and traditional Unix. See example_test.go.

Installation

To add this package as a dependency to your module:

go get github.com/bassosimone/flagscanner

Development

To run the tests:

go test -v .

To measure test coverage:

go test -v -cover .

License

SPDX-License-Identifier: GPL-3.0-or-later

History

Adapted from bassosimone/clip.

Documentation

Overview

Package flagscanner provides low-level tokenization of command-line arguments.

The *Scanner.Scan method breaks command-line arguments into Token based on configurable option prefixes and a separator, allowing higher-level parsers to implement custom parsing logic on top of the tokenized stream.

Token Types

*Scanner.Scan produces these token types:

1. OptionToken: Options started with any configured prefix (e.g., -v, --verbose, +trace)

2. OptionsArgumentsSeparatorToken: Special separator (e.g., -- to stop parsing)

3. PositionalArgumentToken: Everything else (positional arguments)

Option Prefixes

The *Scanner is configured with the option prefixes to use when tokenizing command-line arguments. Prefixes are sorted by length (longest first) to ensure correct tokenization when prefixes overlap (e.g., "-" and "--").

This design allows building parsers for different command-line styles:

1. GNU-style: "-", "--" (e.g., -v, --verbose)

2. Dig-style: "-", "--", "+" (e.g., -v, --verbose, +trace)

3. Windows-style: "/" (e.g., /v, /verbose)

4. Go-style: "-" (e.g., -v, -verbose)

Separator

The *Scanner can be configured to recognize and emit as a token the separator to stop parsing options and treat all remaining arguments as positional.

Example

Given the "--" and "-" option prefixes and the "--" separator, the following command line arguments:

--verbose -k4 -- othercommand -v --trace file.txt

produces the following tokens:

1. OptionToken verbose
2. OptionToken -k4
3. OptionsArgumentsSeparatorToken --
4. PositionalArgumentToken othercommand
5. PositionalArgumentToken -v
6. PositionalArgumentToken --trace
7. PositionalArgumentToken file.txt

Note that everything after the separator becomes a positional argument.

Index

• type OptionToken
  • func (tk OptionToken) Index() int
  • func (tk OptionToken) String() string
• type OptionsArgumentsSeparatorToken
  • func (tk OptionsArgumentsSeparatorToken) Index() int
  • func (tk OptionsArgumentsSeparatorToken) String() string
• type PositionalArgumentToken
  • func (tk PositionalArgumentToken) Index() int
  • func (tk PositionalArgumentToken) String() string
• type Scanner
  • func (sx *Scanner) Scan(args []string) []Token
• type Token

Examples

• Scanner (Dig)
• Scanner (Gnu)
• Scanner (Go)
• Scanner (Unix)

Types

type OptionToken

type OptionToken struct {
	// Idx is the position in the original command line arguments.
	Idx int

	// Prefix is the scanned prefix.
	Prefix string

	// Name is the parsed name.
	Name string
}

OptionToken is a Token containing an option.

func (OptionToken) Index

func (tk OptionToken) Index() int

Index implements Token.

func (OptionToken) String

func (tk OptionToken) String() string

String implements Token.

type OptionsArgumentsSeparatorToken

type OptionsArgumentsSeparatorToken struct {
	// Idx is the position in the original command line arguments.
	Idx int

	// Separator is the parsed separator.
	Separator string
}

OptionsArgumentsSeparatorToken is a Token containing the separator between options and arguments.

func (OptionsArgumentsSeparatorToken) Index

func (tk OptionsArgumentsSeparatorToken) Index() int

Index implements Token.

func (OptionsArgumentsSeparatorToken) String

func (tk OptionsArgumentsSeparatorToken) String() string

String implements Token.

type PositionalArgumentToken

type PositionalArgumentToken struct {
	// Idx is the position in the original command line arguments.
	Idx int

	// Value is the parsed value.
	Value string
}

PositionalArgumentToken is a Token containing a positional argument.

func (PositionalArgumentToken) Index

func (tk PositionalArgumentToken) Index() int

Index implements Token.

func (PositionalArgumentToken) String

func (tk PositionalArgumentToken) String() string

String implements Token.

type Scanner

type Scanner struct {
	// Prefixes contains the prefixes delimiting options.
	//
	// If empty, we don't recognize any prefix.
	Prefixes []string

	// Separator contains the separator between options and arguments.
	//
	// If empty, we don't recognize any separator.
	Separator string
}

Scanner is a command line scanner.

We check for the separator first. Then for option prefixes sorted by length (longest first).

Example (Dig)

ExampleScanner_dig demonstrates dig command-line parsing style.

Dig style:

• Traditional short options with single dash: -v, -f

• Long options with double dash: --verbose, --file

• Plus options for dig-specific features: +trace, +short, +noall

• Plus options are treated as long options (no bundling)

• Options with arguments: -f file, +timeout=5, --port=53

• Double dash separator: -- stops option parsing

• Mixed prefix styles for different option categories

package main

import (
	"fmt"

	"github.com/bassosimone/flagscanner"
)

func main() {
	s := &flagscanner.Scanner{
		Prefixes:  []string{"-", "--", "+"}, // Single dash, double dash, and plus prefixes
		Separator: "--",                     // Only double dash separator supported
	}

	args := []string{
		"program", "-v", "+trace", "--verbose", "+short=yes",
		"-f", "config", "--", "remaining", "-args",
	}

	tokens := s.Scan(args[1:])
	for _, token := range tokens {
		fmt.Printf("%#v\n", token)
	}

}

Output:

flagscanner.OptionToken{Idx:0, Prefix:"-", Name:"v"}
flagscanner.OptionToken{Idx:1, Prefix:"+", Name:"trace"}
flagscanner.OptionToken{Idx:2, Prefix:"--", Name:"verbose"}
flagscanner.OptionToken{Idx:3, Prefix:"+", Name:"short=yes"}
flagscanner.OptionToken{Idx:4, Prefix:"-", Name:"f"}
flagscanner.PositionalArgumentToken{Idx:5, Value:"config"}
flagscanner.OptionsArgumentsSeparatorToken{Idx:6, Separator:"--"}
flagscanner.PositionalArgumentToken{Idx:7, Value:"remaining"}
flagscanner.PositionalArgumentToken{Idx:8, Value:"-args"}

Example (Gnu)

ExampleScanner_gnu demonstrates GNU command-line parsing.

GNU style:

• Short options with single dash: -v, -f

• Long options with double dash: --verbose, --file

• Short options can be bundled: -vf equivalent to -v -f

• Options with arguments: -f file, -ffile, --file=name, --file name

• Double dash separator: -- stops option parsing

• Argument permutation (reordering) typically supported at parser level

package main

import (
	"fmt"

	"github.com/bassosimone/flagscanner"
)

func main() {
	s := &flagscanner.Scanner{
		Prefixes:  []string{"-", "--"}, // Single and double dash prefixes
		Separator: "--",
	}

	args := []string{"program", "-v", "--file=config.txt", "-abc", "--", "--an-option", "input.txt"}

	tokens := s.Scan(args[1:])
	for _, token := range tokens {
		fmt.Printf("%#v\n", token)
	}

}

Output:

flagscanner.OptionToken{Idx:0, Prefix:"-", Name:"v"}
flagscanner.OptionToken{Idx:1, Prefix:"--", Name:"file=config.txt"}
flagscanner.OptionToken{Idx:2, Prefix:"-", Name:"abc"}
flagscanner.OptionsArgumentsSeparatorToken{Idx:3, Separator:"--"}
flagscanner.PositionalArgumentToken{Idx:4, Value:"--an-option"}
flagscanner.PositionalArgumentToken{Idx:5, Value:"input.txt"}

Example (Go)

ExampleScanner_go demonstrates Go command-line parsing style.

Go style:

• Short and long options with single dash: -v, -f, -verbose, -file

• No short option bundling: -vf is treated as single option "vf"

• Options with arguments: -file=name, -file name

• Double dash separator: -- stops option parsing

• Simple and consistent: all options use single dash prefix

package main

import (
	"fmt"

	"github.com/bassosimone/flagscanner"
)

func main() {
	s := &flagscanner.Scanner{
		Prefixes:  []string{"-"}, // Go uses single dash for all options
		Separator: "--",
	}

	args := []string{"program", "-v", "-file=config.txt", "-verbose", "-debug", "input.txt", "--", "extra"}

	tokens := s.Scan(args[1:])
	for _, token := range tokens {
		fmt.Printf("%#v\n", token)
	}

}

Output:

flagscanner.OptionToken{Idx:0, Prefix:"-", Name:"v"}
flagscanner.OptionToken{Idx:1, Prefix:"-", Name:"file=config.txt"}
flagscanner.OptionToken{Idx:2, Prefix:"-", Name:"verbose"}
flagscanner.OptionToken{Idx:3, Prefix:"-", Name:"debug"}
flagscanner.PositionalArgumentToken{Idx:4, Value:"input.txt"}
flagscanner.OptionsArgumentsSeparatorToken{Idx:5, Separator:"--"}
flagscanner.PositionalArgumentToken{Idx:6, Value:"extra"}

Example (Unix)

ExampleScanner_unix demonstrates traditional UNIX command-line parsing.

Traditional UNIX style:

• Only single-dash short options: -v, -f

• No long options (--verbose not supported)

• Short options can be bundled: -vf equivalent to -v -f

• Options with arguments: -f file or -ffile

• No special separators (-- not recognized)

package main

import (
	"fmt"

	"github.com/bassosimone/flagscanner"
)

func main() {
	s := &flagscanner.Scanner{
		Prefixes:  []string{"-"}, // Only single-dash options in traditional UNIX
		Separator: "",            // No separators in traditional UNIX
	}

	args := []string{"program", "-v", "-f", "file.txt", "-abc", "input.txt"}

	tokens := s.Scan(args[1:])
	for _, token := range tokens {
		fmt.Printf("%#v\n", token)
	}

}

Output:

flagscanner.OptionToken{Idx:0, Prefix:"-", Name:"v"}
flagscanner.OptionToken{Idx:1, Prefix:"-", Name:"f"}
flagscanner.PositionalArgumentToken{Idx:2, Value:"file.txt"}
flagscanner.OptionToken{Idx:3, Prefix:"-", Name:"abc"}
flagscanner.PositionalArgumentToken{Idx:4, Value:"input.txt"}

func (*Scanner) Scan

func (sx *Scanner) Scan(args []string) []Token

Scan scans the command line arguments and returns a list of Token.

The args MUST NOT include the program name as the first argument.

This method does not mutate the *Scanner and is safe to call concurrently.

type Token

type Token interface {
	// Index returns the token index.
	Index() int

	// String returns the string representation of the token.
	String() string
}

Token is a token lexed by *Scanner.Scan.