seekret

package module
v0.0.0-...-d09b4f8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2017 License: Apache-2.0 Imports: 10 Imported by: 3

README

|Documentation Status|

=======
seekret
=======

Go library and command line to seek for secrets on various sources.


************
Command Line
************

Description
===========

``seekret`` inspect different sources (files into a directory or git 
repositories) to seek for secrets. It can be used to prevent that secrets are
published in exposed locations.


Installing seekret
==================

``seekret`` can be directly installed by using go get.

::

    go get github.com/18F/seekret/cmd/seekret


The requirements for a success installation are:

 * pkg-config
 * golang >= 1.6
 * libgit2 >= 2.23


Usage
=====

General Options
~~~~~~~~~~~~~~~

::

    NAME:
       seekret - seek for secrets on various sources.   

    USAGE:
       seekret [global options] command [command options] [arguments...]
       
    VERSION:
       0.0.1
       
    AUTHOR(S):
       Albert Puigsech Galicia <albert@puigsech.com> 
       
    COMMANDS:
       seek:
         git    seek for seecrets on a git repository.
         dir    seek for seecrets on a directory.   

    GLOBAL OPTIONS:
       --exception FILE, -x FILE    load exceptions from FILE.
       --rules PATH         PATH with rules. [$SEEKRET_RULES_PATH] 
       --format value, -f value specify the output format. (default: "human")
       --known FILE, -k FILE  load known secrets from FILE.
       --workers value, -w value  number of workers used for the inspection (default: 4)
       --help, -h           show help
       --version, -v        print the version


``-x, --exception``

``--rules``

``-f, --format``

``-k, --known``

``-w, --workers``


Options for Git
~~~~~~~~~~~~~~~

::

    NAME:
       seekret git - seek for seecrets on a git repository.

    USAGE:
       seekret git [command options] [arguments...]

    CATEGORY:
       seek

    OPTIONS:
       --count value, -c value  (default: 0)
   

``-c, --count``


Options for Dir
~~~~~~~~~~~~~~~

::

    NAME:
       seekret dir - seek for seecrets on a directory.  

    USAGE:
       seekret dir [command options] [arguments...] 

    CATEGORY:
       seek 

    OPTIONS:
       --recursive, -r  
       --hidden


``-r, --recursive``

``-h, --hidden``



Examples
========

Scan all files from all commits in a local repo::

    seekret git /path/to/repo

Scan all files from all commits in a remote repo::

    seekret git http://github.com/18F/seekret-exposed

Scan all files from the last commit in a local repo::

    seekret git --count 1 /path/to/repo

Scan all files (including hidden) in a local folder::

    seekret dir --recursive --hidden /path/to/dir


Hands-On
========

The repository seekret-secrets is prepare to test seekret, and can be used to
perform the following hands-on examples:

1. Inspect remote git repository::

    seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ git https://github.com/18F/seekret-secrets.git 

2. Inspect local got repository::

    git clone https://github.com/18F/seekret-secrets.git /tmp/seekret-secrets
    seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ git /tmp/seekret-secrets

3. Inspect only the last 2 commits::

    seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ git -c 2 /tmp/seekret-secrets

4. Inspect with exceptions::

    seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ -x /tmp/seekret-secrets/.exception_1 git /tmp/seekret-secrets


*******
Library
*******

Importing seekret Library
=========================

::

    import seekret "github.com/18F/seekret/lib"


Init Seekret context
====================

::

    s := seekret.NewSeekret()


Loading Rules
=============

::

    s.LoadRulesFromPath("/path/to/main/rues:/path/to/other/rules:/path/to/more/rules")

::

    s.LoadRulesFromDir("/path/to/rules")


::

    s.LoadRulesFromFile("/path/to/file.rule")


Loading Objects
===============

::

    opts := map[string]interface{} {
      "hidden": true,
      "recursive": false,
    }
    s.LoadObjects("dir", "/path/to/inspect", opts)


::

    opts := map[string]interface{} {
      "count": 10,
    }
    s.LoadObjects("dir", "/repo/to/inspect", opts)


Loading Exceptions
==================

::

    s.LoadExceptionsFromFile("/path/to/exceptions/file")



Inspect
=======

::

    s.Inspect(5)



Get Inspect Results
===================

::

    secretsList := s.ListSecrets()



*****
Rules
*****

Secret identification is performed by using a set of rules specified on the
rules files. Those files, with '.rule' extension are defined by using YAML 
following this format: 

::

    rulename:
      match: [regexp]
      unmatch:
        - [regexp]
        - [regexp]
        - ...

For the contents of a file is considered a secret, it must comply with the
'match' regexp and not comply ANY of the 'unmatch' reg rule and comply match 
ANY of the unmatch.


**********
Exceptions
**********

Exceptions determine conditions under which content should not be considered
a secret. The exceptions are specified by using a YAML file that follows this
format:

::
    
      ...
    -
      rule: [rulename]
      object: [regexp]
      line: [linenumber]
      content: [regexp]
    -
      ...


The conditions are optional, so it is not necessary to specify them all, but
for a content deemed exception must meet all the specified conditions.

The meaning of the various conditions explained:

``rule``
Contains the name of the rule.

``object``
Contains a regexp that should match the object name (usually the filename).

``line``
Contains the line number into the object.

``content``
Contains a regexp that should match the content.



.. |Documentation Status| image:: https://godoc.org/github.com/18F/seekret?status.svg
   :target: https://godoc.org/github.com/18F/seekret
   :width: 88px
   :height: 20px

Documentation

Overview

Package seekret provides a framework to create tools to inspect information looking for sensitive information like passwords, tokens, private keys, certificates, etc.

Basics

The current trend of automation of all things and de DevOps culture are very beneficial for efficiency but also come with several problems, being one of them the secret provisioning. Bootstrapping secrets into systems and applications may be complicated and sometimes the straightforward way is to store them into a insecure storage, like github repository, embedded into an artifact or system image, etc. That means that an AWS secret_key end up into a Github repository.

Seekret is an extensible framework that gelps in creating tools for detecting secrets on different sources. The secrets to detect are defined by a set of rules that can help detect passwords, tokens, private keys, certificates, etc.

Tools Using Seekret

Seekret is extensible and can cover various use cases. Below there are some tools that uses seekret:

git-seekret: https://github.com/18F/git-seekret

  Git module that uses local hooks to help develepers to prevent leaking
  sensitive information in a commit.

Using It

Seekret API is very simple and easy to use. This section shows some snippets of code that shows the basic operations you can do with it.

The first thing to be done is to create a new Seekret context:

s := seekret.NewSeekret()

Then the rules must to be loaded. They can be loaded from a path definition, a directory or a single file:

s.LoadRulesFromPath("/path/to/main/rues:/path/to/other/rules:/path/to/more/rules")

s.LoadRulesFromDir("/path/to/rules")

s.LoadRulesFromFile("/path/to/file.rule")

Optionally, exceptions (or false positives) can also be loaded from a file:

s.LoadExceptionsFromFile("/path/to/exceptions/file")

After that, must be loaded the objects to be inspected searching for secrets.

	opts := map[string]interface{} {
  		// Loading options.
	}
	s.LoadObjects(sourceType, source, opts)

sourceType is an interface that implements the interface shown below. We offer sourceType's for Directories and Git Repositories, but you are able to extend it by creating your own.

type SourceType interface {
	LoadObjects(source string, opt LoadOptions) ([]models.Object, error)
}

Currently, there are the following different sources supported:

Directories (and files): https://github.com/18F/seekret/sources/dir

  Load all files contained in a directory (and its sub-directories).

Git Repositories: https://github.com/18F/seekret/sources/git

  Load git objects from commits or staging area.

Having all the rules, exceptions and objects loaded into the contects, it's possible to start the inspection with the following code:

s.Inspect(Nworkers)

Nworkers is an integuer that specify the number of goroutines used on the inspection. The recommended value is runtime.NumCPU().

Finally, it is possible to obtain the list of secrets located and do something with them:

secretsList := s.ListSecrets()
for secret := range secretsList {
	// Do something
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DefaultRulesPath

func DefaultRulesPath() string

DefaultRulesPath return the default PATH that contains rules.

Types

type LoadOptions

type LoadOptions map[string]interface{}

type Seekret

type Seekret struct {
	// contains filtered or unexported fields
}

Seekret contains a seekret context and exposes the API to manipulate it.

func NewSeekret

func NewSeekret() *Seekret

NewSeekret returns a new seekret context.

func (*Seekret) AddException

func (s *Seekret) AddException(exception models.Exception)

AddException adds a new exception into the context.

func (*Seekret) AddRule

func (s *Seekret) AddRule(rule models.Rule, enabled bool)

AddRule adds a new rule into the context.

func (*Seekret) DisableRule

func (s *Seekret) DisableRule(name string) error

DisableRule disables specific rule.

func (*Seekret) DisableRuleByRegexp

func (s *Seekret) DisableRuleByRegexp(name string) int

DisableRule disables rules that match with a regular expression.

func (*Seekret) EnableRule

func (s *Seekret) EnableRule(name string) error

EnableRule enables specific rule.

func (*Seekret) EnableRuleByRegexp

func (s *Seekret) EnableRuleByRegexp(name string) int

EnableRule enables rules that match with a regular expression.

func (*Seekret) GroupObjectsByMetadata

func (s *Seekret) GroupObjectsByMetadata(k string) map[string][]models.Object

GroupObjectsByMetadata returns a map with all objects grouped by specific metadata key.

func (*Seekret) GroupObjectsByPrimaryKeyHash

func (s *Seekret) GroupObjectsByPrimaryKeyHash() map[string][]models.Object

GroupObjectsByPrimaryKeyHash returns a map with all objects grouped by the primary key hash, that is calculated from all metadata keys with the primary attribute. All returned objects could have the same content, even if are not the same.

func (*Seekret) Inspect

func (s *Seekret) Inspect(Nworkers int)

Inspect executes the inspection into all loaded objects, by checking all rules and exceptions loaded.

func (*Seekret) ListRules

func (s *Seekret) ListRules() []models.Rule

ListRules return an array with all loaded rules.

func (*Seekret) ListSecrets

func (s *Seekret) ListSecrets() []models.Secret

ListSecrets return an array with all found secrets after the inspection.

func (*Seekret) LoadExceptionsFromFile

func (s *Seekret) LoadExceptionsFromFile(file string) error

LoadExceptionsFromFile loads exceptions from a YAML file.

func (*Seekret) LoadObjects

func (s *Seekret) LoadObjects(st SourceType, source string, opt LoadOptions) error

LoadObjects loads objects form an specific source. It can load objects from different source types, that are implemented following the SourceType interface.

func (*Seekret) LoadRulesFromDir

func (s *Seekret) LoadRulesFromDir(dir string, defaulEnabled bool) error

LoadRulesFromFile loads rules from all YAML files inside a directory.

func (*Seekret) LoadRulesFromFile

func (s *Seekret) LoadRulesFromFile(file string, defaulEnabled bool) error

LoadRulesFromFile loads rules from a YAML file.

func (*Seekret) LoadRulesFromPath

func (s *Seekret) LoadRulesFromPath(path string, defaulEnabled bool) error

LoadRulesFromFile loads rules from all YAML files inside different directories separated by ':'.

type SourceType

type SourceType interface {
	LoadObjects(source string, opt LoadOptions) ([]models.Object, error)
}

Directories

Path Synopsis
cmd
sources
dir
git

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL