keynuker

package module
v0.0.0-...-b027d78 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 5, 2018 License: Apache-2.0 Imports: 41 Imported by: 0

README 

image:https://img.shields.io/badge/reddit%2Faws%20upvotes-49-ff69b4.svg[link="https://www.reddit.com/r/aws/comments/734lof/keynuker_nuke_aws_keys_accidentally_leaked_to/"] image:https://img.shields.io/badge/license-Apache%202-blue.svg[link=https://www.apache.org/licenses/LICENSE-2.0] 


If you accidentally leak your AWS keys on GitHub, it won't be long before attackers scrape this information and https://web.archive.org/web/20160304044323/https://www.forbes.com/sites/runasandvik/2014/01/14/attackers-scrape-github-for-cloud-service-credentials-hijack-account-to-mine-virtual-currency/#41d040f67cf8:[hijack your account for nefarious purposes].  

KeyNuker scans public activity across all Github users in your Github organization(s) and proactively deletes any AWS keys that are accidentally leaked.  It gets the list of AWS keys to scan by directly connecting to the AWS API.

== 🚁 System Overview

* Polling loop to monitor all AWS keys for users of the AWS account
* Polling loop to monitor Github activity of all users in your github organization(s)
* Reactively nuke any AWS keys detected to have leaked to Github via the AWS API

image::docs/diagrams/architecture.png[Architecture]

== ⏱ System Interaction w/ Timeline

image::docs/diagrams/dataflow.png[Data Flow]

== 🛠 Installing KeyNuker 🔐💥

KeyNuker is packaged as a series of https://github.com/apache/incubator-openwhisk[Apache OpenWhisk] actions, and can run anywhere that OpenWhisk can run.

Get the code:

```
$ go get -u -v -t github.com/tleyden/keynuker
```

Follow the steps in the link:docs/install.adoc[Installation Instructions] to setup OpenWhisk and set the required environment variables, and then run the installation script.

```
$ cd $GOPATH/src/github.com/tleyden/keynuker/
$ python install.py
```

After the installation script completes, you will have several OpenWhisk actions:

```
$ wsk action list
/yourusername_dev/github-user-events-scanner-nuker                     private sequence
/yourusername_dev/fetch-aws-keys-write-doc                             private sequence
etc ...
```

== ✅ Features

. Never has access to AWS Secret Access Keys, only to AWS Access Key IDs.
. Noise-free because it scans actual AWS keys rather than taking a pattern matching approach that produces false positives
. Takes actions rather than sending alerts, since depending on people to respond to alerts might introduce a costly delay in reaction time
. Covers all public github activity of users in your github org(s), since they might leak AWS keys on their personal repos or even 3rd party repos
. Requires minimal IAM Permissions: `iam:DeleteAccessKey`, `iam:ListAccessKeys`, `iam:ListUsers`
. Ultra-low baseline running cost due to serverless architecture
. Lowest common denominator and requires zero effort or workflow change for users

== ⚪️ Roadmap / Goals

. Low-latency round trip between AWS keys leaking and being nuked.  Currently uses polling approach, but the goal is to layer a webhook approach on top in order to lower the latency.
. Scale up to monitoring hundreds of AWS accounts and thousands of github organizations/repos/users.
. Cove as much of the Github API surface area as possible (gists, issue comments, etc)
. Pluggable architecture so that other cloud key providers (Google Cloud Platform, Azure, etc) and other leak sources (BitBucket, Jira, etc) can be added later.


== 🏁 Project status: Early alpha stage

The basic end-to-end functionality is working, except for notifications, but there are still a lot of places where AWS keys can leak to Github and go undetected:

 * Github Gists
 * Other Github API surface area that isn't covered yet


== 📓 Documentation

.Documentation
|===
|Doc |Link

|README (this document)
|link:README.adoc[README]

|Installation guide
|link:docs/install.adoc[Installation Instructions]

|Post-installation verification
|link:docs/verify.adoc[Verify Installation]

|Developer guide
|link:docs/developers.adoc[Developer guide]

|AWS Security Resources
|link:docs/aws_security_resources.adoc[AWS Security Resources]

|===


== 📰 Articles on malicious key scraping

* https://web.archive.org/web/20160304044323/https://www.forbes.com/sites/runasandvik/2014/01/14/attackers-scrape-github-for-cloud-service-credentials-hijack-account-to-mine-virtual-currency/#41d040f67cf8:[Attackers Scrape GitHub For Cloud Service Credentials, Hijack Account To Mine Virtual Currency]
* https://web.archive.org/web/20170111080816/http://www.itnews.com.au/news/aws-urges-developers-to-scrub-github-of-secret-keys-375785[AWS urges developers to scrub GitHub of secret keys]
* https://web.archive.org/web/20170205165621/https://www.theregister.co.uk/2015/01/06/dev_blunder_shows_github_crawling_with_keyslurping_bots/[Dev put AWS keys on Github. Then BAD THINGS happened]


== 📁 Related Projects

* https://github.com/awslabs/git-secrets[Git Secrets] -- Prevents you from committing secrets and credentials into git repositories
* https://github.com/ezekg/git-hound[Git Hound] --  Git plugin that helps prevent sensitive data from being committed into a repository by sniffing potential commits against PCRE regular expressions.
* https://github.com/michenriksen/gitrob[GitRob] -- Reconnaissance tool for GitHub organizations
* http://pre-commit.com/[pre-commit] - A pre-commit framework, that includes the http://pre-commit.com/hooks.html[detect-aws-credentials] plugin
* https://www.reddit.com/r/netsec/comments/5ll7ng/truffle_hog_a_tool_that_searches_entire_commit/[TruffleHog] -- A tool that Searches Entire Commit History in Git Repositories for High Entropy Strings to Find Secrets Accidentally Committed to Version Control
* https://rtyley.github.io/bfg-repo-cleaner/[BFG Repo cleaner] - Removes large or troublesome blobs like git-filter-branch does
* https://aws.amazon.com/blogs/compute/automate-your-it-operations-using-aws-step-functions-and-amazon-cloudwatch-events/[AWS_RISK_CREDENTIALS_EXPOSED CloudWatch Event]

== ⚙ Related Services

* https://www.gitguardian.com/[Git Guardian] -- Scanning service that alerts you of text strings that appear to be API keys (Github and more)
* https://github.com/cloudsploit[CloudSploit] -- Cloud security monitoring service / open source software which has a https://github.com/cloudsploit/scans/issues/10[feature request] very reminiscent of KeyNuker.
* https://evident.io/[Evident.io]
* https://dome9.com/iam-safety/[Dome9]
* https://www.cloudconformity.com[CloudConformity]


== 👀 Related Reddit Discussions

* https://www.reddit.com/r/aws/comments/734lof/keynuker_nuke_aws_keys_accidentally_leaked_to/[reddit/aws: KeyNuker announcement + discussions]
* https://www.reddit.com/r/aws/comments/6pjf7n/we_got_hacked_looking_for_ideas_on_preventative/[reddit/aws: We got hacked. Looking for ideas on preventative measures going forward.]
* https://www.reddit.com/r/aws/comments/6onzgb/what_aws_security_compliances_do_you_guys_have/[reddit/aws: What AWS security compliances do you guys have for your environment?]

== 🔒 Security At Depth

Taking a security-at-depth approach, in addition to running KeyNuker you should also consider the following precautions:

- Limit ec2 actions to only the regions that you use, eg (`"StringEquals": {"ec2:Region": "us-east-1"}`)
- Limit ec2 actions to only the instance types that you use, eg (`"StringLikeIfExists": {"ec2:InstanceType": ["t1.*"]}`)
- Use temporary AWS keys that require MFA
- Minimize chance of AWS keys from ever leaking in the first place using tools such as https://github.com/awslabs/git-secrets[Git Secrets] which can be configured as a pre-commit hook.

Documentation

Index

Constants

View Source 
const (
	MaxPerPage = 100
)

Variables

View Source 
var (
	ArtificialErrorInjectionEnabled = false
)

Functions

func ArtificialErrorInjection 

func ArtificialErrorInjection() bool

func Asset 

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDir 

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy: 

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

func AssetInfo 

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames 

func AssetNames() []string

AssetNames returns the names of the assets.

func CalculateBytesScanned 

func CalculateBytesScanned(logs []string) (int64, error)

Look for log messages with form: 

Scanning 1833 bytes

and extract the number of bytes, and then add them up

func CreateApiHostBaseUrl 

func CreateApiHostBaseUrl(hostname string) (baseUrl *url.URL, err error)

Given a base hostname like "openwhisk.ng.bluemix.net" or "https://openwhisk.ng.bluemix.net:443", return a URL that includes a trailing "/api" in the path. The trailing /api is needed due to https://github.com/apache/incubator-openwhisk-client-go/issues/25

func CreateDBClient 

func CreateDBClient(params ParamsWriteDoc) (db *kivik.DB, err error)

func DeleteDoc 

func DeleteDoc(params ParamsWriteDoc, rev string) (newRev string, err error)

func EventMoreRecent 

func EventMoreRecent(checkpoint, incoming *github.Event) bool

Is the incoming event more recent than the checkpoint event?

func FetchIAMUsers 

func FetchIAMUsers(svc iamiface.IAMAPI) (users []*iam.User, err error)

func GetGithubOrgsFromEnv 

func GetGithubOrgsFromEnv() (githubOrgs []string, err error)

func GetIntegrationGithubApiBaseUrl 

func GetIntegrationGithubApiBaseUrl() string

func GetIntegrationTestAwsCredentials 

func GetIntegrationTestAwsCredentials() (accessKey, secretAccessKey string, err error)

func IntegrationTestsEnabled 

func IntegrationTestsEnabled() bool

func IsKeyNotFoundError 

func IsKeyNotFoundError(err error) bool

func MustAsset 

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func NewMailgunFromEnvironmentVariables 

func NewMailgunFromEnvironmentVariables() (mg mailgun.Mailgun, err error)

func OpenWhiskRecentActivationsStatus 

func OpenWhiskRecentActivationsStatus(maxActivationsToScan int) (keynukerStatus map[string]interface{})

Connect to OpenWhisk API and scan the list of recent activations and look for any failures. If any failures found, return {"status": "failure"}. Otherwise return {"status": "success"}. The idea is that this would be served up by a web action that a monitoring tool could poll and send alerts if any failures occurred.

func RestoreAsset 

func RestoreAsset(dir, name string) error

RestoreAsset restores an asset under the given directory

func RestoreAssets 

func RestoreAssets(dir, name string) error

RestoreAssets restores an asset under the given directory recursively

func ScanActivationsForFailures 

func ScanActivationsForFailures(whiskConfig *whisk.Config, maxActivationsToScan int) (failedActivations []whisk.Activation, err error)

Loop over all activations and return the ones that have a whisk.Result with Success == false. Stop scanning after maxActivationsToScan activations have been scanned

func SendMonitorNotifications 

func SendMonitorNotifications(params ParamsMonitorActivations, activationStatus map[string]interface{}) (deliveryId string, err error)

func SendReportNotifications 

func SendReportNotifications(params ParamsMonitorActivations, report RecentActivationsReportOutput) (deliveryId string, err error)

func SetArtificialErrorInjection 

func SetArtificialErrorInjection(val bool)

func SkipIfIntegrationsTestsNotEnabled 

func SkipIfIntegrationsTestsNotEnabled(t *testing.T)

func WhiskConfigFromEnvironment 

func WhiskConfigFromEnvironment() (config *whisk.Config, err error)

func WhiskConfigFromOwEnvVars 

func WhiskConfigFromOwEnvVars() (config *whisk.Config, err error)

func WhiskPropsMapFromWskConfigFile 

func WhiskPropsMapFromWskConfigFile() (map[string]string, error)

func WriteDocToCloudant 

func WriteDocToCloudant(params ParamsWriteDoc) (interface{}, error)

func WriteDocToDb 

func WriteDocToDb(params ParamsWriteDoc) (interface{}, error)

Write an object specified in params to the underlying database, and return the written object back.

func WriteDocToFauna 

func WriteDocToFauna(params ParamsWriteDoc) (interface{}, error)

Types

type AwsCredentials 

type AwsCredentials struct {

	// The aws access key to connect as.  This only needs permissions to list IAM users and access keys,
	// and delete access keys (in the case they are nuked)
	AwsAccessKeyId string

	// The secret access key corresponding to AwsAccessKeyId
	AwsSecretAccessKey string
}

type DocumentFetchAwsKeys 

type DocumentFetchAwsKeys struct {
	Id string `json:"_id"`

	AccessKeyMetadata []FetchedAwsAccessKey
}

type DocumentGithubUserAggregator 

type DocumentGithubUserAggregator struct {
	Id          string `json:"_id"`
	GithubUsers []*github.User
}

type DocumentNukeLeakedAwsKeys 

type DocumentNukeLeakedAwsKeys struct {
	Id                     string `json:"_id"`
	NukedKeyEvents         []NukedKeyEvent
	GithubEventCheckpoints GithubEventCheckpoints
}

func NukeLeakedAwsKeys 

func NukeLeakedAwsKeys(params ParamsNukeLeakedAwsKeys) (doc DocumentNukeLeakedAwsKeys, err error)

. Connect to AWS . For each leaked key .. Delete the key . Return doc with nuked keys and validated github event checkpoints

type DocumentScanGithubUserEventsForAwsKeys 

type DocumentScanGithubUserEventsForAwsKeys struct {
	Id                     string `json:"_id"`
	LeakedKeyEvents        []LeakedKeyEvent
	GithubEventCheckpoints GithubEventCheckpoints
}

type DocumentWithAwsKeys 

type DocumentWithAwsKeys struct {
	AccessKeyMetadata *json.RawMessage
}

type DocumentWithGithubEventCheckpoints 

type DocumentWithGithubEventCheckpoints struct {
	GithubEventCheckpoints *json.RawMessage
}

type DocumentWithGithubUsers 

type DocumentWithGithubUsers struct {
	GithubUsers *json.RawMessage
}

type DocumentWrapperFetchAwsKeys 

type DocumentWrapperFetchAwsKeys struct {
	// Serialize into a form that the cloudant db adapter expects
	Doc   DocumentFetchAwsKeys `json:"doc"`
	DocId string               `json:"docid"`
}

func FetchAwsKeys 

func FetchAwsKeys(params ParamsFetchAwsKeys) (docWrapper DocumentWrapperFetchAwsKeys, err error)

Look up all the AWS keys associated with the AWS account corresponding to AwsAccessKeyId and return a document suitable for sticking into a Cloudant database.

This is meant to be run in the context of an OpenWhisk Action (see https://tleyden.github.io/blog/2017/07/02/openwhisk-action-sequences/) and so nothing else except the JSON content can be written to standard output.

type DocumentWrapperGithubUserAggregator 

type DocumentWrapperGithubUserAggregator struct {
	// Serialize into a form that the cloudant db adapter expects
	Doc   DocumentGithubUserAggregator `json:"doc"`
	DocId string                       `json:"docid"`
}

func AggregateGithubUsers 

func AggregateGithubUsers(params ParamsGithubUserAggregator) (DocumentWrapperGithubUserAggregator, error)

Given a list of github orgs, aggregate all of the users that belong in the orgs and emit a json to stdout with those users. Intended to be run as an OpenWhisk Action

type DocumentWrapperLookupGithubUsersAwsKeys 

type DocumentWrapperLookupGithubUsersAwsKeys struct {

	// A list of github users
	GithubUsers *json.RawMessage

	// AWS access keys to scan for
	AccessKeyMetadata *json.RawMessage

	// Github event checkpoint which represent the last scanned github event for each known github user
	GithubEventCheckpoints *json.RawMessage
}

func LookupGithubUsersAwsKeys 

func LookupGithubUsersAwsKeys(params ParamsLookupGithubUsersAwsKeys) (docWrapper DocumentWrapperLookupGithubUsersAwsKeys, err error)

type DocumentWrapperPostNukeNotifier 

type DocumentWrapperPostNukeNotifier struct {
	// Serialize into a form that the cloudant db adapter expects
	Doc   ResultPostNukeNotifier `json:"doc"`
	DocId string                 `json:"docid"`
}

type DocumentWrapperScanGithubUserEventsForAwsKeys 

type DocumentWrapperScanGithubUserEventsForAwsKeys struct {
	// Serialize into a form that the cloudant db adapter expects
	Doc   DocumentScanGithubUserEventsForAwsKeys `json:"doc"`
	DocId string                                 `json:"docid"`
}

type EventStack 

type EventStack struct {
	*stack.Stack
}

func NewEventStack 

func NewEventStack() *EventStack

func (*EventStack) Pop 

func (this *EventStack) Pop() *github.Event

func (*EventStack) PopAll 

func (this *EventStack) PopAll() []*github.Event

func (*EventStack) Push 

func (this *EventStack) Push(value *github.Event)

type FetchUserEventsInput 

type FetchUserEventsInput struct {

	// The github username
	Username string

	// For checkpointing purposes, only consider events that are _after_ this timestamp
	SinceEventTimestamp *time.Time

	// For checkpointing purposes.  Ignore events with same ID as checkpoint.  (note: this could
	// eventually replace the time based checkpointing)
	CheckpointID string
}

Input parameters for the github user event fetcher, which include filtering params such as checkpoint filtering

func (FetchUserEventsInput) MatchesCheckpointID 

func (f FetchUserEventsInput) MatchesCheckpointID(event *github.Event) bool

type FetchedAwsAccessKey 

type FetchedAwsAccessKey struct {

	// The ID for this access key.
	AccessKeyId *string `min:"16" type:"string"`

	// The date when the access key was created.
	CreateDate *time.Time `type:"timestamp" timestampFormat:"iso8601"`

	// The status of the access key. Active means the key is valid for API calls;
	// Inactive means it is not.
	Status *string `type:"string" enum:"statusType"`

	// The name of the IAM user that the key is associated with.
	UserName *string `min:"1" type:"string"`

	// The AWS access key used to monitor this AWS account's keys.  Need to track since this same key will need to be used to nuke as well.
	// TODO: this should be the sha1 hash of the key, not the access key itself.  That would keep the access key out of the response json
	MonitorAwsAccessKeyId string
}

This encapsulates all of the fields from iam.AccessKeyMetadata, as well as addding the FetcherAwsAccessKeyId that was used to fetch (and should be used to nuke key if needed)

func FetchAwsKeysTargetAccount 

func FetchAwsKeysTargetAccount(initiatingAwsAccount AwsCredentials, targetAwsAccount TargetAwsAccount) (fetchedAwsKeys []FetchedAwsAccessKey, err error)

func NewFetchedAwsAccessKey 

func NewFetchedAwsAccessKey(accessKeyMetadata *iam.AccessKeyMetadata, monitorAwsAccessKeyId string) *FetchedAwsAccessKey

Create a new FetchedAwsAccessKey

func Scan 

func Scan(accessKeysToScan []FetchedAwsAccessKey, content []byte) (leaks []FetchedAwsAccessKey, err error)

func ScanViaRegexLoop 

func ScanViaRegexLoop(accessKeysToScan []FetchedAwsAccessKey, content []byte) (leaks []FetchedAwsAccessKey, err error)

This is grossly inefficient but it passes all of the tests

func ScanViaTrie 

func ScanViaTrie(accessKeysToScan []FetchedAwsAccessKey, content []byte) (leaks []FetchedAwsAccessKey, err error)

Scan the input in a single pass and use a trie prefix match to figure out if any aws keys match. This is approx 2x faster than ScanViaRegexLoop, but it still feels slow. TODO: try using lexmachine and see if it's a lot faster. Do some post-processing to deal with nested tokens (where one token contains another, which is one of the unit tests)

type GithubClientWrapper 

type GithubClientWrapper struct {
	AccessToken string
	ApiClient   *github.Client
}

func NewGithubClientWrapper 

func NewGithubClientWrapper(accessToken, githubApiBaseUrl string) *GithubClientWrapper

If you want to use the default github API (as opposed to github enterprise), pass in an empty string for the githubApiBaseUrl

type GithubConnectionParams 

type GithubConnectionParams struct {

	// The URL of the github API to connect to.  If blank, will connect to https://api.github.com.
	// Github Enterprise users will need to set this to point to their Github Enterprise server
	GithubApiUrl string

	// The github access token, which needs "read:org" permissions in order to read the concealed "non-public"
	// members of the orgs
	GithubAccessToken string
}

The connection parameters requires to connect to the Github API on github.com or hosted on Github Enterprise.

type GithubEventCheckpoints 

type GithubEventCheckpoints map[string]*github.Event

Githhub User Login -> Last Processed Github Event

func (GithubEventCheckpoints) CheckpointForUser 

func (gec GithubEventCheckpoints) CheckpointForUser(user *github.User) (checkpoint *github.Event, found bool)

type GithubUserAggregator 

type GithubUserAggregator struct {
	AccessToken string
	GithubOrgs  []string
	ApiClient   *github.Client
}

func NewGithubUserAggregator 

func NewGithubUserAggregator(orgs []string, accessToken string) *GithubUserAggregator

func (GithubUserAggregator) CompactedUsers 

func (gua GithubUserAggregator) CompactedUsers(users []*github.User) []*github.User

Get a compacted list of the users that only contains the user data that concerns the keynuker application TODO: this should convert to a keynuker.GithubUser (doesn't exist yet) to increase compile time checking

func (GithubUserAggregator) ListMembers 

func (gua GithubUserAggregator) ListMembers(ctx context.Context) ([]*github.User, error)

func (GithubUserAggregator) ListMembersForOrg 

func (gua GithubUserAggregator) ListMembersForOrg(ctx context.Context, org string) ([]*github.User, error)

type GithubUserEventFetcher 

type GithubUserEventFetcher interface {

	// Given a github username and filtering parameters, fetch events from the user event stream
	FetchUserEvents(ctx context.Context, fetchUserEventsInput FetchUserEventsInput) ([]*github.Event, error)

	// Given a specific github event (eg, a commit), scan the actual content for that event for given aws keys
	ScanDownstreamContent(ctx context.Context, userEvent *github.Event, accessKeysToScan []FetchedAwsAccessKey) (leaks []FetchedAwsAccessKey, err error)
}

Abstract the calls to the github API in an interface for dependency injection / mocking purposes

type GithubUserEventFetcherMock 

type GithubUserEventFetcherMock struct {
	mock.Mock
}

GithubUserEventFetcherMock mock

func NewGithubUserEventFetcherMock 

func NewGithubUserEventFetcherMock() *GithubUserEventFetcherMock

func (*GithubUserEventFetcherMock) FetchUserEvents 

func (m *GithubUserEventFetcherMock) FetchUserEvents(p0 context.Context, p1 FetchUserEventsInput) ([]*github.Event, error)

FetchUserEvents mocked method

func (*GithubUserEventFetcherMock) ScanDownstreamContent 

func (m *GithubUserEventFetcherMock) ScanDownstreamContent(p0 context.Context, p1 *github.Event, p2 []FetchedAwsAccessKey) ([]FetchedAwsAccessKey, error)

ScanDownstreamContent mocked method

type GithubUserEventsScanner 

type GithubUserEventsScanner struct {
	// contains filtered or unexported fields
}

func NewGithubUserEventsScanner 

func NewGithubUserEventsScanner(fetcher GithubUserEventFetcher) *GithubUserEventsScanner

func (GithubUserEventsScanner) ScanAwsKeys 

func (gues GithubUserEventsScanner) ScanAwsKeys(params ParamsScanGithubUserEventsForAwsKeys) (docWrapper DocumentScanGithubUserEventsForAwsKeys, err error)

For each github user, scan their event feed for leaked aws keys This purposely does not make concurrent requests due to: https://developer.github.com/guides/best-practices-for-integrators/#dealing-with-abuse-rate-limits

type GoGithubUserEventFetcher 

type GoGithubUserEventFetcher struct {
	*GithubClientWrapper
}

func NewGoGithubUserEventFetcher 

func NewGoGithubUserEventFetcher(accessToken, githubApiBaseUrl string) *GoGithubUserEventFetcher

If you want to use the default github API (as opposed to github enterprise), pass in an empty string for the githubApiBaseUrl

func (GoGithubUserEventFetcher) FetchUrlContent 

func (guef GoGithubUserEventFetcher) FetchUrlContent(ctx context.Context, url string) (content []byte, err error)

func (GoGithubUserEventFetcher) FetchUserEvents 

func (guef GoGithubUserEventFetcher) FetchUserEvents(ctx context.Context, fetchUserEventsInput FetchUserEventsInput) ([]*github.Event, error)

func (GoGithubUserEventFetcher) ScanBlob 

func (guef GoGithubUserEventFetcher) ScanBlob(owner string, repo string, sha string, accessKeysToScan []FetchedAwsAccessKey) (leaks []FetchedAwsAccessKey, err error)

func (GoGithubUserEventFetcher) ScanCommitsForPushEvent 

func (guef GoGithubUserEventFetcher) ScanCommitsForPushEvent(
	ctx context.Context, userEvent *github.Event, pushEvent *github.PushEvent, accessKeysToScan []FetchedAwsAccessKey) (leaks []FetchedAwsAccessKey, err error)

Since PushEvents only contain 20 commits max, this fetches the remaining commits and writes the content to the writer passed in. For example, pushEvent.Size might indicate that there were 100 commits in the push events, and so the remaining 80 commits will need to be scanned.

Github API: https://developer.github.com/v3/repos/commits/

func (GoGithubUserEventFetcher) ScanContentForCommits 

func (guef GoGithubUserEventFetcher) ScanContentForCommits(ctx context.Context, username, repoName string, commits []WrappedCommit, accessKeysToScan []FetchedAwsAccessKey) (leaks []FetchedAwsAccessKey, err error)

func (GoGithubUserEventFetcher) ScanDownstreamContent 

func (guef GoGithubUserEventFetcher) ScanDownstreamContent(ctx context.Context, userEvent *github.Event, accessKeysToScan []FetchedAwsAccessKey) (leaks []FetchedAwsAccessKey, err error)

type LeakedKeyEvent 

type LeakedKeyEvent struct {
	AccessKeyMetadata FetchedAwsAccessKey
	GithubUser        *github.User
	GithubEvent       *github.Event
	NearbyContent     []byte
	LeakerEmail       string
}

func (LeakedKeyEvent) LeakedKeyIsMonitorKey 

func (l LeakedKeyEvent) LeakedKeyIsMonitorKey() bool

Is the key that was leaked the same (limited permission) key that keynuker is using to monitor aws? Don't nuke it, since this key has very limited permissions and is needed to nuke other keys. Should raise serious alarms if this ever happens.

type MailerParams 

type MailerParams struct {

	// The Mailgun API key for notifications
	ApiKey string `json:"mailer_api_key"`

	// The Mailgun public api key.
	PublicApiKey string `json:"mailer_public_api_key"`

	// The mailgun domain
	Domain string `json:"mailer_domain"`
}

Mailer (Mailgun) Params

type MockMailGun 

type MockMailGun struct {
	SentMessages chan *mailgun.Message

	// Embed the Mailgun interface. If unimplemented methods are called, it will panic
	mailgun.Mailgun
}

func NewMockMailGun 

func NewMockMailGun() *MockMailGun

func (*MockMailGun) Send 

func (mmg *MockMailGun) Send(m *mailgun.Message) (string, string, error)

func (*MockMailGun) WaitForNextMessage 

func (mmg *MockMailGun) WaitForNextMessage(timeout time.Duration) (msg *mailgun.Message, err error)

type NukedKeyEvent 

type NukedKeyEvent struct {
	LeakedKeyEvent LeakedKeyEvent

	DeleteAccessKeyOutput *iam.DeleteAccessKeyOutput

	NukedOn time.Time
}

type ParamsFetchAwsKeys 

type ParamsFetchAwsKeys struct {

	// When using Cross-Account STS AssumeRole, this needs the credentials of the of the "connecting" aka "initiating"
	// account that is being used to connect to the target account being monitored
	InitiatingAwsAccountAssumeRole AwsCredentials

	// The list of AWS accounts to fetch all the access keys for
	TargetAwsAccounts []TargetAwsAccount

	// This is the name of the KeyNuker "org/tenant".  Defaults to "default", but allows to be extended multi-tenant.
	KeyNukerOrg string
}

type ParamsGithubUserAggregator 

type ParamsGithubUserAggregator struct {

	// Github API URL and access token
	GithubConnectionParams

	// This is the name of the KeyNuker "org/tenant".  Defaults to "default", but allows to be extended multi-tenant.
	KeyNukerOrg string

	// A list of github organizations, eg ["acme", "acme-labs", ...]
	GithubOrgs []string

	// A list of individual github user logins you would like to monitor, which is appended to the users found from looking up the users in GithubOrgs.  Eg, ["defunkt", "torvalds"]
	GithubUsers []string
}

func (ParamsGithubUserAggregator) GetGithubUsers 

func (p ParamsGithubUserAggregator) GetGithubUsers() []*github.User

type ParamsLookupGithubUsersAwsKeys 

type ParamsLookupGithubUsersAwsKeys struct {

	// This is the name of the KeyNuker "org/tenant".  Defaults to "default", but allows to be extended multi-tenant.
	KeyNukerOrg string

	// DB connection params
	Username string
	Password string
	Host     string
	DbName   string
}

func (ParamsLookupGithubUsersAwsKeys) Validate 

func (p ParamsLookupGithubUsersAwsKeys) Validate() error

type ParamsMonitorActivations 

type ParamsMonitorActivations struct {

	// MailerParams
	MailerParams

	// This is the name of the KeyNuker "org/tenant".  Defaults to "default", but allows to be extended multi-tenant.
	KeyNukerOrg string

	// The FROM address that will be used for any notifications
	EmailFromAddress string `json:"email_from_address"`

	// Optionally specify the Keynuker admin email to be CC'd about any leaked/nuked keys
	KeynukerAdminEmailCCAddress string `json:"admin_email_cc_address"`
}

type ParamsNukeLeakedAwsKeys 

type ParamsNukeLeakedAwsKeys struct {

	// The list of AWS accounts in scope
	TargetAwsAccounts []TargetAwsAccount

	// This is the name of the KeyNuker "org/tenant".  Defaults to "default", but allows to be extended multi-tenant.
	KeyNukerOrg string

	// The leaked keys to nuke
	LeakedKeyEvents []LeakedKeyEvent

	// The keep per-user checkpoints which will be validated/propagated after keys are successfully nuked
	GithubEventCheckpoints GithubEventCheckpoints
}

func (ParamsNukeLeakedAwsKeys) Validate 

func (p ParamsNukeLeakedAwsKeys) Validate() error

type ParamsPostNukeNotifier 

type ParamsPostNukeNotifier struct {

	// MailerParams
	MailerParams

	// This is the name of the KeyNuker "org/tenant".  Defaults to "default", but allows to be extended multi-tenant.
	KeyNukerOrg string

	// These fields are inputs from the upstream nuke-leaked-aws-keys action
	NukedKeyEvents         []NukedKeyEvent
	GithubEventCheckpoints GithubEventCheckpoints

	// The FROM address that will be used for any notifications
	EmailFromAddress string `json:"email_from_address"`

	// Optionally specify the Keynuker admin email to be CC'd about any leaked/nuked keys
	KeynukerAdminEmailCCAddress string `json:"admin_email_cc_address"`
}

func (ParamsPostNukeNotifier) Validate 

func (p ParamsPostNukeNotifier) Validate() error

type ParamsScanGithubUserEventsForAwsKeys 

type ParamsScanGithubUserEventsForAwsKeys struct {

	// Github API URL and access token
	GithubConnectionParams

	// This is the name of the KeyNuker "org/tenant".  Defaults to "default", but allows to be extended multi-tenant.
	KeyNukerOrg string

	// A list of github users
	GithubUsers []*github.User

	// AWS access keys to scan for
	AccessKeyMetadata []FetchedAwsAccessKey

	// Track the latest event processed for each user in GithubUsers by keeping per-user checkpoints
	GithubEventCheckpoints GithubEventCheckpoints
}

func (ParamsScanGithubUserEventsForAwsKeys) CopyGithubEventCheckpoints 

func (p ParamsScanGithubUserEventsForAwsKeys) CopyGithubEventCheckpoints() GithubEventCheckpoints

func (ParamsScanGithubUserEventsForAwsKeys) CreateFetchUserEventsInput 

func (p ParamsScanGithubUserEventsForAwsKeys) CreateFetchUserEventsInput(user *github.User) FetchUserEventsInput

func (ParamsScanGithubUserEventsForAwsKeys) SetDefaultCheckpointsForMissing 

func (p ParamsScanGithubUserEventsForAwsKeys) SetDefaultCheckpointsForMissing(recentTimeWindow time.Duration) ParamsScanGithubUserEventsForAwsKeys

Update params to set the github event checkpoints to have a recent time window so that it doesn't scan every single user event (which can take too long) in the absence of actual stored checkpoints, which aren't fully working yet as of the time of this writing.

Example recentTimeWindow: time.Duration(time.Hour * -12)

func (ParamsScanGithubUserEventsForAwsKeys) Validate 

func (p ParamsScanGithubUserEventsForAwsKeys) Validate() error

func (ParamsScanGithubUserEventsForAwsKeys) WithDefaultKeynukerOrg 

func (p ParamsScanGithubUserEventsForAwsKeys) WithDefaultKeynukerOrg() ParamsScanGithubUserEventsForAwsKeys

type ParamsWriteDoc 

type ParamsWriteDoc struct {
	Username string
	Password string
	Host     string
	DbName   string
	Doc      map[string]interface{}
	DocId    string
}

func (ParamsWriteDoc) IsCloudantDb 

func (p ParamsWriteDoc) IsCloudantDb() bool

func (ParamsWriteDoc) IsFaunaDb 

func (p ParamsWriteDoc) IsFaunaDb() bool

type PushEventCommit 

type PushEventCommit github.PushEventCommit

func (*PushEventCommit) Sha 

func (p *PushEventCommit) Sha() string

func (*PushEventCommit) Url 

func (p *PushEventCommit) Url() string

type RecentActivationsReportInput 

type RecentActivationsReportInput struct {
	MaxActivationsToScan int
}

type RecentActivationsReportOutput 

type RecentActivationsReportOutput struct {
	FailedActivationIds  []string
	TotalNumBytesScanned int64
}

func OpenWhiskRecentActivationsReport 

func OpenWhiskRecentActivationsReport(input RecentActivationsReportInput) (output RecentActivationsReportOutput, err error)

A more generalized version of OpenWhiskRecentActivationsStatus TODO #1: Moving to structured logging (logrus?) will make this a lot more tenable. Either that or json stats.

type RepositoryCommit 

type RepositoryCommit github.RepositoryCommit

func (*RepositoryCommit) Sha 

func (r *RepositoryCommit) Sha() string

func (*RepositoryCommit) Url 

func (r *RepositoryCommit) Url() string

type ResultPostNukeNotifier 

type ResultPostNukeNotifier struct {
	NukedKeyEvents         []NukedKeyEvent
	GithubEventCheckpoints GithubEventCheckpoints

	// Mailgun delivery id's for messages
	DeliveryIds []string
}

func SendPostNukeMailgunNotifications 

func SendPostNukeMailgunNotifications(params ParamsPostNukeNotifier) (result ResultPostNukeNotifier, err error)

Entry point when using actual OpenWhisk action. Uses live mailgun endpoint.

func SendPostNukeMockNotifications 

func SendPostNukeMockNotifications(mockMailgun mailgun.Mailgun, params ParamsPostNukeNotifier) (result ResultPostNukeNotifier, err error)

Entry point when using test. Uses mock mailgun endpoint.

func SendPostNukeNotifications 

func SendPostNukeNotifications(mailer mailgun.Mailgun, params ParamsPostNukeNotifier) (result ResultPostNukeNotifier, err error)

Entry point with dependency injection that takes a mailer object, might be live mailgun endpoint or mock

type ScanResult 

type ScanResult struct {
	CheckpointEvent *github.Event // Latest event scanned
	User            *github.User
	LeakedKeyEvents []LeakedKeyEvent
	Error           error
}

The result of scanning a user's github events

func (ScanResult) CompactCheckpointEvent 

func (s ScanResult) CompactCheckpointEvent() *github.Event

Return a compact (stripped) version of the checkpoint event that has the minimal fields to still be useful

func (*ScanResult) SetCheckpointIfMostRecent 

func (s *ScanResult) SetCheckpointIfMostRecent(user *github.User, latestEventScanned *github.Event)

func (*ScanResult) SetDefaultResultCheckpoint 

func (s *ScanResult) SetDefaultResultCheckpoint(user *github.User, checkpoints GithubEventCheckpoints)

type ScannerState 

type ScannerState int
const (
	ScannerStateInToken ScannerState = iota
	ScannerStateOutsideToken
)

type TargetAwsAccount 

type TargetAwsAccount struct {
	AwsCredentials

	TargetAwsAccountAssumeRole
}

The TargetAwsAccount can be connected to via two ways: - AwsCredentials: Either using direct credentials of a user with the required permissions - TargetAwsAccountAssumeRole: Using STS AssumeRole

func FindAwsAccount 

func FindAwsAccount(targetAwsAccounts []TargetAwsAccount, monitorAwsAccessKeyId string) (TargetAwsAccount, error)

func FindAwsAccountArtificialError 

func FindAwsAccountArtificialError(targetAwsAccounts []TargetAwsAccount, monitorAwsAccessKeyId string) (TargetAwsAccount, error)

func GetIntegrationTestTargetAwsAccountsFromEnv 

func GetIntegrationTestTargetAwsAccountsFromEnv() (targetAwsAccounts []TargetAwsAccount, err error)

func GetTargetAwsAccountsFromEnv 

func GetTargetAwsAccountsFromEnv() (targetAwsAccounts []TargetAwsAccount, err error)

func (TargetAwsAccount) IsDirect 

func (t TargetAwsAccount) IsDirect() bool

type TargetAwsAccountAssumeRole 

type TargetAwsAccountAssumeRole struct {

	// The target AWS account id.  Eg, 012345
	TargetAwsAccountId string

	// The role on the target account, used to build the role-arn:
	// arn:aws:iam::012345:role/KeyNuker
	TargetRoleName string

	// The ExternalID which provides a layer of security to avoid the "Confused Deputy" attack
	// http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html
	AssumeRoleExternalId string
}

type WrappedCommit 

type WrappedCommit interface {
	Sha() string
	Url() string
}

Wrap the github.RepositoryCommit and github.PushEventCommit into a single common interface

func ConvertPushEventCommits 

func ConvertPushEventCommits(pushEventCommits []github.PushEventCommit) []WrappedCommit

func ConvertRepositoryCommits 

func ConvertRepositoryCommits(repositoryCommits []*github.RepositoryCommit) []WrappedCommit

Source Files

View all Source files

Directories

Path Synopsis
cmd
activity-report
fetch-aws-keys
github-user-aggregator
github-user-events-scanner
lookup-github-users-aws-keys
monitor-activations
nuke-leaked-aws-keys
post-nuke-notifier
write-doc

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.