devstats

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 30, 2018 License: Apache-2.0 Imports: 24 Imported by: 0

README

Build Status CII Best Practices

GitHub archives and git Grafana visualization dashboards

Author: Łukasz Gryglicki lukaszgryglick@o2.pl

This is a toolset to visualize GitHub archives using Grafana dashboards.

Gha2db stands for GitHub Archives to DashBoards.

Goal

We want to create a toolset for visualizing various metrics for the Kubernetes community.

Everything is open source so that it can be used by other CNCF and non-CNCF open source projects.

This tool can also be used for Prometheus and other communities as well.

The only requirement is that project must be hosted on a public GitHub repository/repositories.

Forking and installing locally

This toolset uses only Open Source tools: GitHub archives, git, Postgres databases, InfluxDB time-series databases and multiple Grafana dashboards running in Docker conatiners. It is written in Go, and can be forked and installed by anyone.

Contributions and PRs are welcome. If You see a bug or want to add a new metric please create an issue and/or PR.

To work on this project locally please fork the original repository, and:

Please see Development for local development guide.

For more detailed description of all environment variables, tools, switches etc, please see usage.

Metrics

We want to support all kind of metrics, including historical ones. Please see requested metrics to see what kind of metrics are needed. Many of them cannot be computed based on the data sources currently used.

Repository groups

There are some groups of repositories that are grouped together as a repository groups. They are defined in scripts/kubernetes/repo_groups.sql.

To setup default repository groups:

  • PG_PASS=pwd ./kubernetes/setup_repo_groups.sh.

This is a part of kubernetes/psql.sh script and kubernetes psql dump already has groups configured.

Company Affiliations

We also want to have per company statistics. To implement such metrics we need a mapping of developers and their employers.

There is a project that attempts to create such mapping cncf/gitdm.

Gha2db has an import tool that fetches company affiliations from cncf/gitdm and allows to create per company metrics/statistics.

If you see errors in the company affiliations, please open a pull request on cncf/gitdm and the updates will be reflected on https://k8s.devstats.cncf.io a couple days after the PR has been accepted. Note that gitdm supports mapping based on dates, to account for developers moving between companies.

Architecture

For architecture details please see architecture file.

Detailed usage is here

Adding new metrics

Please see metrics to see how to add new metrics.

Adding new projects

To add new project follow adding new project instructions.

Grafana dashboards

Please see dashboards to see list of already defined Grafana dashboards.

Exporting data

Please see exporting.

Detailed Usage instructions

Servers

The servers to run devstats are generously provided by Packet bare metal hosting as part of CNCF's Community Infrastructure Lab.

Documentation

Index

Constants

View Source 
const DataDir string = "/etc/gha2db/"

DataDir - common constant string

View Source 
const Devstats string = "devstats"

Devstats - common constant string

View Source 
const EngineIsClosedError string = "engine is closed"

EngineIsClosedError - common constant string

View Source 
const GHA string = "gha"

GHA - common constant string

View Source 
const GHAAdmin string = "gha_admin"

GHAAdmin - common constant string

View Source 
const LocalGitScripts string = "./git/"

LocalGitScripts - common constant string

View Source 
const Localhost string = "localhost"

Localhost - common constant string

View Source 
const Now string = "now"

Now - common constant string

View Source 
const Password string = "password"

Password - common constant string

View Source 
const Quarter string = "quarter"

Quarter - common constant string

View Source 
const Retry string = "retry"

Retry - common constant string

View Source 
const TimeoutError string = "{\"error\":\"timeout\"}\n"

TimeoutError - common constant string

View Source 
const Today string = "today"

Today - common constant string

Variables

This section is empty.

Functions

func ActorIDOrNil 

func ActorIDOrNil(actPtr *Actor) interface{}

ActorIDOrNil - return Actor ID from pointer or nil

func ActorLoginOrNil 

func ActorLoginOrNil(actPtr *Actor) interface{}

ActorLoginOrNil - return Actor Login from pointer or nil

func AddNIntervals 

func AddNIntervals(dt time.Time, n int, nextIntervalStart, prevIntervalStart func(time.Time) time.Time) time.Time

AddNIntervals adds (using nextIntervalStart) or subtracts (using prevIntervalStart) N itervals to the given date Functions Next/Prev can use Hour, Day, Week, Month, Quarter, Year functions (defined in this module) or other custom defined functions With `func(time.Time) time.Time` signature

func BoolOrNil 

func BoolOrNil(boolPtr *bool) interface{}

BoolOrNil - return either nil or value of boolPtr

func CleanUTF8 

func CleanUTF8(str string) string

CleanUTF8 - clean UTF8 string to containg only Pq allowed runes

func ClearDBLogs 

func ClearDBLogs()

ClearDBLogs clears logs older by defined period (in context.go) It clears logs on `devstats` database

func CommentIDOrNil 

func CommentIDOrNil(commPtr *Comment) interface{}

CommentIDOrNil - return Comment ID from pointer or nil

func ComputePeriodAtThisDate 

func ComputePeriodAtThisDate(period string, to time.Time) bool

ComputePeriodAtThisDate - for some longer periods, only recalculate them on specific dates hourly period is always calculated daily period is always calculated multiple days period are calculaded at hours: 1, 5, 9, 13, 17, 21 (UTC) annotation ranges are calculated: from last release to now - every 2 hours for past ranges only once (calculation is marked as computed) at 2 AM weekly ranges are calculated at hours: 0, 4, 8, 12, 16, 20 monthly, quarterly, yearly ranges are calculated at midnight

func CreateDatabaseIfNeeded 

func CreateDatabaseIfNeeded(ctx *Ctx) bool

CreateDatabaseIfNeeded - creates requested database if not exists Returns true if database was not existing existed and created dropped

func CreateTable 

func CreateTable(tdef string) string

CreateTable is used to replace DB specific parts of Create Table SQL statement

func DatabaseExists 

func DatabaseExists(ctx *Ctx, closeConn bool) (exists bool, c *sql.DB)

DatabaseExists - checks if database stored in context exists If closeConn is true - then it closes connection after checking if database exists If closeConn is false, then it returns open connection to default database "postgres"

func DayStart 

func DayStart(dt time.Time) time.Time

DayStart - return time rounded to current day start

func DescriblePeriodInHours 

func DescriblePeriodInHours(hrs float64) (desc string)

DescriblePeriodInHours - return string description of a time period given in hours

func DropDatabaseIfExists 

func DropDatabaseIfExists(ctx *Ctx) bool

DropDatabaseIfExists - drops requested database if exists Returns true if database existed and was dropped

func ExecCommand 

func ExecCommand(ctx *Ctx, cmdAndArgs []string, env map[string]string) (string, error)

ExecCommand - execute command given by array of strings with eventual environment map

func ExecSQL 

func ExecSQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) (sql.Result, error)

ExecSQL executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed)

func ExecSQLTx 

func ExecSQLTx(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) (sql.Result, error)

ExecSQLTx executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed) It is for running inside transaction

func ExecSQLTxWithErr 

func ExecSQLTxWithErr(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) sql.Result

ExecSQLTxWithErr wrapper to ExecSQLTx that exists on error It is for running inside transaction

func ExecSQLWithErr 

func ExecSQLWithErr(con *sql.DB, ctx *Ctx, query string, args ...interface{}) sql.Result

ExecSQLWithErr wrapper to ExecSQL that exists on error

func FatalOnError 

func FatalOnError(err error) string

FatalOnError displays error message (if error present) and exits program

func FirstIntOrNil 

func FirstIntOrNil(intPtrs []*int) interface{}

FirstIntOrNil - return either nil or value of intPtr

func ForkeeIDOrNil 

func ForkeeIDOrNil(forkPtr *Forkee) interface{}

ForkeeIDOrNil - return Forkee ID from pointer or nil

func ForkeeNameOrNil 

func ForkeeNameOrNil(forkPtr *Forkee) interface{}

ForkeeNameOrNil - return Forkee Name from pointer or nil

func ForkeeOldIDOrNil 

func ForkeeOldIDOrNil(forkPtr *ForkeeOld) interface{}

ForkeeOldIDOrNil - return ForkeeOld ID from pointer or nil

func GetIntervalFunctions 

func GetIntervalFunctions(intervalAbbr string, allowUnknown bool) (interval string, n int, intervalStart, nextIntervalStart, prevIntervalStart func(time.Time) time.Time)

GetIntervalFunctions - return interval name, interval number, interval start, next, prev function from interval abbr: h|d2|w3|m4|q|y w3 = 3 weeks, q2 = 2 quarters, y = year (1), d7 = 7 days (not the same as w), m3 = 3 months (not the same as q)

func GetTagValues 

func GetTagValues(con client.Client, ctx *Ctx, key string) (ret []string)

GetTagValues returns tag values for a given key

func GetThreadsNum 

func GetThreadsNum(ctx *Ctx) int

GetThreadsNum returns the number of available CPUs If environment variable GHA2DB_ST is set it retuns 1 It can be used to debug single threaded verion It runs on 90% CPU power by default

func HashStrings 

func HashStrings(strs []string) int

HashStrings - returns unique Hash for strings array This value is supposed to be used as ID (negative) to mark it was artificially generated

func HourStart 

func HourStart(dt time.Time) time.Time

HourStart - return time rounded to current hour start

func IDBAddPointN 

func IDBAddPointN(ctx *Ctx, con *client.Client, points *IDBBatchPointsN, pt *client.Point)

IDBAddPointN - adds point to the batch, eventually auto flushing

func IDBAddPointNWithDB 

func IDBAddPointNWithDB(ctx *Ctx, con *client.Client, points *IDBBatchPointsN, pt *client.Point, db string)

IDBAddPointNWithDB - adds point to the batch, eventually auto flushing

func IDBBatchPoints 

func IDBBatchPoints(ctx *Ctx, con *client.Client) client.BatchPoints

IDBBatchPoints returns batch points for given connection and database from context

func IDBBatchPointsWithDB 

func IDBBatchPointsWithDB(ctx *Ctx, con *client.Client, db string) client.BatchPoints

IDBBatchPointsWithDB returns batch points for given connection and database from context

func IDBConn 

func IDBConn(ctx *Ctx) client.Client

IDBConn Connects to InfluxDB database

func IDBNewPointWithErr 

func IDBNewPointWithErr(name string, tags map[string]string, fields map[string]interface{}, dt time.Time) *client.Point

IDBNewPointWithErr - return InfluxDB Point, on error exit

func IDBWritePointsN 

func IDBWritePointsN(ctx *Ctx, con *client.Client, points *IDBBatchPointsN) (err error)

IDBWritePointsN - writes batch points

func InsertIgnore 

func InsertIgnore(query string) string

InsertIgnore - will return insert statement with ignore option specific for DB

func IntOrNil 

func IntOrNil(intPtr *int) interface{}

IntOrNil - return either nil or value of intPtr

func IsProjectDisabled 

func IsProjectDisabled(ctx *Ctx, proj string, yamlDisabled bool) bool

IsProjectDisabled - checks if project is disabled or not: fullName comes from makeOldRepoName for pre-2015 data! yamlDisabled (this is from projects.yaml - can be true or false) it also checks context (which can override `disabled: true` from projects.yaml) +pro1,-pro2 creates map {"pro1":true, "pro2":false}

func IssueIDOrNil 

func IssueIDOrNil(issuePtr *Issue) interface{}

IssueIDOrNil - return Issue ID from pointer or nil

func MakeOldRepoName 

func MakeOldRepoName(repo *ForkeeOld) string

MakeOldRepoName - before 2015 repository name should be Organization/Name (if Organization present) or just Name

func MakeUniqueSort 

func MakeUniqueSort(ary []string) (outAry []string)

MakeUniqueSort - make string array unique & sorted

func Mgetc 

func Mgetc(ctx *Ctx) string

Mgetc waits for single key press and return character pressed

func MilestoneIDOrNil 

func MilestoneIDOrNil(milPtr *Milestone) interface{}

MilestoneIDOrNil - return Milestone ID from pointer or nil

func MonthStart 

func MonthStart(dt time.Time) time.Time

MonthStart - return time rounded to current month start

func NValue 

func NValue(index int) string

NValue will return $n

func NValues 

func NValues(n int) string

NValues will return values($1, $2, .., $n)

func NegatedBoolOrNil 

func NegatedBoolOrNil(boolPtr *bool) interface{}

NegatedBoolOrNil - return either nil or negated value of boolPtr

func NextDayStart 

func NextDayStart(dt time.Time) time.Time

NextDayStart - return time rounded to next day start

func NextHourStart 

func NextHourStart(dt time.Time) time.Time

NextHourStart - return time rounded to next hour start

func NextMonthStart 

func NextMonthStart(dt time.Time) time.Time

NextMonthStart - return time rounded to next month start

func NextQuarterStart 

func NextQuarterStart(dt time.Time) time.Time

NextQuarterStart - return time rounded to next quarter start

func NextWeekStart 

func NextWeekStart(dt time.Time) time.Time

NextWeekStart - return time rounded to next week start

func NextYearStart 

func NextYearStart(dt time.Time) time.Time

NextYearStart - return time rounded to next year start

func NormalizeName 

func NormalizeName(str string) string

NormalizeName - clean DB string from -, /, ., " ", trim leading and trailing space, lowercase Normalize Unicode characters

func OrgIDOrNil 

func OrgIDOrNil(orgPtr *Org) interface{}

OrgIDOrNil - return Org ID from pointer or nil

func OrgLoginOrNil 

func OrgLoginOrNil(orgPtr *Org) interface{}

OrgLoginOrNil - return Org ID from pointer or nil

func PgConn 

func PgConn(ctx *Ctx) *sql.DB

PgConn Connects to Postgres database

func PgConnDB 

func PgConnDB(ctx *Ctx, dbName string) *sql.DB

PgConnDB Connects to Postgres database (with specific DB name) uses database 'dbname' instead of 'PgDB'

func PrepareQuickRangeQuery 

func PrepareQuickRangeQuery(sql, period, from, to string) string

PrepareQuickRangeQuery Perpares query using either ready `period` string or using `from` and `to` strings Values to replace are specially encoded {{period:alias.column}} Can either replace with: (alias.column >= now() - 'period'::interval) Or (alias.column >= 'from' and alias.column < 'to')

func PrettyPrintJSON 

func PrettyPrintJSON(jsonBytes []byte) []byte

PrettyPrintJSON - pretty formats raw JSON bytes

func PrevDayStart 

func PrevDayStart(dt time.Time) time.Time

PrevDayStart - return time rounded to prev day start

func PrevHourStart 

func PrevHourStart(dt time.Time) time.Time

PrevHourStart - return time rounded to prev hour start

func PrevMonthStart 

func PrevMonthStart(dt time.Time) time.Time

PrevMonthStart - return time rounded to prev month start

func PrevQuarterStart 

func PrevQuarterStart(dt time.Time) time.Time

PrevQuarterStart - return time rounded to prev quarter start

func PrevWeekStart 

func PrevWeekStart(dt time.Time) time.Time

PrevWeekStart - return time rounded to prev week start

func PrevYearStart 

func PrevYearStart(dt time.Time) time.Time

PrevYearStart - return time rounded to prev year start

func Printf 

func Printf(format string, args ...interface{}) (n int, err error)

Printf is a wrapper around Printf(...) that supports logging.

func ProcessAnnotations 

func ProcessAnnotations(ctx *Ctx, annotations *Annotations, joinDate *time.Time)

ProcessAnnotations Creates IfluxDB annotations and quick_series

func ProgressInfo 

func ProgressInfo(i, n int, start time.Time, last *time.Time, period time.Duration, msg string)

ProgressInfo display info about progress: i/n if current time >= last + period If displayed info, update last

func PullRequestIDOrNil 

func PullRequestIDOrNil(prPtr *PullRequest) interface{}

PullRequestIDOrNil - return PullRequest ID from pointer or nil

func QuarterStart 

func QuarterStart(dt time.Time) time.Time

QuarterStart - return time rounded to current month start

func QueryIDB 

func QueryIDB(con client.Client, ctx *Ctx, query string) []client.Result

QueryIDB - do InfluxDB query

func QueryIDBWithDB 

func QueryIDBWithDB(con client.Client, ctx *Ctx, query, db string) []client.Result

QueryIDBWithDB - do InfluxDB query

func QueryRowSQL 

func QueryRowSQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) *sql.Row

QueryRowSQL executes given SQL on Postgres DB (and returns single row)

func QuerySQL 

func QuerySQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) (*sql.Rows, error)

QuerySQL executes given SQL on Postgres DB (and returns rowset that needs to be closed)

func QuerySQLTx 

func QuerySQLTx(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) (*sql.Rows, error)

QuerySQLTx executes given SQL on Postgres DB (and returns rowset that needs to be closed) It is for running inside transaction

func QuerySQLTxWithErr 

func QuerySQLTxWithErr(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) *sql.Rows

QuerySQLTxWithErr wrapper to QuerySQLTx that exists on error It is for running inside transaction

func QuerySQLWithErr 

func QuerySQLWithErr(con *sql.DB, ctx *Ctx, query string, args ...interface{}) *sql.Rows

QuerySQLWithErr wrapper to QuerySQL that exists on error

func ReleaseIDOrNil 

func ReleaseIDOrNil(relPtr *Release) interface{}

ReleaseIDOrNil - return Release ID from pointer or nil

func RepoHit 

func RepoHit(ctx *Ctx, fullName string, forg, frepo map[string]struct{}) bool

RepoHit - are we interested in this org/repo ?

func RepoIDOrNil 

func RepoIDOrNil(repoPtr *Repo) interface{}

RepoIDOrNil - return Repo ID from pointer or nil

func RepoNameOrNil 

func RepoNameOrNil(repoPtr *Repo) interface{}

RepoNameOrNil - return Repo Name from pointer or nil

func SafeQueryIDB 

func SafeQueryIDB(con client.Client, ctx *Ctx, query string) (*client.Response, error)

SafeQueryIDB - do InfluxDB query, on error return error data

func SkipEmpty 

func SkipEmpty(arr []string) []string

SkipEmpty - skip one element arrays contining only empty string This is what strings.Split() returns for empty input We expect empty array or empty map returned in such cases

func StringOrNil 

func StringOrNil(strPtr *string) interface{}

StringOrNil - return either nil or value of strPtr

func StringsMapToArray 

func StringsMapToArray(f func(string) string, strArr []string) []string

StringsMapToArray this is a function that calls given function for all array items and returns array of items processed by this func Example call: lib.StringsMapToArray(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "})

func StringsMapToSet 

func StringsMapToSet(f func(string) string, strArr []string) map[string]struct{}

StringsMapToSet this is a function that calls given function for all array items and returns set of items processed by this func Example call: lib.StringsMapToSet(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "})

func StringsSetKeys 

func StringsSetKeys(set map[string]struct{}) []string

StringsSetKeys - returns all keys from string map

func StripUnicode 

func StripUnicode(str string) string

StripUnicode strip non-unicode and control characters from a string From: https://rosettacode.org/wiki/Strip_control_codes_and_extended_characters_from_a_string#Go

func Structure 

func Structure(ctx *Ctx)

Structure creates full database structure, indexes, views/summary tables etc

func TimeOrNil 

func TimeOrNil(timePtr *time.Time) interface{}

TimeOrNil - return either nil or value of timePtr

func TimeParseAny 

func TimeParseAny(dtStr string) time.Time

TimeParseAny - attempts to parse time from string YYYY-MM-DD HH:MI:SS Skipping parts from right until only YYYY id left

func TimeParseIDB 

func TimeParseIDB(dtStr string) time.Time

TimeParseIDB - parse InfluxDB time output string into time.Time

func ToGHADate 

func ToGHADate(dt time.Time) string

ToGHADate - return time formatted as YYYY-MM-DD-H

func ToYMDDate 

func ToYMDDate(dt time.Time) string

ToYMDDate - return time formatted as YYYY-MM-DD

func ToYMDHDate 

func ToYMDHDate(dt time.Time) string

ToYMDHDate - return time formatted as YYYY-MM-DD HH

func ToYMDHMSDate 

func ToYMDHMSDate(dt time.Time) string

ToYMDHMSDate - return time formatted as YYYY-MM-DD HH:MI:SS

func TruncStringOrNil 

func TruncStringOrNil(strPtr *string, maxLen int) interface{}

TruncStringOrNil - return either nil or value of strPtr truncated to maxLen chars

func TruncToBytes 

func TruncToBytes(str string, size int) string

TruncToBytes - truncates text to <= size bytes (note that this can be a lot less UTF-8 runes)

func WeekStart 

func WeekStart(dt time.Time) time.Time

WeekStart - return time rounded to current week start Assumes first week day is Sunday

func YearStart 

func YearStart(dt time.Time) time.Time

YearStart - return time rounded to current month start

Types

type Actor 

type Actor struct {
	ID    int    `json:"id"`
	Login string `json:"login"`
	Name  string `json:"-"`
}

Actor - GHA Actor structure Name is unexported and not used by JSON load/save But is used when importing affiliations from cncf/gitdm:github_users.json

type AllProjects 

type AllProjects struct {
	Projects map[string]Project `yaml:"projects"`
}

AllProjects contain all projects data

type Annotation 

type Annotation struct {
	Name        string
	Description string
	Date        time.Time
}

Annotation contain each annotation data

type Annotations 

type Annotations struct {
	Annotations []Annotation
}

Annotations contain list of annotations

func GetAnnotations 

func GetAnnotations(ctx *Ctx, orgRepo, annoRegexp string) (annotations Annotations)

GetAnnotations queries GitHub `orgRepo` via GitHub API (using ctx.GitHubOAuth) for all tags and returns those matching `annoRegexp`

func GetFakeAnnotations 

func GetFakeAnnotations(startDate, joinDate time.Time) (annotations Annotations)

GetFakeAnnotations - returns 'startDate - joinDate' and 'joinDate - now' annotations

type AnnotationsByDate 

type AnnotationsByDate []Annotation

AnnotationsByDate annotations Sort interface

func (AnnotationsByDate) Len 

func (a AnnotationsByDate) Len() int

func (AnnotationsByDate) Less 

func (a AnnotationsByDate) Less(i, j int) bool

func (AnnotationsByDate) Swap 

func (a AnnotationsByDate) Swap(i, j int)

type AnyArray 

type AnyArray []interface{}

AnyArray - holds array of interface{} - just a shortcut

type Asset 

type Asset struct {
	ID            int       `json:"id"`
	CreatedAt     time.Time `json:"created_at"`
	UpdatedAt     time.Time `json:"updated_at"`
	Name          string    `json:"name"`
	Label         *string   `json:"label"`
	Uploader      Actor     `json:"uploader"`
	ContentType   string    `json:"content_type"`
	State         string    `json:"state"`
	Size          int       `json:"size"`
	DownloadCount int       `json:"download_count"`
}

Asset - GHA Asset structure

type Author 

type Author struct {
	Name  string `json:"name"`
	Email string `json:"email"`
}

Author - GHA Commit Author structure

type Branch 

type Branch struct {
	SHA   string  `json:"sha"`
	User  *Actor  `json:"user"`
	Repo  *Forkee `json:"repo"` // This is confusing, but actually GHA has "repo" fields that holds "forkee" structure
	Label string  `json:"label"`
	Ref   string  `json:"ref"`
}

Branch - GHA Branch structure

type Comment 

type Comment struct {
	ID                  int       `json:"id"`
	Body                string    `json:"body"`
	CreatedAt           time.Time `json:"created_at"`
	UpdatedAt           time.Time `json:"updated_at"`
	User                Actor     `json:"user"`
	CommitID            *string   `json:"commit_id"`
	OriginalCommitID    *string   `json:"original_commit_id"`
	DiffHunk            *string   `json:"diff_hunk"`
	Position            *int      `json:"position"`
	OriginalPosition    *int      `json:"original_position"`
	Path                *string   `json:"path"`
	PullRequestReviewID *int      `json:"pull_request_review_id"`
	Line                *int      `json:"line"`
}

Comment - GHA Comment structure

type Commit 

type Commit struct {
	SHA      string `json:"sha"`
	Author   Author `json:"author"`
	Message  string `json:"message"`
	Distinct bool   `json:"distinct"`
}

Commit - GHA Commit structure

type Ctx 

type Ctx struct {
	Debug             int             // from GHA2DB_DEBUG Debug level: 0-no, 1-info, 2-verbose, including SQLs, default 0
	CmdDebug          int             // from GHA2DB_CMDDEBUG Commands execution Debug level: 0-no, 1-only output commands, 2-output commands and their output, 3-output full environment as well, default 0
	JSONOut           bool            // from GHA2DB_JSON gha2db: write JSON files? default false
	DBOut             bool            // from GHA2DB_NODB gha2db: write to SQL database, default true
	ST                bool            // from GHA2DB_ST true: use single threaded version, false: use multi threaded version, default false
	NCPUs             int             // from GHA2DB_NCPUS, set to override number of CPUs to run, this overwrites GHA2DB_ST, default 0 (which means do not use it)
	PgHost            string          // from PG_HOST, default "localhost"
	PgPort            string          // from PG_PORT, default "5432"
	PgDB              string          // from PG_DB, default "gha"
	PgUser            string          // from PG_USER, default "gha_admin"
	PgPass            string          // from PG_PASS, default "password"
	PgSSL             string          // from PG_SSL, default "disable"
	Index             bool            // from GHA2DB_INDEX Create DB index? default false
	Table             bool            // from GHA2DB_SKIPTABLE Create table structure? default true
	Tools             bool            // from GHA2DB_SKIPTOOLS Create DB tools (like views, summary tables, materialized views etc)? default true
	Mgetc             string          // from GHA2DB_MGETC Character returned by mgetc (if non empty), default ""
	IDBHost           string          // from IDB_HOST, default "http://localhost"
	IDBPort           string          // form IDB_PORT, default 8086
	IDBDB             string          // from IDB_DB, default "gha"
	IDBUser           string          // from IDB_USER, default "gha_admin"
	IDBPass           string          // from IDB_PASS, default "password"
	IDBMaxBatchPoints int             // from IDB_MAXBATCHPONTS, all Influx related tools, default 10240 (10k)
	QOut              bool            // from GHA2DB_QOUT output all SQL queries?, default false
	CtxOut            bool            // from GHA2DB_CTXOUT output all context data (this struct), default false
	LogTime           bool            // from GHA2DB_SKIPTIME, output time with all lib.Printf(...) calls, default true, use GHA2DB_SKIPTIME to disable
	DefaultStartDate  time.Time       // from GHA2DB_STARTDT, default `2014-06-01 00:00 UTC`, expects format "YYYY-MM-DD HH:MI:SS", can be set in `projects.yaml` via `start_date:`, value from projects.yaml (if set) has the highest priority.
	LastSeries        string          // from GHA2DB_LASTSERIES, use this InfluxDB series to determine last timestamp date, default "events_h"
	SkipIDB           bool            // from GHA2DB_SKIPIDB gha2db_sync tool, skip Influx DB processing? for db2influx it skips final series write, default false
	SkipPDB           bool            // from GHA2DB_SKIPPDB gha2db_sync tool, skip Postgres DB processing? default false
	ResetIDB          bool            // from GHA2DB_RESETIDB sync tool, regenerate all InfluxDB points? default false
	ResetRanges       bool            // from GHA2DB_RESETRANGES sync tool, regenerate all past quick ranges? default false
	Explain           bool            // from GHA2DB_EXPLAIN runq tool, prefix query with "explain " - it will display query plan instead of executing real query, default false
	OldFormat         bool            // from GHA2DB_OLDFMT gha2db tool, if set then use pre 2015 GHA JSONs format
	Exact             bool            // From GHA2DB_EXACT gha2db tool, if set then orgs list provided from commandline is used as a list of exact repository full names, like "a/b,c/d,e", if not only full names "a/b,x/y" can be treated like this, names without "/" are either orgs or repos.
	LogToDB           bool            // From GHA2DB_SKIPLOG all tools, if set, DB logging into Postgres table `gha_logs` in `devstats` database will be disabled
	Local             bool            // From GHA2DB_LOCAL gha2db_sync tool, if set, gha2_db will call other tools prefixed with "./" to use local compile ones. Otherwise it will call binaries without prefix (so it will use thos ein /usr/bin/).
	MetricsYaml       string          // From GHA2DB_METRICS_YAML gha2db_sync tool, set other metrics.yaml file, default is "metrics/{{project}}metrics.yaml"
	GapsYaml          string          // From GHA2DB_GAPS_YAML gha2db_sync tool, set other gaps.yaml file, default is "metrics/{{project}}/gaps.yaml"
	TagsYaml          string          // From GHA2DB_TAGS_YAML idb_tags tool, set other idb_tags.yaml file, default is "metrics/{{project}}/idb_tags.yaml"
	GitHubOAuth       string          // From GHA2DB_GITHUB_OAUTH annotations tool, if not set reads from /etc/github/oauth file, set to "-" to force public access.
	ClearDBPeriod     string          // From GHA2DB_MAXLOGAGE gha2db_sync tool, maximum age of devstats.gha_logs entries, default "1 week"
	Trials            []int           // From GHA2DB_TRIALS, all Postgres related tools, retry periods for "too many connections open" error
	WebHookRoot       string          // From GHA2DB_WHROOT, webhook tool, default "/hook", must match .travis.yml notifications webhooks
	WebHookPort       string          // From GHA2DB_WHPORT, webhook tool, default ":1982", note that webhook listens using http:1982, but we use apache on https:2982 (to enable https protocol and proxy requests to http:1982)
	WebHookHost       string          // From GHA2DB_WHHOST, webhook tool, default "127.0.0.1" (this can be localhost to disable access by IP, we use Apache proxy to enable https and then apache only need 127.0.0.1)
	CheckPayload      bool            // From GHA2DB_SKIP_VERIFY_PAYLOAD, webhook tool, default true, use GHA2DB_SKIP_VERIFY_PAYLOAD=1 to manually test payloads
	DeployBranches    []string        // From GHA2DB_DEPLOY_BRANCHES, webhook tool, default "master" - comma separated list
	DeployStatuses    []string        // From GHA2DB_DEPLOY_STATUSES, webhook tool, default "Passed,Fixed", - comma separated list
	DeployResults     []int           // From GHA2DB_DEPLOY_RESULTS, webhook tool, default "0", - comma separated list
	DeployTypes       []string        // From GHA2DB_DEPLOY_TYPES, webhook tool, default "push", - comma separated list
	ProjectRoot       string          // From GHA2DB_PROJECT_ROOT, webhook tool, no default, must be specified to run webhook tool
	ExecFatal         bool            // default true, set this manually to false to avoid lib.ExecCommand calling os.Exit() on failure and return error instead
	ExecQuiet         bool            // default false, set this manually to true to have quite exec failures (for example `get_repos` git-clones or git-pulls on errors).
	ExecOutput        bool            // default false, set to true to capture commands STDOUT
	Project           string          // From GHA2DB_PROJECT, gha2db_sync default "", You should set it to something like "kubernetes", "prometheus" etc.
	TestsYaml         string          // From GHA2DB_TESTS_YAML ./dbtest.sh tool, set other tests.yaml file, default is "tests.yaml"
	ReposDir          string          // From GHA2DB_REPOS_DIR ./get_repos tool, default "~/devstats_repos/"
	ProcessRepos      bool            // From GHA2DB_PROCESS_REPOS ./get_repos tool, enable processing (cloning/pulling) all devstats repos, default false
	ProcessCommits    bool            // From GHA2DB_PROCESS_COMMITS ./get_repos tool, enable update/create mapping table: commit - list of file that commit refers to, default false
	ExternalInfo      bool            // From GHA2DB_EXTERNAL_INFO ./get_repos tool, enable outputing data needed by external tools (cncf/gitdm), default false
	ProjectsCommits   string          // From GHA2DB_PROJECTS_COMMITS ./get_repos tool, set list of projects for commits analysis instead of analysing all, default "" - means all
	ProjectsYaml      string          // From GHA2DB_PROJECTS_YAML, many tools - set main projects file, default "projects.yaml"
	ProjectsOverride  map[string]bool // From GHA2DB_PROJECTS_OVERRIDE, ./get_repos and ./devstats tools - for example "-pro1,+pro2" means never sync pro1 and always sync pro2 (even if disabled in `projects.yaml`).
	ExcludeRepos      map[string]bool // From GHA2DB_EXCLUDE_REPOS, ./gha2db tool, default "" - comma separated list of repos to exclude, example: "theupdateframework/notary,theupdateframework/other"
	InputDBs          []string        // From GHA2DB_INPUT_DBS, ./merge_pdbs tool - list of input databases to merge, order matters - first one will insert on a clean DB, next will do insert ignore (to avoid constraints failure due to common data)
	OutputDB          string          // From GHA2DB_OUTPUT_DB, ./merge_pdbs tool - output database to merge into
}

Ctx - environment context packed in structure

func (*Ctx) Init 

func (ctx *Ctx) Init()

Init - get context from environment variables

func (*Ctx) Print 

func (ctx *Ctx) Print()

Print context contents

type Dummy 

type Dummy struct{}

Dummy - structure with no data pointer to this struct is used to test if such field was present in JSON or not

type Event 

type Event struct {
	ID        string    `json:"id"`
	Type      string    `json:"type"`
	Public    bool      `json:"public"`
	CreatedAt time.Time `json:"created_at"`
	Actor     Actor     `json:"actor"`
	Repo      Repo      `json:"repo"`
	Org       *Org      `json:"org"`
	Payload   Payload   `json:"payload"`
}

Event - full GHA (GitHub Archive) event structure

type EventOld 

type EventOld struct {
	ID         string      `json:"-"`
	Type       string      `json:"type"`
	Public     bool        `json:"public"`
	CreatedAt  time.Time   `json:"created_at"`
	Actor      string      `json:"actor"`
	Repository ForkeeOld   `json:"repository"`
	Payload    *PayloadOld `json:"payload"`
}

EventOld - full GHA (GitHub Archive) event structure, before 2015

type Forkee 

type Forkee struct {
	ID              int        `json:"id"`
	Name            string     `json:"name"`
	FullName        string     `json:"full_name"`
	Owner           Actor      `json:"owner"`
	Description     *string    `json:"description"`
	Public          *bool      `json:"public"`
	Fork            bool       `json:"fork"`
	CreatedAt       time.Time  `json:"created_at"`
	UpdatedAt       time.Time  `json:"updated_at"`
	PushedAt        *time.Time `json:"pushed_at"`
	Homepage        *string    `json:"homepage"`
	Size            int        `json:"size"`
	StargazersCount int        `json:"stargazers_count"`
	HasIssues       bool       `json:"has_issues"`
	HasProjects     *bool      `json:"has_projects"`
	HasDownloads    bool       `json:"has_downloads"`
	HasWiki         bool       `json:"has_wiki"`
	HasPages        *bool      `json:"has_pages"`
	Forks           int        `json:"forks"`
	OpenIssues      int        `json:"open_issues"`
	Watchers        int        `json:"watchers"`
	DefaultBranch   string     `json:"default_branch"`
}

Forkee - GHA Forkee structure

type ForkeeOld 

type ForkeeOld struct {
	ID            int        `json:"id"`
	CreatedAt     time.Time  `json:"created_at"`
	Description   *string    `json:"description"`
	Fork          bool       `json:"fork"`
	Forks         int        `json:"forks"`
	HasDownloads  bool       `json:"has_downloads"`
	HasIssues     bool       `json:"has_issues"`
	HasWiki       bool       `json:"has_wiki"`
	Homepage      *string    `json:"homepage"`
	Language      *string    `json:"language"`
	DefaultBranch string     `json:"master_branch"`
	Name          string     `json:"name"`
	OpenIssues    int        `json:"open_issues"`
	Organization  *string    `json:"organization"`
	Owner         string     `json:"owner"`
	Private       *bool      `json:"private"`
	PushedAt      *time.Time `json:"pushed_at"`
	Size          int        `json:"size"`
	Stargazers    int        `json:"stargazers"`
	Watchers      int        `json:"watchers"`
}

ForkeeOld - GHA Forkee structure (from before 2015) Handle missing 4 last properties (including two non-nulls!)

type IDBBatchPointsN 

type IDBBatchPointsN struct {
	Points *client.BatchPoints

	NPoints int
	// contains filtered or unexported fields
}

IDBBatchPointsN - keeps InfluxDB batch points and number of points in the current batch

type Issue 

type Issue struct {
	ID          int        `json:"id"`
	Number      int        `json:"number"`
	Comments    int        `json:"comments"`
	Title       string     `json:"title"`
	State       string     `json:"state"`
	Locked      bool       `json:"locked"`
	Body        *string    `json:"body"`
	User        Actor      `json:"user"`
	Assignee    *Actor     `json:"assignee"`
	Labels      []Label    `json:"labels"`
	Assignees   []Actor    `json:"assignees"`
	Milestone   *Milestone `json:"milestone"`
	CreatedAt   time.Time  `json:"created_at"`
	UpdatedAt   time.Time  `json:"updated_at"`
	ClosedAt    *time.Time `json:"closed_at"`
	PullRequest *Dummy     `json:"pull_request"`
}

Issue - GHA Issue structure

type Label 

type Label struct {
	ID      *int   `json:"id"`
	Name    string `json:"name"`
	Color   string `json:"color"`
	Default *bool  `json:"default"`
}

Label - GHA Label structure

type Milestone 

type Milestone struct {
	ID           int        `json:"id"`
	Name         string     `json:"name"`
	Number       int        `json:"number"`
	Title        string     `json:"title"`
	Description  *string    `json:"description"`
	Creator      *Actor     `json:"creator"`
	OpenIssues   int        `json:"open_issues"`
	ClosedIssues int        `json:"closed_issues"`
	State        string     `json:"state"`
	CreatedAt    time.Time  `json:"created_at"`
	UpdatedAt    time.Time  `json:"updated_at"`
	ClosedAt     *time.Time `json:"closed_at"`
	DueOn        *time.Time `json:"due_on"`
}

Milestone - GHA Milestone structure

type Org 

type Org struct {
	ID    int    `json:"id"`
	Login string `json:"login"`
}

Org - GHA Org structure

type Page 

type Page struct {
	SHA    string `json:"sha"`
	Action string `json:"action"`
	Title  string `json:"title"`
}

Page - GHA Page structure

type Payload 

type Payload struct {
	PushID       *int         `json:"push_id"`
	Size         *int         `json:"size"`
	Ref          *string      `json:"ref"`
	Head         *string      `json:"head"`
	Before       *string      `json:"before"`
	Action       *string      `json:"action"`
	RefType      *string      `json:"ref_type"`
	MasterBranch *string      `json:"master_branch"`
	Description  *string      `json:"description"`
	Number       *int         `json:"number"`
	Forkee       *Forkee      `json:"forkee"`
	Release      *Release     `json:"release"`
	Member       *Actor       `json:"member"`
	Issue        *Issue       `json:"issue"`
	Comment      *Comment     `json:"comment"`
	Commits      *[]Commit    `json:"commits"`
	Pages        *[]Page      `json:"pages"`
	PullRequest  *PullRequest `json:"pull_request"`
}

Payload - GHA Payload structure

type PayloadOld 

type PayloadOld struct {
	Issue        *int           `json:"issue"`
	IssueID      *int           `json:"issue_id"`
	Comment      *Comment       `json:"comment"`
	CommentID    *int           `json:"comment_id"`
	Description  *string        `json:"description"`
	MasterBranch *string        `json:"master_branch"`
	Ref          *string        `json:"ref"`
	Action       *string        `json:"action"`
	RefType      *string        `json:"ref_type"`
	Head         *string        `json:"head"`
	Size         *int           `json:"size"`
	Number       *int           `json:"number"`
	PullRequest  *PullRequest   `json:"pull_request"`
	Member       *Actor         `json:"member"`
	Release      *Release       `json:"release"`
	Pages        *[]Page        `json:"pages"`
	Commit       *string        `json:"commit"`
	SHAs         *[]interface{} `json:"shas"`
	Repository   *Forkee        `json:"repository"`
	Team         *Team          `json:"team"`
}

PayloadOld - GHA Payload structure (from before 2015)

type Project 

type Project struct {
	CommandLine      []string          `yaml:"command_line"`
	StartDate        *time.Time        `yaml:"start_date"`
	PDB              string            `yaml:"psql_db"`
	IDB              string            `yaml:"influx_db"`
	Disabled         bool              `yaml:"disabled"`
	MainRepo         string            `yaml:"main_repo"`
	AnnotationRegexp string            `yaml:"annotation_regexp"`
	Order            int               `yaml:"order"`
	JoinDate         *time.Time        `yaml:"join_date"`
	FilesSkipPattern string            `yaml:"files_skip_pattern"`
	Env              map[string]string `yaml:"env"`
}

Project contain mapping from project name to its command line used to sync it

type PullRequest 

type PullRequest struct {
	ID                  int        `json:"id"`
	Base                Branch     `json:"base"`
	Head                Branch     `json:"head"`
	User                Actor      `json:"user"`
	Number              int        `json:"number"`
	State               string     `json:"state"`
	Locked              *bool      `json:"locked"`
	Title               string     `json:"title"`
	Body                *string    `json:"body"`
	CreatedAt           time.Time  `json:"created_at"`
	UpdatedAt           time.Time  `json:"updated_at"`
	ClosedAt            *time.Time `json:"closed_at"`
	MergedAt            *time.Time `json:"merged_at"`
	MergeCommitSHA      *string    `json:"merge_commit_sha"`
	Assignee            *Actor     `json:"assignee"`
	Assignees           *[]Actor   `json:"assignees"`
	RequestedReviewers  *[]Actor   `json:"requested_reviewers"`
	Milestone           *Milestone `json:"milestone"`
	Merged              *bool      `json:"merged"`
	Mergeable           *bool      `json:"mergeable"`
	MergedBy            *Actor     `json:"merged_by"`
	MergeableState      *string    `json:"mergeable_state"`
	Rebaseable          *bool      `json:"rebaseable"`
	Comments            *int       `json:"comments"`
	ReviewComments      *int       `json:"review_comments"`
	MaintainerCanModify *bool      `json:"maintainer_can_modify"`
	Commits             *int       `json:"commits"`
	Additions           *int       `json:"additions"`
	Deletions           *int       `json:"deletions"`
	ChangedFiles        *int       `json:"changed_files"`
}

PullRequest - GHA Pull Request structure

type Release 

type Release struct {
	ID              int        `json:"id"`
	TagName         string     `json:"tag_name"`
	TargetCommitish string     `json:"target_commitish"`
	Name            *string    `json:"name"`
	Draft           bool       `json:"draft"`
	Author          Actor      `json:"author"`
	Prerelease      bool       `json:"prerelease"`
	CreatedAt       time.Time  `json:"created_at"`
	PublishedAt     *time.Time `json:"published_at"`
	Body            *string    `json:"body"`
	Assets          []Asset    `json:"assets"`
}

Release - GHA Release structure

type Repo 

type Repo struct {
	ID   int    `json:"id"`
	Name string `json:"name"`
}

Repo - GHA Repo structure

type Team 

type Team struct {
	ID         int    `json:"id"`
	Name       string `json:"name"`
	Slug       string `json:"slug"`
	Permission string `json:"permission"`
}

Team - GHA Team structure (only used before 2015)

Source Files

View all Source files

Directories

Path Synopsis
cmd
annotations
db2influx
devstats
get_repos
gha2db
gha2db_sync
idb_backup
idb_tags
import_affs
merge_pdbs
runq
structure
webhook
z2influx

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.