package module
Version: v0.6.0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: May 30, 2018 License: Apache-2.0 Imports: 26 Imported by: 0


Build Status CII Best Practices

GitHub archives and git Grafana visualization dashboards

Author: Łukasz Gryglicki

This is a toolset to visualize GitHub archives using Grafana dashboards.

GHA2DB stands for GitHub Archives to DashBoards.


We want to create a toolset for visualizing various metrics for the Kubernetes community (and also for all CNCF projects).

Everything is open source so that it can be used by other CNCF and non-CNCF open source projects.

The only requirement is that project must be hosted on a public GitHub repository/repositories.

Data hiding

If you want to hide your data (replace with anon-#) please follow instructions here.

Forking and installing locally

This toolset uses only Open Source tools: GitHub archives, GitHub API, git, Postgres databases and multiple Grafana instances. It is written in Go, and can be forked and installed by anyone.

Contributions and PRs are welcome. If you see a bug or want to add a new metric please create an issue and/or PR.

To work on this project locally please fork the original repository, and:

Please see Development for local development guide.

For more detailed description of all environment variables, tools, switches etc, please see Usage.


We want to support all kind of metrics, including historical ones. Please see requested metrics to see what kind of metrics are needed. Many of them cannot be computed based on the data sources currently used.

Repository groups

There are some groups of repositories that are grouped together as a repository groups. They are defined in scripts/kubernetes/repo_groups.sql.

To setup default repository groups:

  • PG_PASS=pwd ./kubernetes/

This is a part of kubernetes/ script and kubernetes psql dump already has groups configured.

In an 'All' project ( repository groups are mapped to individual CNCF projects scripts/all/repo_groups.sql:

Company Affiliations

We also want to have per company statistics. To implement such metrics we need a mapping of developers and their employers.

There is a project that attempts to create such mapping cncf/gitdm.

DevStats has an import tool that fetches company affiliations from cncf/gitdm and allows to create per company metrics/statistics.

If you see errors in the company affiliations, please open a pull request on cncf/gitdm and the updates will be reflected on a couple days after the PR has been accepted. Note that gitdm supports mapping based on dates, to account for developers moving between companies.


For architecture details please see architecture file.

Detailed usage is here

Adding new metrics

Please see metrics to see how to add new metrics.

Adding new projects

To add new project follow adding new project instructions.

Grafana dashboards

Please see dashboards to see list of already defined Grafana dashboards.

Exporting data

Please see exporting.

Detailed Usage instructions


The servers to run devstats are generously provided by Packet bare metal hosting as part of CNCF's Community Infrastructure Lab.

One line run all projects

  • Use GHA2DB_PROJECTS_OVERRIDE="+cncf,+all" PG_PASS=pwd devstats.
  • Or add this command using crontab -e to run every hour HH:08.

Checking projects activity

  • Use: PG_PASS=... PG_DB=allprj ./devel/ '1 month,,' > all.txt.
  • Example results here - all CNCF project activity during January 2018, excluding bots.




View Source
const DataDir string = "/etc/gha2db/"

DataDir - common constant string

View Source
const Devstats string = "devstats"

Devstats - common constant string

View Source
const EngineIsClosedError string = "engine is closed"

EngineIsClosedError - common constant string

View Source
const GHA string = "gha"

GHA - common constant string

View Source
const GHAAdmin string = "gha_admin"

GHAAdmin - common constant string

View Source
const HideCfgFile string = "hide/hide.csv"

HideCfgFile - common constant string

View Source
const LocalGitScripts string = "./git/"

LocalGitScripts - common constant string

View Source
const Localhost string = "localhost"

Localhost - common constant string

View Source
const Metrics string = "metrics/"

Metrics - common constant string

View Source
const Now string = "now"

Now - common constant string

View Source
const Null string = "null"

Null - common constant string

View Source
const Password string = "password"

Password - common constant string

View Source
const Quarter string = "quarter"

Quarter - common constant string

View Source
const Retry string = "retry"

Retry - common constant string

View Source
const TimeCol string = "time"

TimeCol - common constant string

View Source
const TimeoutError string = "{\"error\":\"timeout\"}\n"

TimeoutError - common constant string

View Source
const Today string = "today"

Today - common constant string

View Source
const Unset string = "{{unset}}"

Unset - common constant string


This section is empty.


func ActorHit added in v0.5.0

func ActorHit(ctx *Ctx, actorName string) bool

ActorHit - are we intereste din this actor?

func ActorIDOrNil

func ActorIDOrNil(actPtr *Actor) interface{}

ActorIDOrNil - return Actor ID from pointer or nil

func ActorLoginOrNil

func ActorLoginOrNil(actPtr *Actor, maybeHide func(string) string) interface{}

ActorLoginOrNil - return Actor Login from pointer or nil

func AddNIntervals

func AddNIntervals(dt time.Time, n int, nextIntervalStart, prevIntervalStart func(time.Time) time.Time) time.Time

AddNIntervals adds (using nextIntervalStart) or subtracts (using prevIntervalStart) N itervals to the given date Functions Next/Prev can use Hour, Day, Week, Month, Quarter, Year functions (defined in this module) or other custom defined functions With `func(time.Time) time.Time` signature

func AddTSPoint added in v0.5.0

func AddTSPoint(ctx *Ctx, pts *TSPoints, pt TSPoint)

AddTSPoint add single point to the batch

func BoolOrNil

func BoolOrNil(boolPtr *bool) interface{}

BoolOrNil - return either nil or value of boolPtr

func CleanUTF8

func CleanUTF8(str string) string

CleanUTF8 - clean UTF8 string to containg only Pq allowed runes

func ClearDBLogs

func ClearDBLogs()

ClearDBLogs clears logs older by defined period (in context.go) It clears logs on `devstats` database

func CommentIDOrNil

func CommentIDOrNil(commPtr *Comment) interface{}

CommentIDOrNil - return Comment ID from pointer or nil

func ComputePeriodAtThisDate

func ComputePeriodAtThisDate(ctx *Ctx, period string, dt time.Time) bool

ComputePeriodAtThisDate - for some longer periods, only recalculate them on specific dates hourly period is always calculated daily period is always calculated multiple days period are calculaded at hours: 1, 5, 9, 13, 17, 21 (UTC) annotation ranges are calculated: from last release to now - every 2 hours for past ranges only once (calculation is marked as computed) at 2 AM weekly ranges are calculated at hours: 0, 4, 8, 12, 16, 20 monthly, quarterly, yearly ranges are calculated at midnight

func CreateDatabaseIfNeeded

func CreateDatabaseIfNeeded(ctx *Ctx) bool

CreateDatabaseIfNeeded - creates requested database if not exists Returns true if database was not existing existed and created dropped

func CreateTable

func CreateTable(tdef string) string

CreateTable is used to replace DB specific parts of Create Table SQL statement

func DatabaseExists

func DatabaseExists(ctx *Ctx, closeConn bool) (exists bool, c *sql.DB)

DatabaseExists - checks if database stored in context exists If closeConn is true - then it closes connection after checking if database exists If closeConn is false, then it returns open connection to default database "postgres"

func DayStart

func DayStart(dt time.Time) time.Time

DayStart - return time rounded to current day start

func DescriblePeriodInHours

func DescriblePeriodInHours(hrs float64) (desc string)

DescriblePeriodInHours - return string description of a time period given in hours

func DropDatabaseIfExists

func DropDatabaseIfExists(ctx *Ctx) bool

DropDatabaseIfExists - drops requested database if exists Returns true if database existed and was dropped

func EnvReplace added in v0.4.0

func EnvReplace(prefix, suffix string) map[string]string

EnvReplace - replace all environment variables starting with "prefix" with contents of variables with "suffix" added - if defined If prefix is "DB_" and suffix is "_SRC" then: So if there is "DB_HOST_SRC" variable defined - it will replace "DB_HOST" and so on

func EnvRestore added in v0.4.0

func EnvRestore(env map[string]string)

EnvRestore - restores all environment variables given in the map

func ExecCommand

func ExecCommand(ctx *Ctx, cmdAndArgs []string, env map[string]string) (string, error)

ExecCommand - execute command given by array of strings with eventual environment map

func ExecSQL

func ExecSQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) (sql.Result, error)

ExecSQL executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed)

func ExecSQLTx

func ExecSQLTx(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) (sql.Result, error)

ExecSQLTx executes given SQL on Postgres DB (and return single state result, that doesn't need to be closed) It is for running inside transaction

func ExecSQLTxWithErr

func ExecSQLTxWithErr(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) sql.Result

ExecSQLTxWithErr wrapper to ExecSQLTx that exists on error It is for running inside transaction

func ExecSQLWithErr

func ExecSQLWithErr(con *sql.DB, ctx *Ctx, query string, args ...interface{}) sql.Result

ExecSQLWithErr wrapper to ExecSQL that exists on error

func FatalNoLog added in v0.5.0

func FatalNoLog(err error) string

FatalNoLog displays error message (if error present) and exits program, should be used for very early init state

func FatalOnError

func FatalOnError(err error) string

FatalOnError displays error message (if error present) and exits program

func Fatalf added in v0.3.0

func Fatalf(f string, a ...interface{})

Fatalf - it will call FatalOnError using fmt.Errorf with args provided

func FirstIntOrNil

func FirstIntOrNil(intPtrs []*int) interface{}

FirstIntOrNil - return either nil or value of intPtr

func ForkeeIDOrNil

func ForkeeIDOrNil(forkPtr *Forkee) interface{}

ForkeeIDOrNil - return Forkee ID from pointer or nil

func ForkeeNameOrNil

func ForkeeNameOrNil(forkPtr *Forkee) interface{}

ForkeeNameOrNil - return Forkee Name from pointer or nil

func ForkeeOldIDOrNil

func ForkeeOldIDOrNil(forkPtr *ForkeeOld) interface{}

ForkeeOldIDOrNil - return ForkeeOld ID from pointer or nil

func GHClient added in v0.5.0

func GHClient(ctx *Ctx) (ghCtx context.Context, client *github.Client)

GHClient - get GitHub client

func GetHidden added in v0.6.0

func GetHidden(configFile string) map[string]string

GetHidden - return list of shas to replace

func GetIntervalFunctions

func GetIntervalFunctions(intervalAbbr string, allowUnknown bool) (interval string, n int, intervalStart, nextIntervalStart, prevIntervalStart func(time.Time) time.Time)

GetIntervalFunctions - return interval name, interval number, interval start, next, prev function from interval abbr: h|d2|w3|m4|q|y w3 = 3 weeks, q2 = 2 quarters, y = year (1), d7 = 7 days (not the same as w), m3 = 3 months (not the same as q)

func GetRateLimits added in v0.5.0

func GetRateLimits(gctx context.Context, gc *github.Client, core bool) (int, int, time.Duration)

GetRateLimits - returns all and remaining API points and duration to wait for reset when core=true - returns Core limits, when core=false returns Search limits

func GetTagValues

func GetTagValues(con *sql.DB, ctx *Ctx, name, key string) (ret []string)

GetTagValues returns tag values for a given key

func GetThreadsNum

func GetThreadsNum(ctx *Ctx) int

GetThreadsNum returns the number of available CPUs If environment variable GHA2DB_ST is set it retuns 1 It can be used to debug single threaded verion It runs on 95% CPU power by default

func HashStrings

func HashStrings(strs []string) int

HashStrings - returns unique Hash for strings array This value is supposed to be used as ID (negative) to mark it was artificially generated

func HourStart

func HourStart(dt time.Time) time.Time

HourStart - return time rounded to current hour start

func InsertIgnore

func InsertIgnore(query string) string

InsertIgnore - will return insert statement with ignore option specific for DB

func IntOrNil

func IntOrNil(intPtr *int) interface{}

IntOrNil - return either nil or value of intPtr

func IsProjectDisabled

func IsProjectDisabled(ctx *Ctx, proj string, yamlDisabled bool) bool

IsProjectDisabled - checks if project is disabled or not: fullName comes from makeOldRepoName for pre-2015 data! yamlDisabled (this is from projects.yaml - can be true or false) it also checks context (which can override `disabled: true` from projects.yaml) +pro1,-pro2 creates map {"pro1":true, "pro2":false}

func IssueIDOrNil

func IssueIDOrNil(issuePtr *Issue) interface{}

IssueIDOrNil - return Issue ID from pointer or nil

func MakeOldRepoName

func MakeOldRepoName(repo *ForkeeOld) string

MakeOldRepoName - before 2015 repository name should be Organization/Name (if Organization present) or just Name

func MakeUniqueSort

func MakeUniqueSort(ary []string) (outAry []string)

MakeUniqueSort - make string array unique & sorted

func MaybeHideFunc added in v0.6.0

func MaybeHideFunc(shas map[string]string) func(string) string

MaybeHideFunc - use closure as a data storage

func Mgetc

func Mgetc(ctx *Ctx) string

Mgetc waits for single key press and return character pressed

func MilestoneIDOrNil

func MilestoneIDOrNil(milPtr *Milestone) interface{}

MilestoneIDOrNil - return Milestone ID from pointer or nil

func MonthStart

func MonthStart(dt time.Time) time.Time

MonthStart - return time rounded to current month start

func NValue

func NValue(index int) string

NValue will return $n

func NValues

func NValues(n int) string

NValues will return values($1, $2, .., $n)

func NegatedBoolOrNil

func NegatedBoolOrNil(boolPtr *bool) interface{}

NegatedBoolOrNil - return either nil or negated value of boolPtr

func NextDayStart

func NextDayStart(dt time.Time) time.Time

NextDayStart - return time rounded to next day start

func NextHourStart

func NextHourStart(dt time.Time) time.Time

NextHourStart - return time rounded to next hour start

func NextMonthStart

func NextMonthStart(dt time.Time) time.Time

NextMonthStart - return time rounded to next month start

func NextQuarterStart

func NextQuarterStart(dt time.Time) time.Time

NextQuarterStart - return time rounded to next quarter start

func NextWeekStart

func NextWeekStart(dt time.Time) time.Time

NextWeekStart - return time rounded to next week start

func NextYearStart

func NextYearStart(dt time.Time) time.Time

NextYearStart - return time rounded to next year start

func NormalizeName

func NormalizeName(str string) string

NormalizeName - clean DB string from -, /, ., " ",), (, ], [ trim leading and trailing space, lowercase Normalize Unicode characters

func OrgIDOrNil

func OrgIDOrNil(orgPtr *Org) interface{}

OrgIDOrNil - return Org ID from pointer or nil

func OrgLoginOrNil

func OrgLoginOrNil(orgPtr *Org) interface{}

OrgLoginOrNil - return Org ID from pointer or nil

func PgConn

func PgConn(ctx *Ctx) *sql.DB

PgConn Connects to Postgres database

func PgConnDB

func PgConnDB(ctx *Ctx, dbName string) *sql.DB

PgConnDB Connects to Postgres database (with specific DB name) uses database 'dbname' instead of 'PgDB'

func PrepareQuickRangeQuery

func PrepareQuickRangeQuery(sql, period, from, to string) string

PrepareQuickRangeQuery Perpares query using either ready `period` string or using `from` and `to` strings Values to replace are specially encoded {{period:alias.column}} Can either replace with: (alias.column >= now() - 'period'::interval) Or (alias.column >= 'from' and alias.column < 'to')

func PrettyPrintJSON

func PrettyPrintJSON(jsonBytes []byte) []byte

PrettyPrintJSON - pretty formats raw JSON bytes

func PrevDayStart

func PrevDayStart(dt time.Time) time.Time

PrevDayStart - return time rounded to prev day start

func PrevHourStart

func PrevHourStart(dt time.Time) time.Time

PrevHourStart - return time rounded to prev hour start

func PrevMonthStart

func PrevMonthStart(dt time.Time) time.Time

PrevMonthStart - return time rounded to prev month start

func PrevQuarterStart

func PrevQuarterStart(dt time.Time) time.Time

PrevQuarterStart - return time rounded to prev quarter start

func PrevWeekStart

func PrevWeekStart(dt time.Time) time.Time

PrevWeekStart - return time rounded to prev week start

func PrevYearStart

func PrevYearStart(dt time.Time) time.Time

PrevYearStart - return time rounded to prev year start

func Printf

func Printf(format string, args ...interface{}) (n int, err error)

Printf is a wrapper around Printf(...) that supports logging.

func ProcessAnnotations

func ProcessAnnotations(ctx *Ctx, annotations *Annotations, startDate, joinDate *time.Time)

ProcessAnnotations Creates IfluxDB annotations and quick_series

func ProcessTag added in v0.5.0

func ProcessTag(con *sql.DB, ctx *Ctx, tg *Tag, replaces [][]string)

ProcessTag - insert given Tag into Postgres TSDB

func ProgressInfo

func ProgressInfo(i, n int, start time.Time, last *time.Time, period time.Duration, msg string)

ProgressInfo display info about progress: i/n if current time >= last + period If displayed info, update last

func PullRequestIDOrNil

func PullRequestIDOrNil(prPtr *PullRequest) interface{}

PullRequestIDOrNil - return PullRequest ID from pointer or nil

func QuarterStart

func QuarterStart(dt time.Time) time.Time

QuarterStart - return time rounded to current month start

func QueryRowSQL

func QueryRowSQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) *sql.Row

QueryRowSQL executes given SQL on Postgres DB (and returns single row)

func QueryRowSQLTx added in v0.5.0

func QueryRowSQLTx(tx *sql.Tx, ctx *Ctx, query string, args ...interface{}) *sql.Row

QueryRowSQLTx executes given SQL on Postgres DB (and returns single row)

func QuerySQL

func QuerySQL(con *sql.DB, ctx *Ctx, query string, args ...interface{}) (*sql.Rows, error)

QuerySQL executes given SQL on Postgres DB (and returns rowset that needs to be closed)

func QuerySQLTx

func QuerySQLTx(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) (*sql.Rows, error)

QuerySQLTx executes given SQL on Postgres DB (and returns rowset that needs to be closed) It is for running inside transaction

func QuerySQLTxWithErr

func QuerySQLTxWithErr(con *sql.Tx, ctx *Ctx, query string, args ...interface{}) *sql.Rows

QuerySQLTxWithErr wrapper to QuerySQLTx that exists on error It is for running inside transaction

func QuerySQLWithErr

func QuerySQLWithErr(con *sql.DB, ctx *Ctx, query string, args ...interface{}) *sql.Rows

QuerySQLWithErr wrapper to QuerySQL that exists on error

func ReadFile added in v0.5.0

func ReadFile(ctx *Ctx, path string) ([]byte, error)

ReadFile tries to read any filename, but have a fallback it attempts to replace current project name with shared: /proj/ -> /shared/ This is to allow reading files that can be shared between projects

func ReleaseIDOrNil

func ReleaseIDOrNil(relPtr *Release) interface{}

ReleaseIDOrNil - return Release ID from pointer or nil

func RepoHit

func RepoHit(ctx *Ctx, fullName string, forg, frepo map[string]struct{}) bool

RepoHit - are we interested in this org/repo ?

func RepoIDOrNil

func RepoIDOrNil(repoPtr *Repo) interface{}

RepoIDOrNil - return Repo ID from pointer or nil

func RepoNameOrNil

func RepoNameOrNil(repoPtr *Repo) interface{}

RepoNameOrNil - return Repo Name from pointer or nil

func SkipEmpty

func SkipEmpty(arr []string) []string

SkipEmpty - skip one element arrays contining only empty string This is what strings.Split() returns for empty input We expect empty array or empty map returned in such cases

func Slugify added in v0.5.0

func Slugify(arg string) string

Slugify replace all whitespace with "-", remove all non-word letters downcase

func StringOrNil

func StringOrNil(strPtr *string) interface{}

StringOrNil - return either nil or value of strPtr

func StringsMapToArray

func StringsMapToArray(f func(string) string, strArr []string) []string

StringsMapToArray this is a function that calls given function for all array items and returns array of items processed by this func Example call: lib.StringsMapToArray(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "})

func StringsMapToSet

func StringsMapToSet(f func(string) string, strArr []string) map[string]struct{}

StringsMapToSet this is a function that calls given function for all array items and returns set of items processed by this func Example call: lib.StringsMapToSet(func(x string) string { return strings.TrimSpace(x) }, []string{" a", " b ", "c "})

func StringsSetKeys

func StringsSetKeys(set map[string]struct{}) []string

StringsSetKeys - returns all keys from string map

func StripUnicode

func StripUnicode(str string) string

StripUnicode strip non-unicode and control characters from a string From:

func Structure

func Structure(ctx *Ctx)

Structure creates full database structure, indexes, views/summary tables etc

func TableColumnExists added in v0.5.0

func TableColumnExists(con *sql.DB, ctx *Ctx, tableName, columnName string) bool

TableColumnExists - checks if a given table's has a given column

func TableExists added in v0.5.0

func TableExists(con *sql.DB, ctx *Ctx, tableName string) bool

TableExists - checks if a given table exists

func TimeOrNil

func TimeOrNil(timePtr *time.Time) interface{}

TimeOrNil - return either nil or value of timePtr

func TimeParseAny

func TimeParseAny(dtStr string) time.Time

TimeParseAny - attempts to parse time from string YYYY-MM-DD HH:MI:SS Skipping parts from right until only YYYY id left

func ToGHADate

func ToGHADate(dt time.Time) string

ToGHADate - return time formatted as YYYY-MM-DD-H

func ToYMDDate

func ToYMDDate(dt time.Time) string

ToYMDDate - return time formatted as YYYY-MM-DD

func ToYMDHDate

func ToYMDHDate(dt time.Time) string

ToYMDHDate - return time formatted as YYYY-MM-DD HH

func ToYMDHMSDate

func ToYMDHMSDate(dt time.Time) string

ToYMDHMSDate - return time formatted as YYYY-MM-DD HH:MI:SS

func TruncStringOrNil

func TruncStringOrNil(strPtr *string, maxLen int) interface{}

TruncStringOrNil - return either nil or value of strPtr truncated to maxLen chars

func TruncToBytes

func TruncToBytes(str string, size int) string

TruncToBytes - truncates text to <= size bytes (note that this can be a lot less UTF-8 runes)

func WeekStart

func WeekStart(dt time.Time) time.Time

WeekStart - return time rounded to current week start Assumes first week day is Sunday

func WriteTSPoints added in v0.5.0

func WriteTSPoints(ctx *Ctx, con *sql.DB, pts *TSPoints, mergeSeries string, mut *sync.Mutex)

WriteTSPoints write batch of points to postgresql use mergeSeries = "name" to put all series in "name" table, and create "series" column that conatins all point names.

without merge, alee names will create separate tables.

use non-null mut when you are using this function from multiple threads that write to the same series name at the same time

use non-null mut only then.

No more giant lock approach here, but it is up to user to spcify call context, especially 2 last parameters!

func YearStart

func YearStart(dt time.Time) time.Time

YearStart - return time rounded to current month start


type Actor

type Actor struct {
	ID    int    `json:"id"`
	Login string `json:"login"`
	Name  string `json:"-"`

Actor - GHA Actor structure Name is unexported and not used by JSON load/save But is used when importing affiliations from cncf/gitdm:github_users.json

type AllProjects

type AllProjects struct {
	Projects map[string]Project `yaml:"projects"`

AllProjects contain all projects data

type Annotation

type Annotation struct {
	Name        string
	Description string
	Date        time.Time

Annotation contain each annotation data

type Annotations

type Annotations struct {
	Annotations []Annotation

Annotations contain list of annotations

func GetAnnotations

func GetAnnotations(ctx *Ctx, orgRepo, annoRegexp string) (annotations Annotations)

GetAnnotations queries uses `git` to get `orgRepo` all tags list for all tags and returns those matching `annoRegexp`

func GetFakeAnnotations

func GetFakeAnnotations(startDate, joinDate time.Time) (annotations Annotations)

GetFakeAnnotations - returns 'startDate - joinDate' and 'joinDate - now' annotations

type AnnotationsByDate

type AnnotationsByDate []Annotation

AnnotationsByDate annotations Sort interface

func (AnnotationsByDate) Len

func (a AnnotationsByDate) Len() int

func (AnnotationsByDate) Less

func (a AnnotationsByDate) Less(i, j int) bool

func (AnnotationsByDate) Swap

func (a AnnotationsByDate) Swap(i, j int)

type AnyArray

type AnyArray []interface{}

AnyArray - holds array of interface{} - just a shortcut

type Asset

type Asset struct {
	ID            int       `json:"id"`
	CreatedAt     time.Time `json:"created_at"`
	UpdatedAt     time.Time `json:"updated_at"`
	Name          string    `json:"name"`
	Label         *string   `json:"label"`
	Uploader      Actor     `json:"uploader"`
	ContentType   string    `json:"content_type"`
	State         string    `json:"state"`
	Size          int       `json:"size"`
	DownloadCount int       `json:"download_count"`

Asset - GHA Asset structure

type Author

type Author struct {
	Name  string `json:"name"`
	Email string `json:"email"`

Author - GHA Commit Author structure

type Branch

type Branch struct {
	SHA   string  `json:"sha"`
	User  *Actor  `json:"user"`
	Repo  *Forkee `json:"repo"` // This is confusing, but actually GHA has "repo" fields that holds "forkee" structure
	Label string  `json:"label"`
	Ref   string  `json:"ref"`

Branch - GHA Branch structure

type Comment

type Comment struct {
	ID                  int       `json:"id"`
	Body                string    `json:"body"`
	CreatedAt           time.Time `json:"created_at"`
	UpdatedAt           time.Time `json:"updated_at"`
	User                Actor     `json:"user"`
	CommitID            *string   `json:"commit_id"`
	OriginalCommitID    *string   `json:"original_commit_id"`
	DiffHunk            *string   `json:"diff_hunk"`
	Position            *int      `json:"position"`
	OriginalPosition    *int      `json:"original_position"`
	Path                *string   `json:"path"`
	PullRequestReviewID *int      `json:"pull_request_review_id"`
	Line                *int      `json:"line"`

Comment - GHA Comment structure

type Commit

type Commit struct {
	SHA      string `json:"sha"`
	Author   Author `json:"author"`
	Message  string `json:"message"`
	Distinct bool   `json:"distinct"`

Commit - GHA Commit structure

type Ctx

type Ctx struct {
	Debug               int             // From GHA2DB_DEBUG Debug level: 0-no, 1-info, 2-verbose, including SQLs, default 0
	CmdDebug            int             // From GHA2DB_CMDDEBUG Commands execution Debug level: 0-no, 1-only output commands, 2-output commands and their output, 3-output full environment as well, default 0
	JSONOut             bool            // From GHA2DB_JSON gha2db: write JSON files? default false
	DBOut               bool            // From GHA2DB_NODB gha2db: write to SQL database, default true
	ST                  bool            // From GHA2DB_ST true: use single threaded version, false: use multi threaded version, default false
	NCPUs               int             // From GHA2DB_NCPUS, set to override number of CPUs to run, this overwrites GHA2DB_ST, default 0 (which means do not use it)
	PgHost              string          // From PG_HOST, default "localhost"
	PgPort              string          // From PG_PORT, default "5432"
	PgDB                string          // From PG_DB, default "gha"
	PgUser              string          // From PG_USER, default "gha_admin"
	PgPass              string          // From PG_PASS, default "password"
	PgSSL               string          // From PG_SSL, default "disable"
	Index               bool            // From GHA2DB_INDEX Create DB index? default false
	Table               bool            // From GHA2DB_SKIPTABLE Create table structure? default true
	Tools               bool            // From GHA2DB_SKIPTOOLS Create DB tools (like views, summary tables, materialized views etc)? default true
	Mgetc               string          // From GHA2DB_MGETC Character returned by mgetc (if non empty), default ""
	QOut                bool            // From GHA2DB_QOUT output all SQL queries?, default false
	CtxOut              bool            // From GHA2DB_CTXOUT output all context data (this struct), default false
	LogTime             bool            // From GHA2DB_SKIPTIME, output time with all lib.Printf(...) calls, default true, use GHA2DB_SKIPTIME to disable
	DefaultStartDate    time.Time       // From GHA2DB_STARTDT, default `2014-01-01 00:00 UTC`, expects format "YYYY-MM-DD HH:MI:SS", can be set in `projects.yaml` via `start_date:`, value from projects.yaml (if set) has the highest priority.
	ForceStartDate      bool            // From GHA2DB_STARTDT_FORCE, default false
	LastSeries          string          // From GHA2DB_LASTSERIES, use this TSDB series to determine last timestamp date, default "events_h"
	SkipTSDB            bool            // From GHA2DB_SKIPTSDB gha2db_sync tool, skip TS DB processing? for calc_metric it skips final series write, default false
	SkipPDB             bool            // From GHA2DB_SKIPPDB gha2db_sync tool, skip Postgres DB processing? default false
	ResetTSDB           bool            // From GHA2DB_RESETTSDB sync tool, regenerate all TS points? default false
	ResetRanges         bool            // From GHA2DB_RESETRANGES sync tool, regenerate all past quick ranges? default false
	Explain             bool            // From GHA2DB_EXPLAIN runq tool, prefix query with "explain " - it will display query plan instead of executing real query, default false
	OldFormat           bool            // From GHA2DB_OLDFMT gha2db tool, if set then use pre 2015 GHA JSONs format
	Exact               bool            // From GHA2DB_EXACT gha2db tool, if set then orgs list provided from commandline is used as a list of exact repository full names, like "a/b,c/d,e", if not only full names "a/b,x/y" can be treated like this, names without "/" are either orgs or repos.
	LogToDB             bool            // From GHA2DB_SKIPLOG all tools, if set, DB logging into Postgres table `gha_logs` in `devstats` database will be disabled
	Local               bool            // From GHA2DB_LOCAL gha2db_sync tool, if set, gha2_db will call other tools prefixed with "./" to use local compile ones. Otherwise it will call binaries without prefix (so it will use thos ein /usr/bin/).
	MetricsYaml         string          // From GHA2DB_METRICS_YAML gha2db_sync tool, set other metrics.yaml file, default is "metrics/{{project}}metrics.yaml"
	TagsYaml            string          // From GHA2DB_TAGS_YAML tags tool, set other tags.yaml file, default is "metrics/{{project}}/tags.yaml"
	ColumnsYaml         string          // From GHA2DB_COLUMNS_YAML tags tool, set other columns.yaml file, default is "metrics/{{project}}/columns.yaml"
	VarsYaml            string          // From GHA2DB_VARS_YAML db_vars tool, set other vars.yaml file, default is "metrics/{{project}}/vars.yaml"
	GitHubOAuth         string          // From GHA2DB_GITHUB_OAUTH ghapi2db tool, if not set reads from /etc/github/oauth file, set to "-" to force public access.
	ClearDBPeriod       string          // From GHA2DB_MAXLOGAGE gha2db_sync tool, maximum age of devstats.gha_logs entries, default "1 week"
	Trials              []int           // From GHA2DB_TRIALS, all Postgres related tools, retry periods for "too many connections open" error
	WebHookRoot         string          // From GHA2DB_WHROOT, webhook tool, default "/hook", must match .travis.yml notifications webhooks
	WebHookPort         string          // From GHA2DB_WHPORT, webhook tool, default ":1982", note that webhook listens using http:1982, but we use apache on https:2982 (to enable https protocol and proxy requests to http:1982)
	WebHookHost         string          // From GHA2DB_WHHOST, webhook tool, default "" (this can be localhost to disable access by IP, we use Apache proxy to enable https and then apache only need
	CheckPayload        bool            // From GHA2DB_SKIP_VERIFY_PAYLOAD, webhook tool, default true, use GHA2DB_SKIP_VERIFY_PAYLOAD=1 to manually test payloads
	FullDeploy          bool            // From GHA2DB_SKIP_FULL_DEPLOY, webhook tool, default true, use GHA2DB_SKIP_FULL_DEPLOY=1 to ignore "[deploy]" requests that call `./devel/`.
	DeployBranches      []string        // From GHA2DB_DEPLOY_BRANCHES, webhook tool, default "master" - comma separated list
	DeployStatuses      []string        // From GHA2DB_DEPLOY_STATUSES, webhook tool, default "Passed,Fixed", - comma separated list
	DeployResults       []int           // From GHA2DB_DEPLOY_RESULTS, webhook tool, default "0", - comma separated list
	DeployTypes         []string        // From GHA2DB_DEPLOY_TYPES, webhook tool, default "push", - comma separated list
	ProjectRoot         string          // From GHA2DB_PROJECT_ROOT, webhook tool, no default, must be specified to run webhook tool
	ExecFatal           bool            // default true, set this manually to false to avoid lib.ExecCommand calling os.Exit() on failure and return error instead
	ExecQuiet           bool            // default false, set this manually to true to have quite exec failures (for example `get_repos` git-clones or git-pulls on errors).
	ExecOutput          bool            // default false, set to true to capture commands STDOUT
	Project             string          // From GHA2DB_PROJECT, gha2db_sync default "", You should set it to something like "kubernetes", "prometheus" etc.
	TestsYaml           string          // From GHA2DB_TESTS_YAML ./ tool, set other tests.yaml file, default is "tests.yaml"
	ReposDir            string          // From GHA2DB_REPOS_DIR get_repos tool, default "~/devstats_repos/"
	ProcessRepos        bool            // From GHA2DB_PROCESS_REPOS get_repos tool, enable processing (cloning/pulling) all devstats repos, default false
	ProcessCommits      bool            // From GHA2DB_PROCESS_COMMITS get_repos tool, enable update/create mapping table: commit - list of file that commit refers to, default false
	ExternalInfo        bool            // From GHA2DB_EXTERNAL_INFO get_repos tool, enable outputing data needed by external tools (cncf/gitdm), default false
	ProjectsCommits     string          // From GHA2DB_PROJECTS_COMMITS get_repos tool, set list of projects for commits analysis instead of analysing all, default "" - means all
	ProjectsYaml        string          // From GHA2DB_PROJECTS_YAML, many tools - set main projects file, default "projects.yaml"
	ProjectsOverride    map[string]bool // From GHA2DB_PROJECTS_OVERRIDE, get_repos and ./devstats tools - for example "-pro1,+pro2" means never sync pro1 and always sync pro2 (even if disabled in `projects.yaml`).
	ExcludeRepos        map[string]bool // From GHA2DB_EXCLUDE_REPOS, gha2db tool, default "" - comma separated list of repos to exclude, example: "theupdateframework/notary,theupdateframework/other"
	InputDBs            []string        // From GHA2DB_INPUT_DBS, merge_dbs tool - list of input databases to merge, order matters - first one will insert on a clean DB, next will do insert ignore (to avoid constraints failure due to common data)
	OutputDB            string          // From GHA2DB_OUTPUT_DB, merge_dbs tool - output database to merge into
	TmOffset            int             // From GHA2DB_TMOFFSET, gha2db_sync tool - uses time offset to decide when to calculate various metrics, default offset is 0 which means UTC, good offset for USA is -6, and for Poland is 1 or 2
	DefaultHostname     string          // ""
	RecentRange         string          // From GHA2DB_RECENT_RANGE, ghapi2db tool, default '2 hours'. This is a recent period to check open issues/PR to fix their labels and milestones.
	MinGHAPIPoints      int             // From GHA2DB_MIN_GHAPI_POINTS, ghapi2db tool, minimum GitHub API points, before waiting for reset.
	MaxGHAPIWaitSeconds int             // From GHA2DB_MAX_GHAPI_WAIT, ghapi2db tool, maximum wait time for GitHub API points reset (in seconds).
	MaxGHAPIRetry       int             // From GHA2DB_MAX_GHAPI_RETRY, ghapi2db tool, maximum wait retries
	SkipGHAPI           bool            // From GHA2DB_GHAPISKIP, ghapi2db tool, if set then tool is not creating artificial events using GitHub API
	SkipArtificailClean bool            // From GHA2DB_AECLEANSKIP, ghapi2db tool, if set then tool is not attempting to clean unneeded artificial events
	SkipGetRepos        bool            // From GHA2DB_GETREPOSSKIP, get_repos tool, if set then tool does nothing
	OnlyIssues          []int64         // From GHA2DB_ONLY_ISSUES, ghapi2db tool, process a user provided list of issues "issue_id1,issue_id2,...,issue_idN", default "". This is for GH API debugging.
	OnlyEvents          []int64         // From GHA2DB_ONLY_EVENTS, ghapi2db tool, process a user provided list of events "event_id1,event_id2,...,event_idN", default "". This is for artificial events cleanup debugging.
	CSVFile             string          // From GHA2DB_CSVOUT, runq tool, if set, saves result in this file
	ComputeAll          bool            // From GHA2DB_COMPUTE_ALL, all tools, if set then no period decisions are taken based on time, but all possible periods are recalculated
	ActorsFilter        bool            // From GHA2DB_ACTORS_FILTER gha2db tool, if enabled then actor filterning will be added, default false
	ActorsAllow         *regexp.Regexp  // From GHA2DB_ACTORS_ALLOW, gha2db tool, process JSON if actor matches this regexp, default ""
	ActorsForbid        *regexp.Regexp  // From GHA2DB_ACTORS_FORBID, gha2db tool, process JSON if actor matches this regexp, default ""
	OnlyMetrics         map[string]bool // From GHA2DB_ONLY_METRICS, gha2db_sync tool, default "" - comma separated list of metrics to process, as fiven my "sql: name" in the "metrics.yaml" file. Only those metrics will be calculated.

Ctx - environment context packed in structure

func (*Ctx) Init

func (ctx *Ctx) Init()

Init - get context from environment variables

func (*Ctx) Print

func (ctx *Ctx) Print()

Print context contents

type Dummy

type Dummy struct{}

Dummy - structure with no data pointer to this struct is used to test if such field was present in JSON or not

type Event

type Event struct {
	ID        string    `json:"id"`
	Type      string    `json:"type"`
	Public    bool      `json:"public"`
	CreatedAt time.Time `json:"created_at"`
	Actor     Actor     `json:"actor"`
	Repo      Repo      `json:"repo"`
	Org       *Org      `json:"org"`
	Payload   Payload   `json:"payload"`

Event - full GHA (GitHub Archive) event structure

type EventOld

type EventOld struct {
	ID         string      `json:"-"`
	Type       string      `json:"type"`
	Public     bool        `json:"public"`
	CreatedAt  time.Time   `json:"created_at"`
	Actor      string      `json:"actor"`
	Repository ForkeeOld   `json:"repository"`
	Payload    *PayloadOld `json:"payload"`

EventOld - full GHA (GitHub Archive) event structure, before 2015

type Forkee

type Forkee struct {
	ID              int        `json:"id"`
	Name            string     `json:"name"`
	FullName        string     `json:"full_name"`
	Owner           Actor      `json:"owner"`
	Description     *string    `json:"description"`
	Public          *bool      `json:"public"`
	Fork            bool       `json:"fork"`
	CreatedAt       time.Time  `json:"created_at"`
	UpdatedAt       time.Time  `json:"updated_at"`
	PushedAt        *time.Time `json:"pushed_at"`
	Homepage        *string    `json:"homepage"`
	Size            int        `json:"size"`
	StargazersCount int        `json:"stargazers_count"`
	HasIssues       bool       `json:"has_issues"`
	HasProjects     *bool      `json:"has_projects"`
	HasDownloads    bool       `json:"has_downloads"`
	HasWiki         bool       `json:"has_wiki"`
	HasPages        *bool      `json:"has_pages"`
	Forks           int        `json:"forks"`
	OpenIssues      int        `json:"open_issues"`
	Watchers        int        `json:"watchers"`
	DefaultBranch   string     `json:"default_branch"`

Forkee - GHA Forkee structure

type ForkeeOld

type ForkeeOld struct {
	ID            int        `json:"id"`
	CreatedAt     time.Time  `json:"created_at"`
	Description   *string    `json:"description"`
	Fork          bool       `json:"fork"`
	Forks         int        `json:"forks"`
	HasDownloads  bool       `json:"has_downloads"`
	HasIssues     bool       `json:"has_issues"`
	HasWiki       bool       `json:"has_wiki"`
	Homepage      *string    `json:"homepage"`
	Language      *string    `json:"language"`
	DefaultBranch string     `json:"master_branch"`
	Name          string     `json:"name"`
	OpenIssues    int        `json:"open_issues"`
	Organization  *string    `json:"organization"`
	Owner         string     `json:"owner"`
	Private       *bool      `json:"private"`
	PushedAt      *time.Time `json:"pushed_at"`
	Size          int        `json:"size"`
	Stargazers    int        `json:"stargazers"`
	Watchers      int        `json:"watchers"`

ForkeeOld - GHA Forkee structure (from before 2015) Handle missing 4 last properties (including two non-nulls!)

type Int64Ary added in v0.5.0

type Int64Ary []int64

Int64Ary - sortable Int64 array

func (Int64Ary) Len added in v0.5.0

func (a Int64Ary) Len() int

func (Int64Ary) Less added in v0.5.0

func (a Int64Ary) Less(i, j int) bool

func (Int64Ary) Swap added in v0.5.0

func (a Int64Ary) Swap(i, j int)

type Issue

type Issue struct {
	ID          int        `json:"id"`
	Number      int        `json:"number"`
	Comments    int        `json:"comments"`
	Title       string     `json:"title"`
	State       string     `json:"state"`
	Locked      bool       `json:"locked"`
	Body        *string    `json:"body"`
	User        Actor      `json:"user"`
	Assignee    *Actor     `json:"assignee"`
	Labels      []Label    `json:"labels"`
	Assignees   []Actor    `json:"assignees"`
	Milestone   *Milestone `json:"milestone"`
	CreatedAt   time.Time  `json:"created_at"`
	UpdatedAt   time.Time  `json:"updated_at"`
	ClosedAt    *time.Time `json:"closed_at"`
	PullRequest *Dummy     `json:"pull_request"`

Issue - GHA Issue structure

type Label

type Label struct {
	ID      *int   `json:"id"`
	Name    string `json:"name"`
	Color   string `json:"color"`
	Default *bool  `json:"default"`

Label - GHA Label structure

type Milestone

type Milestone struct {
	ID           int        `json:"id"`
	Name         string     `json:"name"`
	Number       int        `json:"number"`
	Title        string     `json:"title"`
	Description  *string    `json:"description"`
	Creator      *Actor     `json:"creator"`
	OpenIssues   int        `json:"open_issues"`
	ClosedIssues int        `json:"closed_issues"`
	State        string     `json:"state"`
	CreatedAt    time.Time  `json:"created_at"`
	UpdatedAt    time.Time  `json:"updated_at"`
	ClosedAt     *time.Time `json:"closed_at"`
	DueOn        *time.Time `json:"due_on"`

Milestone - GHA Milestone structure

type Org

type Org struct {
	ID    int    `json:"id"`
	Login string `json:"login"`

Org - GHA Org structure

type Page

type Page struct {
	SHA    string `json:"sha"`
	Action string `json:"action"`
	Title  string `json:"title"`

Page - GHA Page structure

type Payload

type Payload struct {
	PushID       *int         `json:"push_id"`
	Size         *int         `json:"size"`
	Ref          *string      `json:"ref"`
	Head         *string      `json:"head"`
	Before       *string      `json:"before"`
	Action       *string      `json:"action"`
	RefType      *string      `json:"ref_type"`
	MasterBranch *string      `json:"master_branch"`
	Description  *string      `json:"description"`
	Number       *int         `json:"number"`
	Forkee       *Forkee      `json:"forkee"`
	Release      *Release     `json:"release"`
	Member       *Actor       `json:"member"`
	Issue        *Issue       `json:"issue"`
	Comment      *Comment     `json:"comment"`
	Commits      *[]Commit    `json:"commits"`
	Pages        *[]Page      `json:"pages"`
	PullRequest  *PullRequest `json:"pull_request"`

Payload - GHA Payload structure

type PayloadOld

type PayloadOld struct {
	Issue        *int           `json:"issue"`
	IssueID      *int           `json:"issue_id"`
	Comment      *Comment       `json:"comment"`
	CommentID    *int           `json:"comment_id"`
	Description  *string        `json:"description"`
	MasterBranch *string        `json:"master_branch"`
	Ref          *string        `json:"ref"`
	Action       *string        `json:"action"`
	RefType      *string        `json:"ref_type"`
	Head         *string        `json:"head"`
	Size         *int           `json:"size"`
	Number       *int           `json:"number"`
	PullRequest  *PullRequest   `json:"pull_request"`
	Member       *Actor         `json:"member"`
	Release      *Release       `json:"release"`
	Pages        *[]Page        `json:"pages"`
	Commit       *string        `json:"commit"`
	SHAs         *[]interface{} `json:"shas"`
	Repository   *Forkee        `json:"repository"`
	Team         *Team          `json:"team"`

PayloadOld - GHA Payload structure (from before 2015)

type Project

type Project struct {
	CommandLine      []string          `yaml:"command_line"`
	StartDate        *time.Time        `yaml:"start_date"`
	PDB              string            `yaml:"psql_db"`
	Disabled         bool              `yaml:"disabled"`
	MainRepo         string            `yaml:"main_repo"`
	AnnotationRegexp string            `yaml:"annotation_regexp"`
	Order            int               `yaml:"order"`
	JoinDate         *time.Time        `yaml:"join_date"`
	FilesSkipPattern string            `yaml:"files_skip_pattern"`
	Env              map[string]string `yaml:"env"`

Project contain mapping from project name to its command line used to sync it

type PullRequest

type PullRequest struct {
	ID                  int        `json:"id"`
	Base                Branch     `json:"base"`
	Head                Branch     `json:"head"`
	User                Actor      `json:"user"`
	Number              int        `json:"number"`
	State               string     `json:"state"`
	Locked              *bool      `json:"locked"`
	Title               string     `json:"title"`
	Body                *string    `json:"body"`
	CreatedAt           time.Time  `json:"created_at"`
	UpdatedAt           time.Time  `json:"updated_at"`
	ClosedAt            *time.Time `json:"closed_at"`
	MergedAt            *time.Time `json:"merged_at"`
	MergeCommitSHA      *string    `json:"merge_commit_sha"`
	Assignee            *Actor     `json:"assignee"`
	Assignees           *[]Actor   `json:"assignees"`
	RequestedReviewers  *[]Actor   `json:"requested_reviewers"`
	Milestone           *Milestone `json:"milestone"`
	Merged              *bool      `json:"merged"`
	Mergeable           *bool      `json:"mergeable"`
	MergedBy            *Actor     `json:"merged_by"`
	MergeableState      *string    `json:"mergeable_state"`
	Rebaseable          *bool      `json:"rebaseable"`
	Comments            *int       `json:"comments"`
	ReviewComments      *int       `json:"review_comments"`
	MaintainerCanModify *bool      `json:"maintainer_can_modify"`
	Commits             *int       `json:"commits"`
	Additions           *int       `json:"additions"`
	Deletions           *int       `json:"deletions"`
	ChangedFiles        *int       `json:"changed_files"`

PullRequest - GHA Pull Request structure

type Release

type Release struct {
	ID              int        `json:"id"`
	TagName         string     `json:"tag_name"`
	TargetCommitish string     `json:"target_commitish"`
	Name            *string    `json:"name"`
	Draft           bool       `json:"draft"`
	Author          Actor      `json:"author"`
	Prerelease      bool       `json:"prerelease"`
	CreatedAt       time.Time  `json:"created_at"`
	PublishedAt     *time.Time `json:"published_at"`
	Body            *string    `json:"body"`
	Assets          []Asset    `json:"assets"`

Release - GHA Release structure

type Repo

type Repo struct {
	ID   int    `json:"id"`
	Name string `json:"name"`

Repo - GHA Repo structure

type TSPoint added in v0.5.0

type TSPoint struct {
	// contains filtered or unexported fields

TSPoint keeps single time series point

func NewTSPoint added in v0.5.0

func NewTSPoint(ctx *Ctx, name, period string, tags map[string]string, fields map[string]interface{}, t time.Time) TSPoint

NewTSPoint returns new point as specified by args

func (*TSPoint) Str added in v0.5.0

func (p *TSPoint) Str() string

Str - string pretty print

type TSPoints added in v0.5.0

type TSPoints []TSPoint

TSPoints keeps batch of TSPoint values to write

func (*TSPoints) Str added in v0.5.0

func (ps *TSPoints) Str() string

Str - string pretty print

type Tag added in v0.5.0

type Tag struct {
	Name       string            `yaml:"name"`
	SQLFile    string            `yaml:"sql"`
	SeriesName string            `yaml:"series_name"`
	NameTag    string            `yaml:"name_tag"`
	ValueTag   string            `yaml:"value_tag"`
	OtherTags  map[string]string `yaml:"other_tags"`

Tag contain each TSDB tag data

type Tags added in v0.5.0

type Tags struct {
	Tags []Tag `yaml:"tags"`

Tags contain list of TSDB tags

type Team

type Team struct {
	ID         int    `json:"id"`
	Name       string `json:"name"`
	Slug       string `json:"slug"`
	Permission string `json:"permission"`

Team - GHA Team structure (only used before 2015)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL