goblet

README

Goblet: Git caching proxy

Goblet is a Git proxy server that caches repositories for read access. Git clients can configure their repositories to use this as an HTTP proxy server, and this proxy server serves git-fetch requests if it can be served from the local cache.

In the Git protocol, the server creates a pack-file dynamically based on the objects that the clients have. Because of this, caching Git protocol response is hard as different client needs a different response. Goblet parses the content of the HTTP POST requests and tells if the request can be served from the local cache.

This is developed to reduce the automation traffic to googlesource.com. Goblet would be useful if you need to run a Git read-only mirroring server to offload the traffic.

This is not an official Google product (i.e. a 20% project).

Usage

Goblet is intended to be used as a library. You would need to write some glue code. This repository includes the glue code for googlesource.com. See goblet-server and google directories.

Limitations

Note that Goblet forwards the ls-refs traffic to the upstream server. If the upstream server is down, Goblet is effectively down. Technically, we can modify Goblet to serve even if the upstream is down, but the current implementation doesn't do such thing.

Documentation

Index

• Variables
• func HTTPHandler(config *ServerConfig) http.Handler
• func ListManagedRepositories(fn func(ManagedRepository))
• type ManagedRepository
  • func OpenManagedRepository(config *ServerConfig, u *url.URL) (ManagedRepository, error)
• type RunningOperation
• type ServerConfig

Variables

var (
	// CommandTypeKey indicates a command type ("ls-refs", "fetch",
	// "not-a-command").
	CommandTypeKey = tag.MustNewKey("github.com/google/goblet/command-type")

	// CommandCacheStateKey indicates whether the command response is cached
	// or not ("locally-served", "queried-upstream").
	CommandCacheStateKey = tag.MustNewKey("github.com/google/goblet/command-cache-state")

	// CommandCanonicalStatusKey indicates whether the command is succeeded
	// or not ("OK", "Unauthenticated").
	CommandCanonicalStatusKey = tag.MustNewKey("github.com/google/goblet/command-status")

	// InboundCommandProcessingTime is a processing time of the inbound
	// commands.
	InboundCommandProcessingTime = stats.Int64("github.com/google/goblet/inbound-command-processing-time", "processing time of inbound commands", stats.UnitMilliseconds)

	// OutboundCommandProcessingTime is a processing time of the outbound
	// commands.
	OutboundCommandProcessingTime = stats.Int64("github.com/google/goblet/outbound-command-processing-time", "processing time of outbound commands", stats.UnitMilliseconds)

	// UpstreamFetchWaitingTime is a duration that a fetch request waited
	// for the upstream.
	UpstreamFetchWaitingTime = stats.Int64("github.com/google/goblet/upstream-fetch-waiting-time", "waiting time of upstream fetch command", stats.UnitMilliseconds)

	// InboundCommandCount is a count of inbound commands.
	InboundCommandCount = stats.Int64("github.com/google/goblet/inbound-command-count", "number of inbound commands", stats.UnitDimensionless)

	// OutboundCommandCount is a count of outbound commands.
	OutboundCommandCount = stats.Int64("github.com/google/goblet/outbound-command-count", "number of outbound commands", stats.UnitDimensionless)
)

Functions

func HTTPHandler

func HTTPHandler(config *ServerConfig) http.Handler

func ListManagedRepositories

func ListManagedRepositories(fn func(ManagedRepository))

Types

type ManagedRepository

type ManagedRepository interface {
	UpstreamURL() *url.URL

	LastUpdateTime() time.Time

	RecoverFromBundle(string) error

	WriteBundle(io.Writer) error
}

func OpenManagedRepository

func OpenManagedRepository(config *ServerConfig, u *url.URL) (ManagedRepository, error)

type RunningOperation

type RunningOperation interface {
	Printf(format string, a ...interface{})

	Done(error)
}

type ServerConfig

type ServerConfig struct {
	LocalDiskCacheRoot string

	URLCanonializer func(*url.URL) (*url.URL, error)

	RequestAuthorizer func(*http.Request) error

	TokenSource oauth2.TokenSource

	ErrorReporter func(*http.Request, error)

	RequestLogger func(r *http.Request, status int, requestSize, responseSize int64, latency time.Duration)

	LongRunningOperationLogger func(string, *url.URL) RunningOperation
}