reconciler

Documentation

Overview

@Title This file is used to cache the objects that need to be reused in the reconciler @Description The cached data is placed in the global sync.Map, you need to call clearPipelineContextCaches to clean up the corresponding data after you use it at will

Index

• Constants
• type NewRun
  • func (run NewRun) ResetTask(task *spec.PipelineTask, options PolicyHandlerOptions) (*spec.PipelineTask, error)
• type PipelineSvcFunc
• type PolicyHandlerOptions
• type PolicyType
• type Reconciler
  • func New(js jsonstore.JsonStore, etcd *etcd.Store, bdl *bundle.Bundle, ...) (*Reconciler, error)
  • func (r *Reconciler) Add(pipelineID uint64)
  • func (r *Reconciler) CompensateGCNamespaces(ctx context.Context)
  • func (r *Reconciler) DoDBGC(pipelineID uint64, gcOption apistructs.PipelineGCDBOption) error
  • func (r *Reconciler) EnsureDatabaseGC()
  • func (r *Reconciler) Listen(ctx context.Context)
  • func (r *Reconciler) ListenDatabaseGC()
  • func (r *Reconciler) ListenGC(ctx context.Context)
  • func (r *Reconciler) LoadQueueManger(ctx context.Context) error
  • func (r *Reconciler) PipelineDatabaseGC(ctx context.Context)
  • func (r *Reconciler) WaitDBGC(pipelineID uint64, ttl uint64, needArchive bool)
  • func (r *Reconciler) YmlTaskMergeDBTasks(pipeline *spec.Pipeline) ([]*spec.PipelineTask, error)
• type TryLastSuccessResult
  • func (t TryLastSuccessResult) ResetTask(task *spec.PipelineTask, opt PolicyHandlerOptions) (*spec.PipelineTask, error)

Constants

const (
	EtcdNeedCompensatePrefix = "/devops/pipeline/compensate/"
)

const PipelineContextCacheKey = "reconciler_caches"

const PipelinePassedDataWhenCreateContextCachesPrefixKey = "reconciler_caches_passed_data_when_create"

const PipelineRerunSuccessTaskContextCachesPrefixKey = "reconciler_caches_rerun_success_tasks"

const PipelineStagesContextCachesPrefixKey = "reconciler_caches_stages"

const PipelineYmlContextCachesPrefixKey = "reconciler_caches_yml"

Types

type NewRun

type NewRun struct{}

func (NewRun) ResetTask

func (run NewRun) ResetTask(task *spec.PipelineTask, options PolicyHandlerOptions) (*spec.PipelineTask, error)

type PipelineSvcFunc

type PipelineSvcFunc struct {
	CronNotExecuteCompensate                func(id uint64) error
	MergePipelineYmlTasks                   func(pipelineYml *pipelineyml.PipelineYml, dbTasks []spec.PipelineTask, p *spec.Pipeline, dbStages []spec.PipelineStage, passedDataWhenCreate *action_info.PassedDataWhenCreate) (mergeTasks []spec.PipelineTask, err error)
	HandleQueryPipelineYamlBySnippetConfigs func(sourceSnippetConfigs []apistructs.SnippetConfig) (map[string]string, error)
	MakeSnippetPipeline4Create              func(p *spec.Pipeline, snippetTask *spec.PipelineTask, yamlContent string) (*spec.Pipeline, error)
	CreatePipelineGraph                     func(p *spec.Pipeline) (err error)
}

In order to solve the problem of circular dependency if Reconciler introduces pipelinesvc, the svc method is mounted in this structure. todo resolve cycle import here through better module architecture

type PolicyHandlerOptions

type PolicyHandlerOptions struct {
	// contains filtered or unexported fields
}

type PolicyType

type PolicyType interface {
	ResetTask(*spec.PipelineTask, PolicyHandlerOptions) (*spec.PipelineTask, error)
}

type Reconciler

type Reconciler struct {
	QueueManager  types.QueueManager
	TaskThrottler throttler.Throttler // TODO remove the throttler.Throttler, after release/1.3 iteration throttler is not necessary
	// contains filtered or unexported fields
}

func New

func New(js jsonstore.JsonStore, etcd *etcd.Store, bdl *bundle.Bundle, dbClient *dbclient.Client,
	actionAgentSvc *actionagentsvc.ActionAgentSvc,
	extMarketSvc *extmarketsvc.ExtMarketSvc,
	pipelineSvcFunc *PipelineSvcFunc,
) (*Reconciler, error)

New generate a new reconciler.

func (*Reconciler) Add

func (r *Reconciler) Add(pipelineID uint64)

Add add pipelineID to reconciler, until add success

func (*Reconciler) CompensateGCNamespaces

func (r *Reconciler) CompensateGCNamespaces(ctx context.Context)

sometimes the pipeline is in downtime or restart time then the etcd lease of gc may expire at this time and then there is no instance get lease, which results in some namespaces pod not being gc

func (*Reconciler) DoDBGC

func (r *Reconciler) DoDBGC(pipelineID uint64, gcOption apistructs.PipelineGCDBOption) error

func (*Reconciler) EnsureDatabaseGC

func (r *Reconciler) EnsureDatabaseGC()

EnsureDatabaseGC etcd lease ttl reset 存在问题，因此要定期巡检，主动 delete 那些已经到了 gcAt 时间仍然存在的 etcd key 来触发 dbgc github issue: https://github.com/etcd-io/etcd/issues/9395

func (*Reconciler) Listen

func (r *Reconciler) Listen(ctx context.Context)

Listen watch incoming pipelines which need to be scheduled from etcd.

func (*Reconciler) ListenDatabaseGC

func (r *Reconciler) ListenDatabaseGC()

ListenDatabaseGC 监听需要 GC 的 pipeline database record.

func (*Reconciler) ListenGC

func (r *Reconciler) ListenGC(ctx context.Context)

func (*Reconciler) LoadQueueManger

func (r *Reconciler) LoadQueueManger(ctx context.Context) error

loadQueueManger

func (*Reconciler) PipelineDatabaseGC

func (r *Reconciler) PipelineDatabaseGC(ctx context.Context)

remove ListenDatabaseGC and EnsureDatabaseGC these two methods， these two methods will create a lot of etcd ttl, will cause high load on etcd use fixed gc time, traverse the data in the database every day

func (*Reconciler) WaitDBGC

func (r *Reconciler) WaitDBGC(pipelineID uint64, ttl uint64, needArchive bool)

func (*Reconciler) YmlTaskMergeDBTasks

func (r *Reconciler) YmlTaskMergeDBTasks(pipeline *spec.Pipeline) ([]*spec.PipelineTask, error)

parse out tasks according to the yml structure, and then query the created tasks from the database, and replace the tasks that already exist in the database with yml tasks

type TryLastSuccessResult

type TryLastSuccessResult struct{}

func (TryLastSuccessResult) ResetTask

func (t TryLastSuccessResult) ResetTask(task *spec.PipelineTask, opt PolicyHandlerOptions) (*spec.PipelineTask, error)