taskPipelines

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 3, 2024 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func WithHypothesisTemplate 

func WithHypothesisTemplate(hypothesisTemplate string) pipelines.PipelineOption[*ZeroShotClassificationPipeline]

WithHypothesisTemplate can be used to set the hypothesis template for classification.

func WithIgnoreLabels 

func WithIgnoreLabels(ignoreLabels []string) pipelines.PipelineOption[*TokenClassificationPipeline]

func WithLabels 

func WithLabels(labels []string) pipelines.PipelineOption[*ZeroShotClassificationPipeline]

WithLabels can be used to set the labels to classify the examples.

func WithMultiLabel 

func WithMultiLabel() pipelines.PipelineOption[*TextClassificationPipeline]

func WithMultilabel 

func WithMultilabel(multilabel bool) pipelines.PipelineOption[*ZeroShotClassificationPipeline]

WithMultilabel can be used to set whether the pipeline is multilabel.

func WithNormalization 

func WithNormalization() pipelines.PipelineOption[*FeatureExtractionPipeline]

WithNormalization applies normalization to the mean pooled output of the feature pipeline.

func WithOutputName 

func WithOutputName(outputName string) pipelines.PipelineOption[*FeatureExtractionPipeline]

WithOutputName if there are multiple outputs from the underlying model, which output should be returned. If not passed, the first output from the feature pipeline is returned.

func WithSigmoid 

func WithSigmoid() pipelines.PipelineOption[*TextClassificationPipeline]

func WithSimpleAggregation 

func WithSimpleAggregation() pipelines.PipelineOption[*TokenClassificationPipeline]

WithSimpleAggregation sets the aggregation strategy for the token labels to simple It reproduces simple aggregation from the huggingface implementation.

func WithSingleLabel 

func WithSingleLabel() pipelines.PipelineOption[*TextClassificationPipeline]

func WithSoftmax 

func WithSoftmax() pipelines.PipelineOption[*TextClassificationPipeline]

func WithoutAggregation 

func WithoutAggregation() pipelines.PipelineOption[*TokenClassificationPipeline]

WithoutAggregation returns the token labels.

Types

type ClassificationOutput 

type ClassificationOutput struct {
	Label string
	Score float32
}

type Entity 

type Entity struct {
	Entity    string
	Score     float32
	Scores    []float32
	Index     int
	Word      string
	TokenID   uint32
	Start     uint
	End       uint
	IsSubword bool
}

type FeatureExtractionOutput 

type FeatureExtractionOutput struct {
	Embeddings [][]float32
}

func (*FeatureExtractionOutput) GetOutput 

func (t *FeatureExtractionOutput) GetOutput() []any

type FeatureExtractionPipeline 

type FeatureExtractionPipeline struct {
	*pipelines.BasePipeline
	Normalization bool
	OutputName    string
	Output        pipelines.InputOutputInfo
}

FeatureExtractionPipeline A feature extraction pipeline is a go version of https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/feature_extraction.py

func NewFeatureExtractionPipeline 

func NewFeatureExtractionPipeline(config pipelines.PipelineConfig[*FeatureExtractionPipeline], s *options.Options, model *pipelines.Model) (*FeatureExtractionPipeline, error)

NewFeatureExtractionPipeline init a feature extraction pipeline.

func (*FeatureExtractionPipeline) Forward 

func (p *FeatureExtractionPipeline) Forward(batch *pipelines.PipelineBatch) error

Forward performs the forward inference of the feature extraction pipeline.

func (*FeatureExtractionPipeline) GetMetadata 

func (p *FeatureExtractionPipeline) GetMetadata() pipelines.PipelineMetadata

GetMetadata returns metadata information about the pipeline, in particular: OutputInfo: names and dimensions of the output layer.

func (*FeatureExtractionPipeline) GetStats 

func (p *FeatureExtractionPipeline) GetStats() []string

GetStats returns the runtime statistics for the pipeline.

func (*FeatureExtractionPipeline) Postprocess 

func (p *FeatureExtractionPipeline) Postprocess(batch *pipelines.PipelineBatch) (*FeatureExtractionOutput, error)

Postprocess parses the first output from the network similar to the transformers' implementation.

func (*FeatureExtractionPipeline) Preprocess 

func (p *FeatureExtractionPipeline) Preprocess(batch *pipelines.PipelineBatch, inputs []string) error

Preprocess tokenizes the input strings.

func (*FeatureExtractionPipeline) Run 

func (p *FeatureExtractionPipeline) Run(inputs []string) (pipelines.PipelineBatchOutput, error)

Run the pipeline on a batch of strings.

func (*FeatureExtractionPipeline) RunPipeline 

func (p *FeatureExtractionPipeline) RunPipeline(inputs []string) (*FeatureExtractionOutput, error)

RunPipeline is like Run, but returns the concrete feature extraction output type rather than the interface.

func (*FeatureExtractionPipeline) Validate 

func (p *FeatureExtractionPipeline) Validate() error

Validate checks that the pipeline is valid.

type TextClassificationOutput 

type TextClassificationOutput struct {
	ClassificationOutputs [][]ClassificationOutput
}

func (*TextClassificationOutput) GetOutput 

func (t *TextClassificationOutput) GetOutput() []any

type TextClassificationPipeline 

type TextClassificationPipeline struct {
	*pipelines.BasePipeline
	IDLabelMap              map[int]string
	AggregationFunctionName string
	ProblemType             string
}

func NewTextClassificationPipeline 

func NewTextClassificationPipeline(config pipelines.PipelineConfig[*TextClassificationPipeline], s *options.Options, model *pipelines.Model) (*TextClassificationPipeline, error)

NewTextClassificationPipeline initializes a new text classification pipeline.

func (*TextClassificationPipeline) Forward 

func (p *TextClassificationPipeline) Forward(batch *pipelines.PipelineBatch) error

func (*TextClassificationPipeline) GetMetadata 

func (p *TextClassificationPipeline) GetMetadata() pipelines.PipelineMetadata

GetMetadata returns metadata information about the pipeline, in particular: OutputInfo: names and dimensions of the output layer used for text classification.

func (*TextClassificationPipeline) GetStats 

func (p *TextClassificationPipeline) GetStats() []string

GetStats returns the runtime statistics for the pipeline.

func (*TextClassificationPipeline) Postprocess 

func (p *TextClassificationPipeline) Postprocess(batch *pipelines.PipelineBatch) (*TextClassificationOutput, error)

func (*TextClassificationPipeline) Preprocess 

func (p *TextClassificationPipeline) Preprocess(batch *pipelines.PipelineBatch, inputs []string) error

Preprocess tokenizes the input strings.

func (*TextClassificationPipeline) Run 

func (p *TextClassificationPipeline) Run(inputs []string) (pipelines.PipelineBatchOutput, error)

Run the pipeline on a string batch.

func (*TextClassificationPipeline) RunPipeline 

func (p *TextClassificationPipeline) RunPipeline(inputs []string) (*TextClassificationOutput, error)

func (*TextClassificationPipeline) Validate 

func (p *TextClassificationPipeline) Validate() error

Validate checks that the pipeline is valid.

type TextClassificationPipelineConfig 

type TextClassificationPipelineConfig struct {
	IDLabelMap map[int]string `json:"id2label"`
}

type TokenClassificationOutput 

type TokenClassificationOutput struct {
	Entities [][]Entity
}

func (*TokenClassificationOutput) GetOutput 

func (t *TokenClassificationOutput) GetOutput() []any

type TokenClassificationPipeline 

type TokenClassificationPipeline struct {
	*pipelines.BasePipeline
	IDLabelMap          map[int]string
	AggregationStrategy string
	IgnoreLabels        []string
}

TokenClassificationPipeline is a go version of huggingface tokenClassificationPipeline. https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/token_classification.py

func NewTokenClassificationPipeline 

func NewTokenClassificationPipeline(config pipelines.PipelineConfig[*TokenClassificationPipeline], s *options.Options, model *pipelines.Model) (*TokenClassificationPipeline, error)

NewTokenClassificationPipeline Initializes a feature extraction pipeline.

func (*TokenClassificationPipeline) Aggregate 

func (p *TokenClassificationPipeline) Aggregate(input pipelines.TokenizedInput, preEntities []Entity) ([]Entity, error)

func (*TokenClassificationPipeline) Forward 

func (p *TokenClassificationPipeline) Forward(batch *pipelines.PipelineBatch) error

Forward performs the forward inference of the pipeline.

func (*TokenClassificationPipeline) GatherPreEntities 

func (p *TokenClassificationPipeline) GatherPreEntities(input pipelines.TokenizedInput, output [][]float32) []Entity

GatherPreEntities from batch of logits to list of pre-aggregated outputs

func (*TokenClassificationPipeline) GetMetadata 

func (p *TokenClassificationPipeline) GetMetadata() pipelines.PipelineMetadata

GetMetadata returns metadata information about the pipeline, in particular: OutputInfo: names and dimensions of the output layer used for token classification.

func (*TokenClassificationPipeline) GetStats 

func (p *TokenClassificationPipeline) GetStats() []string

GetStats returns the runtime statistics for the pipeline.

func (*TokenClassificationPipeline) GroupEntities 

func (p *TokenClassificationPipeline) GroupEntities(entities []Entity) ([]Entity, error)

GroupEntities group together adjacent tokens with the same entity predicted.

func (*TokenClassificationPipeline) Postprocess 

func (p *TokenClassificationPipeline) Postprocess(batch *pipelines.PipelineBatch) (*TokenClassificationOutput, error)

Postprocess function for a token classification pipeline.

func (*TokenClassificationPipeline) Preprocess 

func (p *TokenClassificationPipeline) Preprocess(batch *pipelines.PipelineBatch, inputs []string) error

Preprocess tokenizes the input strings.

func (*TokenClassificationPipeline) Run 

func (p *TokenClassificationPipeline) Run(inputs []string) (pipelines.PipelineBatchOutput, error)

Run the pipeline on a string batch.

func (*TokenClassificationPipeline) RunPipeline 

func (p *TokenClassificationPipeline) RunPipeline(inputs []string) (*TokenClassificationOutput, error)

RunPipeline is like Run but returns the concrete type rather than the interface.

func (*TokenClassificationPipeline) Validate 

func (p *TokenClassificationPipeline) Validate() error

Validate checks that the pipeline is valid.

type TokenClassificationPipelineConfig 

type TokenClassificationPipelineConfig struct {
	IDLabelMap map[int]string `json:"id2label"`
}

type ZeroShotClassificationOutput 

type ZeroShotClassificationOutput struct {
	Sequence     string
	SortedValues []struct {
		Key   string
		Value float64
	}
}

type ZeroShotClassificationPipeline 

type ZeroShotClassificationPipeline struct {
	*pipelines.BasePipeline
	IDLabelMap         map[int]string
	Sequences          []string
	Labels             []string
	HypothesisTemplate string
	Multilabel         bool
	// contains filtered or unexported fields
}

func NewZeroShotClassificationPipeline 

func NewZeroShotClassificationPipeline(config pipelines.PipelineConfig[*ZeroShotClassificationPipeline], s *options.Options, model *pipelines.Model) (*ZeroShotClassificationPipeline, error)

NewZeroShotClassificationPipeline create new Zero Shot Classification Pipeline.

func (*ZeroShotClassificationPipeline) Forward 

func (p *ZeroShotClassificationPipeline) Forward(batch *pipelines.PipelineBatch) error

func (*ZeroShotClassificationPipeline) GetMetadata 

func (p *ZeroShotClassificationPipeline) GetMetadata() pipelines.PipelineMetadata

func (*ZeroShotClassificationPipeline) GetStats 

func (p *ZeroShotClassificationPipeline) GetStats() []string

func (*ZeroShotClassificationPipeline) Postprocess 

func (p *ZeroShotClassificationPipeline) Postprocess(outputTensors [][][]float32, labels []string, sequences []string) (*ZeroShotOutput, error)

func (*ZeroShotClassificationPipeline) Preprocess 

func (p *ZeroShotClassificationPipeline) Preprocess(batch *pipelines.PipelineBatch, inputs []string) error

func (*ZeroShotClassificationPipeline) Run 

func (p *ZeroShotClassificationPipeline) Run(inputs []string) (pipelines.PipelineBatchOutput, error)

func (*ZeroShotClassificationPipeline) RunPipeline 

func (p *ZeroShotClassificationPipeline) RunPipeline(inputs []string) (*ZeroShotOutput, error)

func (*ZeroShotClassificationPipeline) Validate 

func (p *ZeroShotClassificationPipeline) Validate() error

type ZeroShotClassificationPipelineConfig 

type ZeroShotClassificationPipelineConfig struct {
	IDLabelMap map[int]string `json:"id2label"`
}

type ZeroShotOutput 

type ZeroShotOutput struct {
	ClassificationOutputs []ZeroShotClassificationOutput
}

func (*ZeroShotOutput) GetOutput 

func (t *ZeroShotOutput) GetOutput() []any

GetOutput converts raw output to readable output.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.