output

package
v0.0.0-...-6c742d3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 30, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package output writes FOCUS 1.3 rows to various output formats.

Package output provides writers for FOCUS cost data export targets.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ContentTypeForFormat 

func ContentTypeForFormat(format string) string

ContentTypeForFormat maps a file format name to its MIME type. 

"parquet" -> "application/vnd.apache.parquet"
"csv"     -> "text/csv"

Unknown formats fall back to "application/octet-stream".

func FindMissingSlots 

func FindMissingSlots(expected []time.Time, existing map[time.Time]bool) []time.Time

FindMissingSlots returns the subset of expected slot times that are not present in the existing set. The result preserves the order of expected.

func GenerateKey 

func GenerateKey(prefix, orgID string, t time.Time, format string) string

GenerateKey builds the S3 object key for a given organisation, timestamp, and file format. The layout follows Hive-style partitioning so that tools such as Athena and Spark can prune efficiently: 

{prefix}version=focus-1.3/year=2025/month=03/day=10/castai-focus-{orgID}-2025-03-10-14.parquet

func GenerateSlots 

func GenerateSlots(start, end time.Time, granularity time.Duration) []time.Time

GenerateSlots returns every granularity-aligned slot start time in the half-open interval [start, end). Times are truncated to granularity boundaries before iteration (hour-aligned for hourly, day-aligned for daily in UTC).

Example: GenerateSlots(10:17, 13:45, 1h) → [10:00, 11:00, 12:00, 13:00]

func ParseKeyTimestamp 

func ParseKeyTimestamp(key string) (time.Time, error)

ParseKeyTimestamp extracts the slot start time from an S3 object key produced by GenerateKey. The key has the form: 

{prefix}version=focus-1.3/year=YYYY/month=MM/day=DD/castai-focus-{orgId}-YYYY-MM-DD-HH.{format}

It parses the YYYY-MM-DD-HH portion from the filename component.

func WriteCSV 

func WriteCSV(w io.Writer, rows []focus.FocusRow) error

WriteCSV writes FOCUS rows as CSV to the provided writer.

func WriteCSVFile 

func WriteCSVFile(path string, rows []focus.FocusRow) error

WriteCSVFile writes FOCUS rows to a CSV file at the given path.

func WriteParquetFile 

func WriteParquetFile(path string, rows []focus.FocusRow) error

WriteParquetFile writes FOCUS rows to a local Parquet file at the given path.

func WriteParquetToWriter 

func WriteParquetToWriter(w io.Writer, rows []focus.FocusRow) error

WriteParquetToWriter writes FOCUS rows as Parquet to the provided io.Writer. This is useful for streaming to S3 or other remote destinations.

Types

type ParquetFocusRow 

type ParquetFocusRow struct {
	BilledCost          float64 `parquet:"name=BilledCost, type=DOUBLE, repetitiontype=REQUIRED"`
	BillingAccountId    string  `parquet:"name=BillingAccountId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	BillingAccountName  string  `parquet:"name=BillingAccountName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	BillingCurrency     string  `parquet:"name=BillingCurrency, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	BillingPeriodStart  int64   `parquet:"name=BillingPeriodStart, type=INT64, convertedtype=TIMESTAMP_MILLIS, repetitiontype=REQUIRED"`
	BillingPeriodEnd    int64   `parquet:"name=BillingPeriodEnd, type=INT64, convertedtype=TIMESTAMP_MILLIS, repetitiontype=REQUIRED"`
	ChargeCategory      string  `parquet:"name=ChargeCategory, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	ChargeDescription   string  `parquet:"name=ChargeDescription, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	ChargePeriodStart   int64   `parquet:"name=ChargePeriodStart, type=INT64, convertedtype=TIMESTAMP_MILLIS, repetitiontype=REQUIRED"`
	ChargePeriodEnd     int64   `parquet:"name=ChargePeriodEnd, type=INT64, convertedtype=TIMESTAMP_MILLIS, repetitiontype=REQUIRED"`
	ContractedCost      float64 `parquet:"name=ContractedCost, type=DOUBLE, repetitiontype=REQUIRED"`
	EffectiveCost       float64 `parquet:"name=EffectiveCost, type=DOUBLE, repetitiontype=REQUIRED"`
	HostProviderName    string  `parquet:"name=HostProviderName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	InvoiceIssuerName   string  `parquet:"name=InvoiceIssuerName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	ListCost            float64 `parquet:"name=ListCost, type=DOUBLE, repetitiontype=REQUIRED"`
	PricingQuantity     float64 `parquet:"name=PricingQuantity, type=DOUBLE, repetitiontype=REQUIRED"`
	PricingUnit         string  `parquet:"name=PricingUnit, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	ServiceCategory     string  `parquet:"name=ServiceCategory, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	ServiceName         string  `parquet:"name=ServiceName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	ServiceProviderName string  `parquet:"name=ServiceProviderName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`
	PublisherName       string  `parquet:"name=PublisherName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"`

	AvailabilityZone    *string  `parquet:"name=AvailabilityZone, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ChargeClass         *string  `parquet:"name=ChargeClass, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ChargeFrequency     *string  `parquet:"name=ChargeFrequency, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ConsumedQuantity    *float64 `parquet:"name=ConsumedQuantity, type=DOUBLE, repetitiontype=OPTIONAL"`
	ConsumedUnit        *string  `parquet:"name=ConsumedUnit, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	InvoiceId           *string  `parquet:"name=InvoiceId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	PricingCategory     *string  `parquet:"name=PricingCategory, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	RegionId            *string  `parquet:"name=RegionId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	RegionName          *string  `parquet:"name=RegionName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ResourceId          *string  `parquet:"name=ResourceId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ResourceName        *string  `parquet:"name=ResourceName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ResourceType        *string  `parquet:"name=ResourceType, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ServiceSubcategory  *string  `parquet:"name=ServiceSubcategory, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	SkuId               *string  `parquet:"name=SkuId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	SkuPriceId          *string  `parquet:"name=SkuPriceId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	SubAccountId        *string  `parquet:"name=SubAccountId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	SubAccountName      *string  `parquet:"name=SubAccountName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	ContractedUnitPrice *float64 `parquet:"name=ContractedUnitPrice, type=DOUBLE, repetitiontype=OPTIONAL"`
	ListUnitPrice       *float64 `parquet:"name=ListUnitPrice, type=DOUBLE, repetitiontype=OPTIONAL"`
	Tags                *string  `parquet:"name=Tags, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`

	XCastAIClusterId    *string  `parquet:"name=x_CastAIClusterId, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	XCastAIWorkloadType *string  `parquet:"name=x_CastAIWorkloadType, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	XCastAINamespace    *string  `parquet:"name=x_CastAINamespace, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	XCastAIPricingTier  *string  `parquet:"name=x_CastAIPricingTier, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"`
	XCastAICpuCost      *float64 `parquet:"name=x_CastAICpuCost, type=DOUBLE, repetitiontype=OPTIONAL"`
	XCastAIRamCost      *float64 `parquet:"name=x_CastAIRamCost, type=DOUBLE, repetitiontype=OPTIONAL"`
	XCastAIGpuCost      *float64 `parquet:"name=x_CastAIGpuCost, type=DOUBLE, repetitiontype=OPTIONAL"`
	XCastAIStorageCost  *float64 `parquet:"name=x_CastAIStorageCost, type=DOUBLE, repetitiontype=OPTIONAL"`
}

ParquetFocusRow is a flat, Parquet-friendly representation of a FOCUS 1.3 row. Required fields are value types; optional fields are pointers so parquet-go can encode them with OPTIONAL repetition type.

type S3Uploader 

type S3Uploader struct {
	// contains filtered or unexported fields
}

S3Uploader uploads FOCUS export files (Parquet or CSV) to an S3 bucket.

func NewS3Uploader 

func NewS3Uploader(ctx context.Context, cfg S3UploaderConfig) (*S3Uploader, error)

NewS3Uploader creates an S3Uploader from the supplied configuration.

When AccessKeyID and SecretAccessKey are provided, static credentials are used. Otherwise the default credential chain is consulted, allowing transparent use of IRSA, environment variables, or EC2 instance profiles.

func (*S3Uploader) ListKeys 

func (u *S3Uploader) ListKeys(ctx context.Context, prefix string) ([]string, error)

ListKeys returns all S3 object keys under the given prefix. It handles pagination transparently via the continuation token. This is used to discover which time slots already have exported files in S3.

func (*S3Uploader) Upload 

func (u *S3Uploader) Upload(ctx context.Context, key string, data io.Reader, contentType string) error

Upload writes the contents of data to the specified S3 key. The caller must supply a valid MIME type in contentType (see ContentTypeForFormat).

func (*S3Uploader) UploadFile 

func (u *S3Uploader) UploadFile(ctx context.Context, key string, filePath string) error

UploadFile uploads a local file to S3 under the given key. The content type is inferred from the file extension.

type S3UploaderConfig 

type S3UploaderConfig struct {
	Bucket          string
	Region          string
	Prefix          string
	AccessKeyID     string
	SecretAccessKey string
}

S3UploaderConfig holds the settings needed to create an S3Uploader.

If AccessKeyID and SecretAccessKey are both set, static credentials are used. Otherwise the default AWS credential chain is used, which supports IRSA, environment variables, EC2 instance profiles, and more.

type TimeRange 

type TimeRange struct {
	Start time.Time
	End   time.Time
}

TimeRange represents a contiguous half-open time interval [Start, End).

func CoalesceSlots 

func CoalesceSlots(missing []time.Time, granularity time.Duration) []TimeRange

CoalesceSlots merges a sorted slice of missing slot start times into the smallest set of contiguous TimeRange values. Two slots are considered contiguous when one immediately follows the other (i.e. the gap between them equals granularity exactly).

The returned ranges are half-open: End = last slot start + granularity.

Example: slots [10:00, 11:00, 12:00, 15:00, 16:00] with 1h granularity → [{10:00, 13:00}, {15:00, 17:00}]

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.