hf

Documentation

Overview

Package hf contains HuggingFace integration logic for AIS.

• Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.

Package hf contains HuggingFace integration logic for AIS.

• Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.

Package hf contains HuggingFace integration logic for AIS.

• Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.

Index

• Constants
• func BuildHuggingFaceURL(model, dataset, file, revision string) (string, error)
• func ExtractDatasetFromHFMarker(marker string) (string, error)
• func ExtractFileName(url string) string
• func ExtractModelFromHFMarker(marker string) (string, error)
• func ExtractParquetFiles(data map[string]any) cos.StrKVs
• func GetHFDatasetParquetFiles(dataset, token string) (cos.StrKVs, error)
• func GetHFModelFiles(model, token string) (cos.StrKVs, error)
• func HasHuggingFaceRepoFlags(hasModel, hasDataset bool) bool
• func IsHuggingFaceURL(rawURL string) bool
• type FileInfo
  • func GetFileSizes(files cos.StrKVs, authToken string) ([]FileInfo, error)
• type HFModelFile
• type HFModelResponse

Constants

const (
	HfFullRepoMarker = "HF_FULL_REPO_DOWNLOAD:"
)

HuggingFace URL patterns and constants

Functions

func BuildHuggingFaceURL

func BuildHuggingFaceURL(model, dataset, file, revision string) (string, error)

BuildHuggingFaceURL constructs a HuggingFace download URL from parameters

func ExtractDatasetFromHFMarker

func ExtractDatasetFromHFMarker(marker string) (string, error)

ExtractDatasetFromHFMarker parses dataset name from hfFullRepoMarker

func ExtractFileName

func ExtractFileName(url string) string

ExtractFileName extracts filename from HuggingFace URL

func ExtractModelFromHFMarker

func ExtractModelFromHFMarker(marker string) (string, error)

ExtractModelFromHFMarker parses model name from hfFullRepoMarker

func ExtractParquetFiles

func ExtractParquetFiles(data map[string]any) cos.StrKVs

ExtractParquetFiles parses the JSON structure from HF API

func GetHFDatasetParquetFiles

func GetHFDatasetParquetFiles(dataset, token string) (cos.StrKVs, error)

GetHFDatasetParquetFiles fetches parquet file mappings for a HuggingFace dataset

func GetHFModelFiles

func GetHFModelFiles(model, token string) (cos.StrKVs, error)

GetHFModelFiles fetches all files in a HuggingFace model repository

func HasHuggingFaceRepoFlags

func HasHuggingFaceRepoFlags(hasModel, hasDataset bool) bool

HasHuggingFaceRepoFlags checks if only ModelFlag OR DatasetFlag (not both) is set

func IsHuggingFaceURL

func IsHuggingFaceURL(rawURL string) bool

IsHuggingFaceURL checks if a URL is a HuggingFace URL

Types

type FileInfo

type FileInfo struct {
	URL  string
	Size *int64
}

FileInfo represents a file with its URL and size

func GetFileSizes

func GetFileSizes(files cos.StrKVs, authToken string) ([]FileInfo, error)

GetFileSizes makes concurrent HEAD requests to get file sizes for HuggingFace files

type HFModelFile

type HFModelFile struct {
	Filename string `json:"rfilename"`
}

HFModelFile represents a file in HuggingFace model repository

type HFModelResponse

type HFModelResponse struct {
	Siblings []HFModelFile `json:"siblings"`
}

HFModelResponse represents the HuggingFace model API response