objects

package
v0.0.0-...-c9f6891 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 12, 2024 License: Apache-2.0 Imports: 24 Imported by: 2

README

Objects

PANTAHUB Objects API for CDN ready storage of blob objects (aka files).

Start Service

Development

If you do not set PANTAHUB_S3PATH ennvironment to production we will use local fake s3. If no path is set it will use the ./local-s3 directory as file storage on local disk

Production

In order to enable production S3 usage you have to set the PANTAHUB_S3PATH environment like:

export PANTAHUB_S3PATH=production

Before starting, set your AWS credentials in your environment:

AWS_ACCESS_KEY_ID=XXXX
AWS_SECRET_ACCESS_KEY=YYYYYYYY

export AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID

Now start your server:

./pantahub-base

Login

TOKEN=`http localhost:12365/auth/login username=user1 password=user1 | json token`

... will store access token in TOKEN for requests below

Upload File

Register Object
# adjust the below to be correct:
upload_file=myfile.jpg
upload_size=12365
upload_shasum256=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

http POST localhost:12365/objects/  Authorization:"Bearer $TOKEN" \
	objectname=$upload_file \
	size=$upload_size \
	sha256sum=$upload_shasum256
	
HTTP/1.1 200 OK
Content-Length: 152
Content-Type: application/json; charset=utf-8
Date: Fri, 19 Aug 2016 12:24:58 GMT
X-Powered-By: go-json-rest

{
    "id": "57b6fa9ac094f67942000002", 
    "mime-type": "", 
    "objectname": "myfile.jpg", 
    "owner": "prn:pantahub.com:auth:/user1", 
    "sha256sum": "xxxxxxxxxxx", 
    "size": "12356"
}
Get Put Url

Use the id from above to get info about where to upload:

http GET localhost:12365/objects/57b6fa9ac094f67942000002  Authorization:"Bearer $TOKEN"

HTTP/1.1 200 OK
Content-Length: 941
Content-Type: application/json; charset=utf-8
Date: Fri, 19 Aug 2016 12:26:41 GMT
X-Powered-By: go-json-rest

{
    "expire-time": "900", 
    "id": "57b6fa9ac094f67942000002", 
    "mime-type": "", 
    "now": "1471609601", 
    "objectname": "myfile.jpg", 
    "owner": "prn:pantahub.com:auth:/user1", 
    "sha256sum": "xxxxxxxxxxx", 
    "signed-geturl": "https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=19801ba6781f9b10d7d108cc429e55942497b9bc5b46aafba7325709a82c0029", 
    "signed-puturl": "https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=64ade69ed129f2aedcee9b84cd2e318b4863e2b6518301fbae9e53703c794e73", 
    "size": "12356"
}

SIGNED_GETURL="https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=19801ba6781f9b10d7d108cc429e55942497b9bc5b46aafba7325709a82c0029"
SIGNED_PUTURL="https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=64ade69ed129f2aedcee9b84cd2e318b4863e2b6518301fbae9e53703c794e73"
Put File to S3
cat $upload_file | http PUT $SIGNED_PUTURL
Get File from S3
http GET $SIGNED_GETURL

Documentation

Index

Constants

View Source 
const (
	HttpHeaderPantahubObjectType = "Pantahub-Object-Type"
	ObjectTypeLink               = "link"
	ObjectTypeObject             = "object"
)
View Source 
const (
	// ObjectTokenValidSec expiration time for object token
	ObjectTokenValidSec = 86400
)

Variables

View Source 
var ErrNoBackingFile = errors.New("No backing file nor link")

ErrNoBackingFile error signals that an object is not fully resolvable as it has no backing file yet.

View Source 
var ErrNoLinkTargetAvail = errors.New("No link target available")
View Source 
var ErrObjectS3PathAlreadyExists error = errors.New("local s3 file path for object is already exists")

ErrObjectS3PathAlreadyExists erro variable for "local s3 file path for object is already exists"

Functions

func GetDiskQuota 

func GetDiskQuota(ctx context.Context, prn string) (float64, error)

GetDiskQuota public function to get the default disk quota

func HasBackingFile 

func HasBackingFile(object *Object) (bool, error)

func MakeStorageID 

func MakeStorageID(owner string, sha []byte) string

MakeStorageID crerate a new storage ID

func PantahubS3DevURL 

func PantahubS3DevURL() string

PantahubS3DevURL s3 dev url

func SyncObjectSizes 

func SyncObjectSizes(obj *Object)

SyncObjectSizes syncronize objects sizes

Types

type App 

type App struct {
	API *rest.Api
	// contains filtered or unexported fields
}

App objects rest application

func Build 

func Build(mongoClient *mongo.Client) *App

Build factory a new Object App with mongoClient

func New 

func New(jwtMiddleware *jwt.JWTMiddleware, subService subscriptions.SubscriptionService,
	mongoClient *mongo.Client) *App

New create a new object rest application

func (*App) FindObjectByShaByOwner 

func (a *App) FindObjectByShaByOwner(
	parentCtx context.Context,
	Sha, Owner string,
	obj *Object,
) error

FindObjectByShaByOwner is to find object by sha & by owner

func (*App) FindObjectByStorageID 

func (a *App) FindObjectByStorageID(parentCtx context.Context, storageID string, object *Object) error

FindObjectByStorageID find object instance by _id

func (*App) FindPublicObjectOwner 

func (a *App) FindPublicObjectOwner(parentCtx context.Context, sha string, notOwnedBy string) (
	ownerStr string,
	err error,
)

FindPublicObjectOwner is to check if the object is used in any of the public steps, if yes return the owner string

func (*App) GetDiskQuota 

func (a *App) GetDiskQuota(pctx context.Context, prn string) (float64, error)

GetDiskQuota get disk quota for a object

func (*App) LinkifyObject 

func (a *App) LinkifyObject(ctx context.Context, object *Object) (
	linked bool,
	err error)

LinkifyObject checks if there is any public object available to link and link if available

func (*App) ResolveObjectWithBacking 

func (a *App) ResolveObjectWithBacking(pctx context.Context, owner string, sha string) (*Object, error)
func (a *App) ResolveObjectWithLinks(ctx context.Context, owner string, sha string, autoLink bool) (*Object, error)

func (*App) SaveObject 

func (a *App) SaveObject(parentCtx context.Context, object *Object, localS3Check bool) (err error)

SaveObject saves an object

type DiskQuotaUsageResult 

type DiskQuotaUsageResult struct {
	ID    string  `json:"id" bson:"_id"`
	Total float64 `json:"total"`
}

DiskQuotaUsageResult payload for disk quota usage

func CalcUsageAfterPost 

func CalcUsageAfterPost(parentCtx context.Context, owner string, mongoClient *mongo.Client,
	objectID string, newSize int64) (*DiskQuotaUsageResult, error)

CalcUsageAfterPost calculate usage after post new object

func CalcUsageAfterPut 

func CalcUsageAfterPut(parentCtx context.Context, owner string, mongoClient *mongo.Client,
	objectID string, newSize int64) (*DiskQuotaUsageResult, error)

CalcUsageAfterPut calculate disk usage after update object

type Object 

type Object struct {
	ID         string `json:"id" bson:"id"`
	StorageID  string `json:"storage-id" bson:"_id"`
	Owner      string `json:"owner"`
	ObjectName string `json:"objectname"`
	Sha        string `json:"sha256sum"`
	Size       string `json:"size"`
	SizeInt    int64  `json:"sizeint"`
	MimeType   string `json:"mime-type"`

	LinkedObject string    `json:"-" bson:"linked_object"`
	TimeCreated  time.Time `json:"time-created" bson:"timecreated"`
	TimeModified time.Time `json:"time-modified" bson:"timemodified"`
	// contains filtered or unexported fields
}

Object define a object structure

func NewObject 

func NewObject(shaStr, owner, objectName string) (
	newObject *Object,
	err error,
)

NewObject makes a new object

type ObjectAccessClaims 

type ObjectAccessClaims struct {
	jwt.StandardClaims
	DispositionName string
	Size            int64
	Method          string

	// sha in hex encoding
	Sha string
}

ObjectAccessClaims object claims for access

type ObjectAccessToken 

type ObjectAccessToken struct {
	// we use iss for identifying the trails endpoint
	// we use sub for identifying the requesting user wethat this claim was issued to
	// we use aud to identify the access URI in format: http://endpoint/storage-id
	// we use issued at for the time we issued this
	// we use use expires at for issuing time constrained grants
	*jwt.Token
}

ObjectAccessToken access token for objects

func NewFromValidToken 

func NewFromValidToken(encodedToken string) (*ObjectAccessToken, error)

NewFromValidToken create a object token from another valid token

func NewObjectAccessForSec 

func NewObjectAccessForSec(
	name string,
	method string,
	size int64,
	sha string,
	issuer string,
	subject string,
	audience string,
	validSec int64) *ObjectAccessToken

NewObjectAccessForSec create new access token valid for a second

func NewObjectAccessToken 

func NewObjectAccessToken(
	name string,
	method string,
	size int64,
	sha string,
	issuer string,
	subject string,
	audience string,
	issuedAt int64,
	expiresAt int64) *ObjectAccessToken

NewObjectAccessToken Create new Object access token

func (*ObjectAccessToken) Sign 

func (o *ObjectAccessToken) Sign() (string, error)

Sign sign a access token

type ObjectWithAccess 

type ObjectWithAccess struct {
	Object       `bson:",inline"`
	SignedPutURL string `json:"signed-puturl"`
	SignedGetURL string `json:"signed-geturl"`
	Now          string `json:"now"`
	ExpireTime   string `json:"expire-time"`
}

ObjectWithAccess extends object to add access information

func GetObjectWithAccess 

func GetObjectWithAccess(object Object, endPoint string) *ObjectWithAccess

GetObjectWithAccess returns an ObjectWithAccess instance

func MakeObjAccessible 

func MakeObjAccessible(Issuer string, Subject string, obj Object, storageID string) ObjectWithAccess

MakeObjAccessible make a object accessible

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.