objects

package
Version: v0.0.0-...-16621ce Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 14, 2021 License: Apache-2.0 Imports: 23 Imported by: 1

README

Objects

PANTAHUB Objects API for CDN ready storage of blob objects (aka files).

Start Service

Development

If you do not set PANTAHUB_S3PATH ennvironment to production we will use local fake s3. If no path is set it will use the ./local-s3 directory as file storage on local disk

Production

In order to enable production S3 usage you have to set the PANTAHUB_S3PATH environment like:

export PANTAHUB_S3PATH=production

Before starting, set your AWS credentials in your environment:

AWS_ACCESS_KEY_ID=XXXX
AWS_SECRET_ACCESS_KEY=YYYYYYYY

export AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID

Now start your server:

./pantahub-base

Login

TOKEN=`http localhost:12365/auth/login username=user1 password=user1 | json token`

... will store access token in TOKEN for requests below

Upload File

Register Object
# adjust the below to be correct:
upload_file=myfile.jpg
upload_size=12365
upload_shasum256=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

http POST localhost:12365/objects/  Authorization:"Bearer $TOKEN" \
	objectname=$upload_file \
	size=$upload_size \
	sha256sum=$upload_shasum256
	
HTTP/1.1 200 OK
Content-Length: 152
Content-Type: application/json; charset=utf-8
Date: Fri, 19 Aug 2016 12:24:58 GMT
X-Powered-By: go-json-rest

{
    "id": "57b6fa9ac094f67942000002", 
    "mime-type": "", 
    "objectname": "myfile.jpg", 
    "owner": "prn:pantahub.com:auth:/user1", 
    "sha256sum": "xxxxxxxxxxx", 
    "size": "12356"
}

Get Put Url

Use the id from above to get info about where to upload:

http GET localhost:12365/objects/57b6fa9ac094f67942000002  Authorization:"Bearer $TOKEN"

HTTP/1.1 200 OK
Content-Length: 941
Content-Type: application/json; charset=utf-8
Date: Fri, 19 Aug 2016 12:26:41 GMT
X-Powered-By: go-json-rest

{
    "expire-time": "900", 
    "id": "57b6fa9ac094f67942000002", 
    "mime-type": "", 
    "now": "1471609601", 
    "objectname": "myfile.jpg", 
    "owner": "prn:pantahub.com:auth:/user1", 
    "sha256sum": "xxxxxxxxxxx", 
    "signed-geturl": "https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=19801ba6781f9b10d7d108cc429e55942497b9bc5b46aafba7325709a82c0029", 
    "signed-puturl": "https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=64ade69ed129f2aedcee9b84cd2e318b4863e2b6518301fbae9e53703c794e73", 
    "size": "12356"
}

SIGNED_GETURL="https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=19801ba6781f9b10d7d108cc429e55942497b9bc5b46aafba7325709a82c0029"
SIGNED_PUTURL="https://systemcloud-001.s3.amazonaws.com/57b6fa9ac094f67942000002?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCANUJOIDFTXDLJA%2F20160819%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160819T122641Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=64ade69ed129f2aedcee9b84cd2e318b4863e2b6518301fbae9e53703c794e73"


Put File to S3
cat $upload_file | http PUT $SIGNED_PUTURL
Get File from S3
http GET $SIGNED_GETURL

Documentation

Index

Constants

View Source
const (
	HttpHeaderPantahubObjectType = "Pantahub-Object-Type"
	ObjectTypeLink               = "link"
	ObjectTypeObject             = "object"
)
View Source
const (
	// ObjectTokenValidSec expiration time for object token
	ObjectTokenValidSec = 86400
)

Variables

View Source
var ErrNoBackingFile = errors.New("No backing file nor link")

ErrNoBackingFile error signals that an object is not fully resolvable as it has no backing file yet.

View Source
var ErrNoLinkTargetAvail = errors.New("No link target available")
View Source
var ErrObjectS3PathAlreadyExists error = errors.New("local s3 file path for object is already exists")

ErrObjectS3PathAlreadyExists erro variable for "local s3 file path for object is already exists"

Functions

func GetDiskQuota

func GetDiskQuota(prn string) (float64, error)

GetDiskQuota public function to get the default disk quota

func HasBackingFile

func HasBackingFile(object *Object) (bool, error)

func MakeStorageID

func MakeStorageID(owner string, sha []byte) string

MakeStorageID crerate a new storage ID

func PantahubS3DevURL

func PantahubS3DevURL() string

PantahubS3DevURL s3 dev url

func SyncObjectSizes

func SyncObjectSizes(obj *Object)

SyncObjectSizes syncronize objects sizes

Types

type App

type App struct {
	API *rest.Api
	// contains filtered or unexported fields
}

App objects rest application

func Build

func Build(mongoClient *mongo.Client) *App

Build factory a new Object App with mongoClient

func New

func New(jwtMiddleware *jwt.JWTMiddleware, subService subscriptions.SubscriptionService,
	mongoClient *mongo.Client) *App

New create a new object rest application

func (*App) FindObjectByShaByOwner

func (a *App) FindObjectByShaByOwner(
	Sha, Owner string,
	obj *Object,
) error

FindObjectByShaByOwner is to find object by sha & by owner

func (*App) FindObjectByStorageID

func (a *App) FindObjectByStorageID(storageID string, object *Object) error

FindObjectByStorageID find object instance by _id

func (*App) FindPublicObjectOwner

func (a *App) FindPublicObjectOwner(sha string, notOwnedBy string) (
	ownerStr string,
	err error,
)

FindPublicObjectOwner is to check if the object is used in any of the public steps, if yes return the owner string

func (*App) GetDiskQuota

func (a *App) GetDiskQuota(prn string) (float64, error)

GetDiskQuota get disk quota for a object

func (*App) LinkifyObject

func (a *App) LinkifyObject(object *Object) (
	linked bool,
	err error)

LinkifyObject checks if there is any public object available to link and link if available

func (*App) ResolveObjectWithBacking

func (a *App) ResolveObjectWithBacking(owner string, sha string) (*Object, error)
func (a *App) ResolveObjectWithLinks(owner string, sha string, autoLink bool) (*Object, error)

func (*App) SaveObject

func (a *App) SaveObject(object *Object, localS3Check bool) (err error)

SaveObject saves an object

type DiskQuotaUsageResult

type DiskQuotaUsageResult struct {
	ID    string  `json:"id" bson:"_id"`
	Total float64 `json:"total"`
}

DiskQuotaUsageResult payload for disk quota usage

func CalcUsageAfterPost

func CalcUsageAfterPost(owner string, mongoClient *mongo.Client,
	objectID string, newSize int64) (*DiskQuotaUsageResult, error)

CalcUsageAfterPost calculate usage after post new object

func CalcUsageAfterPut

func CalcUsageAfterPut(owner string, mongoClient *mongo.Client,
	objectID string, newSize int64) (*DiskQuotaUsageResult, error)

CalcUsageAfterPut calculate disk usage after update object

type Object

type Object struct {
	ID         string `json:"id" bson:"id"`
	StorageID  string `json:"storage-id" bson:"_id"`
	Owner      string `json:"owner"`
	ObjectName string `json:"objectname"`
	Sha        string `json:"sha256sum"`
	Size       string `json:"size"`
	SizeInt    int64  `json:"sizeint"`
	MimeType   string `json:"mime-type"`

	LinkedObject string    `json:"-" bson:"linked_object"`
	TimeCreated  time.Time `json:"time-created" bson:"timecreated"`
	TimeModified time.Time `json:"time-modified" bson:"timemodified"`
	// contains filtered or unexported fields
}

Object define a object structure

func NewObject

func NewObject(shaStr, owner, objectName string) (
	newObject *Object,
	err error,
)

NewObject makes a new object

type ObjectAccessClaims

type ObjectAccessClaims struct {
	jwt.StandardClaims
	DispositionName string
	Size            int64
	Method          string

	// sha in hex encoding
	Sha string
}

ObjectAccessClaims object claims for access

type ObjectAccessToken

type ObjectAccessToken struct {
	// we use iss for identifying the trails endpoint
	// we use sub for identifying the requesting user wethat this claim was issued to
	// we use aud to identify the access URI in format: http://endpoint/storage-id
	// we use issued at for the time we issued this
	// we use use expires at for issuing time constrained grants
	*jwt.Token
}

ObjectAccessToken access token for objects

func NewFromValidToken

func NewFromValidToken(encodedToken string) (*ObjectAccessToken, error)

NewFromValidToken create a object token from another valid token

func NewObjectAccessForSec

func NewObjectAccessForSec(
	name string,
	method string,
	size int64,
	sha string,
	issuer string,
	subject string,
	audience string,
	validSec int64) *ObjectAccessToken

NewObjectAccessForSec create new access token valid for a second

func NewObjectAccessToken

func NewObjectAccessToken(
	name string,
	method string,
	size int64,
	sha string,
	issuer string,
	subject string,
	audience string,
	issuedAt int64,
	expiresAt int64) *ObjectAccessToken

NewObjectAccessToken Create new Object access token

func (*ObjectAccessToken) Sign

func (o *ObjectAccessToken) Sign() (string, error)

Sign sign a access token

type ObjectWithAccess

type ObjectWithAccess struct {
	Object       `bson:",inline"`
	SignedPutURL string `json:"signed-puturl"`
	SignedGetURL string `json:"signed-geturl"`
	Now          string `json:"now"`
	ExpireTime   string `json:"expire-time"`
}

ObjectWithAccess extends object to add access information

func GetObjectWithAccess

func GetObjectWithAccess(object Object, endPoint string) *ObjectWithAccess

GetObjectWithAccess returns an ObjectWithAccess instance

func MakeObjAccessible

func MakeObjAccessible(Issuer string, Subject string, obj Object, storageID string) ObjectWithAccess

MakeObjAccessible make a object accessible

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL