Documentation ¶
Overview ¶
Package swift provides an easy to use interface to Swift / Openstack Object Storage / Rackspace Cloud Files
Standard Usage ¶
Most of the work is done through the Container*() and Object*() methods.
All methods are safe to use concurrently in multiple go routines.
Object Versioning ¶
As defined by http://docs.openstack.org/api/openstack-object-storage/1.0/content/Object_Versioning-e1e3230.html#d6e983 one can create a container which allows for version control of files. The suggested method is to create a version container for holding all non-current files, and a current container for holding the latest version that the file points to. The container and objects inside it can be used in the standard manner, however, pushing a file multiple times will result in it being copied to the version container and the new file put in it's place. If the current file is deleted, the previous file in the version container will replace it. This means that if a file is updated 5 times, it must be deleted 5 times to be completely removed from the system.
Rackspace Sub Module ¶
This module specifically allows the enabling/disabling of Rackspace Cloud File CDN management on a container. This is specific to the Rackspace API and not Swift/Openstack, therefore it has been placed in a submodule. One can easily create a RsConnection and use it like the standard Connection to access and manipulate containers and objects.
Index ¶
- Constants
- Variables
- func AddExpectAndTransferEncoding(req *http.Request, hasContentLength bool)
- func FloatStringToTime(s string) (t time.Time, err error)
- func SetExpectContinueTimeout(tr *http.Transport, t time.Duration)
- func TimeToFloatString(t time.Time) string
- type Account
- type Authenticator
- type BulkDeleteResult
- type BulkUploadResult
- type Connection
- func (c *Connection) Account() (info Account, headers Headers, err error)
- func (c *Connection) AccountUpdate(h Headers) error
- func (c *Connection) ApplyEnvironment() (err error)
- func (c *Connection) Authenticate() (err error)
- func (c *Connection) Authenticated() bool
- func (c *Connection) BulkDelete(container string, objectNames []string) (result BulkDeleteResult, err error)
- func (c *Connection) BulkDeleteHeaders(container string, objectNames []string, h Headers) (result BulkDeleteResult, err error)
- func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string, h Headers) (result BulkUploadResult, err error)
- func (c *Connection) Call(targetUrl string, p RequestOpts) (resp *http.Response, headers Headers, err error)
- func (c *Connection) Container(container string) (info Container, headers Headers, err error)
- func (c *Connection) ContainerCreate(container string, h Headers) error
- func (c *Connection) ContainerDelete(container string) error
- func (c *Connection) ContainerNames(opts *ContainersOpts) ([]string, error)
- func (c *Connection) ContainerNamesAll(opts *ContainersOpts) ([]string, error)
- func (c *Connection) ContainerUpdate(container string, h Headers) error
- func (c *Connection) Containers(opts *ContainersOpts) ([]Container, error)
- func (c *Connection) ContainersAll(opts *ContainersOpts) ([]Container, error)
- func (c *Connection) DynamicLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)
- func (c *Connection) DynamicLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)
- func (c *Connection) DynamicLargeObjectDelete(container string, path string) error
- func (c *Connection) DynamicLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, ...) error
- func (c *Connection) LargeObjectDelete(container string, objectName string) error
- func (c *Connection) LargeObjectGetSegments(container string, path string) (string, []Object, error)
- func (c *Connection) Object(container string, objectName string) (info Object, headers Headers, err error)
- func (c *Connection) ObjectCopy(srcContainer string, srcObjectName string, dstContainer string, ...) (headers Headers, err error)
- func (c *Connection) ObjectCreate(container string, objectName string, checkHash bool, Hash string, ...) (file *ObjectCreateFile, err error)
- func (c *Connection) ObjectDelete(container string, objectName string) error
- func (c *Connection) ObjectGet(container string, objectName string, contents io.Writer, checkHash bool, ...) (headers Headers, err error)
- func (c *Connection) ObjectGetBytes(container string, objectName string) (contents []byte, err error)
- func (c *Connection) ObjectGetString(container string, objectName string) (contents string, err error)
- func (c *Connection) ObjectMove(srcContainer string, srcObjectName string, dstContainer string, ...) (err error)
- func (c *Connection) ObjectNames(container string, opts *ObjectsOpts) ([]string, error)
- func (c *Connection) ObjectNamesAll(container string, opts *ObjectsOpts) ([]string, error)
- func (c *Connection) ObjectOpen(container string, objectName string, checkHash bool, h Headers) (file *ObjectOpenFile, headers Headers, err error)
- func (c *Connection) ObjectPut(container string, objectName string, contents io.Reader, checkHash bool, ...) (headers Headers, err error)
- func (c *Connection) ObjectPutBytes(container string, objectName string, contents []byte, contentType string) (err error)
- func (c *Connection) ObjectPutString(container string, objectName string, contents string, contentType string) (err error)
- func (c *Connection) ObjectSymlinkCreate(container string, symlink string, targetAccount string, targetContainer string, ...) (headers Headers, err error)
- func (c *Connection) ObjectTempUrl(container string, objectName string, secretKey string, method string, ...) string
- func (c *Connection) ObjectUpdate(container string, objectName string, h Headers) error
- func (c *Connection) ObjectUpdateContentType(container string, objectName string, contentType string) (err error)
- func (c *Connection) Objects(container string, opts *ObjectsOpts) ([]Object, error)
- func (c *Connection) ObjectsAll(container string, opts *ObjectsOpts) ([]Object, error)
- func (c *Connection) ObjectsWalk(container string, opts *ObjectsOpts, walkFn ObjectsWalkFn) error
- func (c *Connection) QueryInfo() (infos SwiftInfo, err error)
- func (c *Connection) StaticLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)
- func (c *Connection) StaticLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)
- func (c *Connection) StaticLargeObjectDelete(container string, path string) error
- func (c *Connection) StaticLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, ...) error
- func (c *Connection) UnAuthenticate()
- func (c *Connection) VersionContainerCreate(current, version string) error
- func (c *Connection) VersionDisable(current string) error
- func (c *Connection) VersionEnable(current, version string) error
- func (c *Connection) VersionObjectList(version, object string) ([]string, error)
- type Container
- type ContainersOpts
- type CustomEndpointAuthenticator
- type DynamicLargeObjectCreateFile
- func (file *DynamicLargeObjectCreateFile) Close() error
- func (file *DynamicLargeObjectCreateFile) Flush() error
- func (file *DynamicLargeObjectCreateFile) Seek(offset int64, whence int) (int64, error)
- func (file *DynamicLargeObjectCreateFile) Size() int64
- func (file *DynamicLargeObjectCreateFile) Write(buf []byte) (int, error)
- type EndpointType
- type Error
- type Expireser
- type Headers
- func (h Headers) AccountMetadata() Metadata
- func (h Headers) ContainerMetadata() Metadata
- func (headers Headers) IsLargeObject() bool
- func (headers Headers) IsLargeObjectDLO() bool
- func (headers Headers) IsLargeObjectSLO() bool
- func (h Headers) Metadata(metaPrefix string) Metadata
- func (h Headers) ObjectMetadata() Metadata
- type LargeObjectFile
- type LargeObjectOpts
- type Metadata
- type Object
- type ObjectCreateFile
- type ObjectOpenFile
- type ObjectType
- type ObjectsOpts
- type ObjectsWalkFn
- type RequestOpts
- type StaticLargeObjectCreateFile
- func (file *StaticLargeObjectCreateFile) Close() error
- func (file *StaticLargeObjectCreateFile) Flush() error
- func (file *StaticLargeObjectCreateFile) Seek(offset int64, whence int) (int64, error)
- func (file *StaticLargeObjectCreateFile) Size() int64
- func (file *StaticLargeObjectCreateFile) Write(buf []byte) (int, error)
- type SwiftInfo
Examples ¶
Constants ¶
const ( // Use public URL as storage URL EndpointTypePublic = EndpointType("public") // Use internal URL as storage URL EndpointTypeInternal = EndpointType("internal") // Use admin URL as storage URL EndpointTypeAdmin = EndpointType("admin") )
const ( DefaultUserAgent = "goswift/1.0" // Default user agent DefaultRetries = 3 // Default number of retries on token expiry TimeFormat = "2006-01-02T15:04:05" // Python date format for json replies parsed as UTC UploadTar = "tar" // Data format specifier for Connection.BulkUpload(). UploadTarGzip = "tar.gz" // Data format specifier for Connection.BulkUpload(). UploadTarBzip2 = "tar.bz2" // Data format specifier for Connection.BulkUpload(). )
const IS_AT_LEAST_GO_16 = true
Variables ¶
var ( // Specific Errors you might want to check for equality NotModified = newError(304, "Not Modified") BadRequest = newError(400, "Bad Request") AuthorizationFailed = newError(401, "Authorization Failed") ContainerNotFound = newError(404, "Container Not Found") ContainerNotEmpty = newError(409, "Container Not Empty") ObjectNotFound = newError(404, "Object Not Found") ObjectCorrupted = newError(422, "Object Corrupted") TimeoutError = newError(408, "Timeout when reading or writing data") Forbidden = newError(403, "Operation forbidden") TooLargeObject = newError(413, "Too Large Object") RateLimit = newError(498, "Rate Limit") TooManyRequests = newError(429, "TooManyRequests") // Mappings for container errors ContainerErrorMap = errorMap{ 400: BadRequest, 403: Forbidden, 404: ContainerNotFound, 409: ContainerNotEmpty, 498: RateLimit, } )
var NotLargeObject = errors.New("Not a large object")
NotLargeObject is returned if an operation is performed on an object which isn't large.
var SLONotSupported = errors.New("SLO not supported")
Functions ¶
func AddExpectAndTransferEncoding ¶ added in v1.0.43
func FloatStringToTime ¶
FloatStringToTime converts a floating point number string to a time.Time
The string is floating point number of seconds since the epoch (Unix time). The number should be in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z"
Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.
If an error is returned then time will be returned as the zero time.
func SetExpectContinueTimeout ¶ added in v1.0.43
func TimeToFloatString ¶
TimeToFloatString converts a time.Time object to a floating point string
The string is floating point number of seconds since the epoch (Unix time). The number is in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z". Trailing zeros will be dropped from the output.
Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.
Types ¶
type Account ¶
type Account struct { BytesUsed int64 // total number of bytes used Containers int64 // total number of containers Objects int64 // total number of objects }
Account contains information about this account.
type Authenticator ¶ added in v1.0.2
type Authenticator interface { // Request creates an http.Request for the auth - return nil if not needed Request(*Connection) (*http.Request, error) // Response parses the http.Response Response(resp *http.Response) error // The public storage URL - set Internal to true to read // internal/service net URL StorageUrl(Internal bool) string // The access token Token() string // The CDN url if available CdnUrl() string }
Auth defines the operations needed to authenticate with swift
This encapsulates the different authentication schemes in use
type BulkDeleteResult ¶ added in v1.0.3
type BulkDeleteResult struct { NumberNotFound int64 // # of objects not found. NumberDeleted int64 // # of deleted objects. Errors map[string]error // Mapping between object name and an error. Headers Headers // Response HTTP headers. }
BulkDeleteResult stores results of BulkDelete().
Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where the object was to be deleted, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".
type BulkUploadResult ¶ added in v1.0.4
type BulkUploadResult struct { NumberCreated int64 // # of created objects. Errors map[string]error // Mapping between object name and an error. Headers Headers // Response HTTP headers. }
BulkUploadResult stores results of BulkUpload().
Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where an object was to be created, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".
type Connection ¶
type Connection struct { // Parameters - fill these in before calling Authenticate // They are all optional except UserName, ApiKey and AuthUrl Domain string // User's domain name DomainId string // User's domain Id UserName string // UserName for api UserId string // User Id ApiKey string // Key for api access ApplicationCredentialId string // Application Credential ID ApplicationCredentialName string // Application Credential Name ApplicationCredentialSecret string // Application Credential Secret AuthUrl string // Auth URL Retries int // Retries on error (default is 3) UserAgent string // Http User agent (default goswift/1.0) ConnectTimeout time.Duration // Connect channel timeout (default 10s) Timeout time.Duration // Data channel timeout (default 60s) Region string // Region to use eg "LON", "ORD" - default is use first region (v2,v3 auth only) AuthVersion int // Set to 1, 2 or 3 or leave at 0 for autodetect Internal bool // Set this to true to use the the internal / service network Tenant string // Name of the tenant (v2,v3 auth only) TenantId string // Id of the tenant (v2,v3 auth only) EndpointType EndpointType // Endpoint type (v2,v3 auth only) (default is public URL unless Internal is set) TenantDomain string // Name of the tenant's domain (v3 auth only), only needed if it differs from the user domain TenantDomainId string // Id of the tenant's domain (v3 auth only), only needed if it differs the from user domain TrustId string // Id of the trust (v3 auth only) Transport http.RoundTripper `json:"-" xml:"-"` // Optional specialised http.Transport (eg. for Google Appengine) // These are filled in after Authenticate is called as are the defaults for above StorageUrl string AuthToken string Expires time.Time // time the token expires, may be Zero if unknown Auth Authenticator `json:"-" xml:"-"` // the current authenticator // contains filtered or unexported fields }
Connection holds the details of the connection to the swift server.
You need to provide UserName, ApiKey and AuthUrl when you create a connection then call Authenticate on it.
The auth version in use will be detected from the AuthURL - you can override this with the AuthVersion parameter.
If using v2 auth you can also set Region in the Connection structure. If you don't set Region you will get the default region which may not be what you want.
For reference some common AuthUrls looks like this:
Rackspace US https://auth.api.rackspacecloud.com/v1.0 Rackspace UK https://lon.auth.api.rackspacecloud.com/v1.0 Rackspace v2 https://identity.api.rackspacecloud.com/v2.0 Memset Memstore UK https://auth.storage.memset.com/v1.0 Memstore v2 https://auth.storage.memset.com/v2.0
When using Google Appengine you must provide the Connection with an appengine-specific Transport:
import ( "appengine/urlfetch" "fmt" "github.com/ncw/swift" ) func handler(w http.ResponseWriter, r *http.Request) { ctx := appengine.NewContext(r) tr := urlfetch.Transport{Context: ctx} c := swift.Connection{ UserName: "user", ApiKey: "key", AuthUrl: "auth_url", Transport: tr, } _ := c.Authenticate() containers, _ := c.ContainerNames(nil) fmt.Fprintf(w, "containers: %q", containers) }
If you don't supply a Transport, one is made which relies on http.ProxyFromEnvironment (http://golang.org/pkg/net/http/#ProxyFromEnvironment). This means that the connection will respect the HTTP proxy specified by the environment variables $HTTP_PROXY and $NO_PROXY.
Example ¶
package main import ( "fmt" "github.com/ncw/swift" ) func main() { // Create a v1 auth connection c := &swift.Connection{ // This should be your username UserName: "user", // This should be your api key ApiKey: "key", // This should be a v1 auth url, eg // Rackspace US https://auth.api.rackspacecloud.com/v1.0 // Rackspace UK https://lon.auth.api.rackspacecloud.com/v1.0 // Memset Memstore UK https://auth.storage.memset.com/v1.0 AuthUrl: "auth_url", } // Authenticate err := c.Authenticate() if err != nil { panic(err) } // List all the containers containers, err := c.ContainerNames(nil) fmt.Println(containers) // etc... // ------ or alternatively create a v2 connection ------ // Create a v2 auth connection c = &swift.Connection{ // This is the sub user for the storage - eg "admin" UserName: "user", // This should be your api key ApiKey: "key", // This should be a version2 auth url, eg // Rackspace v2 https://identity.api.rackspacecloud.com/v2.0 // Memset Memstore v2 https://auth.storage.memset.com/v2.0 AuthUrl: "v2_auth_url", // Region to use - default is use first region if unset Region: "LON", // Name of the tenant - this is likely your username Tenant: "jim", } // as above... }
Output:
func (*Connection) Account ¶
func (c *Connection) Account() (info Account, headers Headers, err error)
Account returns info about the account in an Account struct.
func (*Connection) AccountUpdate ¶
func (c *Connection) AccountUpdate(h Headers) error
AccountUpdate adds, replaces or remove account metadata.
Add or update keys by mentioning them in the Headers.
Remove keys by setting them to an empty string.
func (*Connection) ApplyEnvironment ¶ added in v1.0.34
func (c *Connection) ApplyEnvironment() (err error)
ApplyEnvironment reads environment variables and applies them to the Connection structure. It won't overwrite any parameters which are already set in the Connection struct.
To make a new Connection object entirely from the environment you would do:
c := new(Connection) err := c.ApplyEnvironment() if err != nil { log.Fatal(err) }
The naming of these variables follows the official Openstack naming scheme so it should be compatible with OpenStack rc files.
For v1 authentication (obsolete)
ST_AUTH - Auth URL ST_USER - UserName for api ST_KEY - Key for api access
For v2 authentication
OS_AUTH_URL - Auth URL OS_USERNAME - UserName for api OS_PASSWORD - Key for api access OS_TENANT_NAME - Name of the tenant OS_TENANT_ID - Id of the tenant OS_REGION_NAME - Region to use - default is use first region
For v3 authentication
OS_AUTH_URL - Auth URL OS_USERNAME - UserName for api OS_USER_ID - User Id OS_PASSWORD - Key for api access OS_APPLICATION_CREDENTIAL_ID - Application Credential ID OS_APPLICATION_CREDENTIAL_NAME - Application Credential Name OS_APPLICATION_CREDENTIAL_SECRET - Application Credential Secret OS_USER_DOMAIN_NAME - User's domain name OS_USER_DOMAIN_ID - User's domain Id OS_PROJECT_NAME - Name of the project OS_PROJECT_DOMAIN_NAME - Name of the tenant's domain, only needed if it differs from the user domain OS_PROJECT_DOMAIN_ID - Id of the tenant's domain, only needed if it differs the from user domain OS_TRUST_ID - If of the trust OS_REGION_NAME - Region to use - default is use first region
Other
OS_ENDPOINT_TYPE - Endpoint type public, internal or admin ST_AUTH_VERSION - Choose auth version - 1, 2 or 3 or leave at 0 for autodetect
For manual authentication
OS_STORAGE_URL - storage URL from alternate authentication OS_AUTH_TOKEN - Auth Token from alternate authentication
Library specific
GOSWIFT_RETRIES - Retries on error (default is 3) GOSWIFT_USER_AGENT - HTTP User agent (default goswift/1.0) GOSWIFT_CONNECT_TIMEOUT - Connect channel timeout with unit, eg "10s", "100ms" (default "10s") GOSWIFT_TIMEOUT - Data channel timeout with unit, eg "10s", "100ms" (default "60s") GOSWIFT_INTERNAL - Set this to "true" to use the the internal network (obsolete - use OS_ENDPOINT_TYPE)
func (*Connection) Authenticate ¶
func (c *Connection) Authenticate() (err error)
Authenticate connects to the Swift server.
If you don't call it before calling one of the connection methods then it will be called for you on the first access.
func (*Connection) Authenticated ¶
func (c *Connection) Authenticated() bool
Authenticated returns a boolean to show if the current connection is authenticated.
Doesn't actually check the credentials against the server.
func (*Connection) BulkDelete ¶ added in v1.0.3
func (c *Connection) BulkDelete(container string, objectNames []string) (result BulkDeleteResult, err error)
BulkDelete deletes multiple objectNames from container in one operation.
Some servers may not accept bulk-delete requests since bulk-delete is an optional feature of swift - these will return the Forbidden error.
See also: * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-bulk-delete.html * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html
func (*Connection) BulkDeleteHeaders ¶ added in v1.0.52
func (c *Connection) BulkDeleteHeaders(container string, objectNames []string, h Headers) (result BulkDeleteResult, err error)
BulkDeleteHeaders deletes multiple objectNames from container in one operation.
Some servers may not accept bulk-delete requests since bulk-delete is an optional feature of swift - these will return the Forbidden error.
See also: * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-bulk-delete.html * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html
func (*Connection) BulkUpload ¶ added in v1.0.4
func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string, h Headers) (result BulkUploadResult, err error)
BulkUpload uploads multiple files in one operation.
uploadPath can be empty, a container name, or a pseudo-directory within a container. If uploadPath is empty, new containers may be automatically created.
Files are read from dataStream. The format of the stream is specified by the format parameter. Available formats are: * UploadTar - Plain tar stream. * UploadTarGzip - Gzip compressed tar stream. * UploadTarBzip2 - Bzip2 compressed tar stream.
Some servers may not accept bulk-upload requests since bulk-upload is an optional feature of swift - these will return the Forbidden error.
See also: * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-extract-archive.html * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Extract_Archive-d1e2338.html
func (*Connection) Call ¶ added in v1.0.1
func (c *Connection) Call(targetUrl string, p RequestOpts) (resp *http.Response, headers Headers, err error)
Call runs a remote command on the targetUrl, returns a response, headers and possible error.
operation is GET, HEAD etc container is the name of a container Any other parameters (if not None) are added to the targetUrl
Returns a response or an error. If response is returned then the resp.Body must be read completely and resp.Body.Close() must be called on it, unless noResponse is set in which case the body will be closed in this function
If "Content-Length" is set in p.Headers it will be used - this can be used to override the default chunked transfer encoding for uploads.
This will Authenticate if necessary, and re-authenticate if it receives a 401 error which means the token has expired
This method is exported so extensions can call it.
func (*Connection) Container ¶
func (c *Connection) Container(container string) (info Container, headers Headers, err error)
Container returns info about a single container including any metadata in the headers.
func (*Connection) ContainerCreate ¶
func (c *Connection) ContainerCreate(container string, h Headers) error
ContainerCreate creates a container.
If you don't want to add Headers just pass in nil
No error is returned if it already exists but the metadata if any will be updated.
func (*Connection) ContainerDelete ¶
func (c *Connection) ContainerDelete(container string) error
ContainerDelete deletes a container.
May return ContainerDoesNotExist or ContainerNotEmpty
func (*Connection) ContainerNames ¶
func (c *Connection) ContainerNames(opts *ContainersOpts) ([]string, error)
ContainerNames returns a slice of names of containers in this account.
func (*Connection) ContainerNamesAll ¶
func (c *Connection) ContainerNamesAll(opts *ContainersOpts) ([]string, error)
ContainerNamesAll is like ContainerNames but it returns all the Containers
It calls ContainerNames multiple times using the Marker parameter ¶
It has a default Limit parameter but you may pass in your own
func (*Connection) ContainerUpdate ¶
func (c *Connection) ContainerUpdate(container string, h Headers) error
ContainerUpdate adds, replaces or removes container metadata.
Add or update keys by mentioning them in the Metadata.
Remove keys by setting them to an empty string.
Container metadata can only be read with Container() not with Containers().
func (*Connection) Containers ¶
func (c *Connection) Containers(opts *ContainersOpts) ([]Container, error)
Containers returns a slice of structures with full information as described in Container.
func (*Connection) ContainersAll ¶
func (c *Connection) ContainersAll(opts *ContainersOpts) ([]Container, error)
ContainersAll is like Containers but it returns all the Containers
It calls Containers multiple times using the Marker parameter ¶
It has a default Limit parameter but you may pass in your own
func (*Connection) DynamicLargeObjectCreate ¶ added in v1.0.26
func (c *Connection) DynamicLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)
DynamicLargeObjectCreate creates or truncates an existing dynamic large object returning a writeable object. This sets opts.Flags to an appropriate value before calling DynamicLargeObjectCreateFile
func (*Connection) DynamicLargeObjectCreateFile ¶ added in v1.0.26
func (c *Connection) DynamicLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)
DynamicLargeObjectCreateFile creates a dynamic large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passes to the largeObjectCreate method.
func (*Connection) DynamicLargeObjectDelete ¶ added in v1.0.26
func (c *Connection) DynamicLargeObjectDelete(container string, path string) error
DynamicLargeObjectDelete deletes a dynamic large object and all of its segments.
func (*Connection) DynamicLargeObjectMove ¶ added in v1.0.26
func (c *Connection) DynamicLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) error
DynamicLargeObjectMove moves a dynamic large object from srcContainer, srcObjectName to dstContainer, dstObjectName
func (*Connection) LargeObjectDelete ¶ added in v1.0.26
func (c *Connection) LargeObjectDelete(container string, objectName string) error
LargeObjectDelete deletes the large object named by container, path
func (*Connection) LargeObjectGetSegments ¶ added in v1.0.26
func (c *Connection) LargeObjectGetSegments(container string, path string) (string, []Object, error)
LargeObjectGetSegments returns all the segments that compose an object If the object is a Dynamic Large Object (DLO), it just returns the objects that have the prefix as indicated by the manifest. If the object is a Static Large Object (SLO), it retrieves the JSON content of the manifest and return all the segments of it.
func (*Connection) Object ¶
func (c *Connection) Object(container string, objectName string) (info Object, headers Headers, err error)
Object returns info about a single object including any metadata in the header.
May return ObjectNotFound.
Use headers.ObjectMetadata() to read the metadata in the Headers.
func (*Connection) ObjectCopy ¶
func (c *Connection) ObjectCopy(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string, h Headers) (headers Headers, err error)
ObjectCopy does a server side copy of an object to a new position
All metadata is preserved. If metadata is set in the headers then it overrides the old metadata on the copied object.
The destination container must exist before the copy.
You can use this to copy an object to itself - this is the only way to update the content type of an object.
func (*Connection) ObjectCreate ¶
func (c *Connection) ObjectCreate(container string, objectName string, checkHash bool, Hash string, contentType string, h Headers) (file *ObjectCreateFile, err error)
ObjectCreate creates or updates the object in the container. It returns an io.WriteCloser you should write the contents to. You MUST call Close() on it and you MUST check the error return from Close().
If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted on Close()
If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted on Close() if it is incorrect.
If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".
If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension
func (*Connection) ObjectDelete ¶
func (c *Connection) ObjectDelete(container string, objectName string) error
ObjectDelete deletes the object.
May return ObjectNotFound if the object isn't found
func (*Connection) ObjectGet ¶
func (c *Connection) ObjectGet(container string, objectName string, contents io.Writer, checkHash bool, h Headers) (headers Headers, err error)
ObjectGet gets the object into the io.Writer contents.
Returns the headers of the response.
If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.
headers["Content-Type"] will give the content type if desired.
func (*Connection) ObjectGetBytes ¶
func (c *Connection) ObjectGetBytes(container string, objectName string) (contents []byte, err error)
ObjectGetBytes returns an object as a []byte.
This is a simplified interface which checks the MD5
func (*Connection) ObjectGetString ¶
func (c *Connection) ObjectGetString(container string, objectName string) (contents string, err error)
ObjectGetString returns an object as a string.
This is a simplified interface which checks the MD5
func (*Connection) ObjectMove ¶
func (c *Connection) ObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) (err error)
ObjectMove does a server side move of an object to a new position
This is a convenience method which calls ObjectCopy then ObjectDelete ¶
All metadata is preserved.
The destination container must exist before the copy.
func (*Connection) ObjectNames ¶
func (c *Connection) ObjectNames(container string, opts *ObjectsOpts) ([]string, error)
ObjectNames returns a slice of names of objects in a given container.
func (*Connection) ObjectNamesAll ¶
func (c *Connection) ObjectNamesAll(container string, opts *ObjectsOpts) ([]string, error)
ObjectNamesAll is like ObjectNames but it returns all the Objects
It calls ObjectNames multiple times using the Marker parameter. Marker is reset unless KeepMarker is set
It has a default Limit parameter but you may pass in your own
func (*Connection) ObjectOpen ¶
func (c *Connection) ObjectOpen(container string, objectName string, checkHash bool, h Headers) (file *ObjectOpenFile, headers Headers, err error)
ObjectOpen returns an ObjectOpenFile for reading the contents of the object. This satisfies the io.ReadCloser and the io.Seeker interfaces.
You must call Close() on contents when finished ¶
Returns the headers of the response.
If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted. It will also check the length returned. No checking will be done if you don't read all the contents.
Note that objects with X-Object-Manifest or X-Static-Large-Object set won't ever have their md5sum's checked as the md5sum reported on the object is actually the md5sum of the md5sums of the parts. This isn't very helpful to detect a corrupted download as the size of the parts aren't known without doing more operations. If you want to ensure integrity of an object with a manifest then you will need to download everything in the manifest separately.
headers["Content-Type"] will give the content type if desired.
func (*Connection) ObjectPut ¶
func (c *Connection) ObjectPut(container string, objectName string, contents io.Reader, checkHash bool, Hash string, contentType string, h Headers) (headers Headers, err error)
ObjectPut creates or updates the path in the container from contents. contents should be an open io.Reader which will have all its contents read.
This is a low level interface.
If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.
If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted if it is incorrect.
If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".
If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension
func (*Connection) ObjectPutBytes ¶
func (c *Connection) ObjectPutBytes(container string, objectName string, contents []byte, contentType string) (err error)
ObjectPutBytes creates an object from a []byte in a container.
This is a simplified interface which checks the MD5.
func (*Connection) ObjectPutString ¶
func (c *Connection) ObjectPutString(container string, objectName string, contents string, contentType string) (err error)
ObjectPutString creates an object from a string in a container.
This is a simplified interface which checks the MD5
func (*Connection) ObjectSymlinkCreate ¶ added in v1.0.50
func (*Connection) ObjectTempUrl ¶ added in v1.0.16
func (c *Connection) ObjectTempUrl(container string, objectName string, secretKey string, method string, expires time.Time) string
ObjectTempUrl returns a temporary URL for an object
func (*Connection) ObjectUpdate ¶
func (c *Connection) ObjectUpdate(container string, objectName string, h Headers) error
ObjectUpdate adds, replaces or removes object metadata.
Add or Update keys by mentioning them in the Metadata. Use Metadata.ObjectHeaders and Headers.ObjectMetadata to convert your Metadata to and from normal HTTP headers.
This removes all metadata previously added to the object and replaces it with that passed in so to delete keys, just don't mention them the headers you pass in.
Object metadata can only be read with Object() not with Objects().
This can also be used to set headers not already assigned such as X-Delete-At or X-Delete-After for expiring objects.
You cannot use this to change any of the object's other headers such as Content-Type, ETag, etc.
Refer to copying an object when you need to update metadata or other headers such as Content-Type or CORS headers.
May return ObjectNotFound.
func (*Connection) ObjectUpdateContentType ¶
func (c *Connection) ObjectUpdateContentType(container string, objectName string, contentType string) (err error)
ObjectUpdateContentType updates the content type of an object
This is a convenience method which calls ObjectCopy ¶
All other metadata is preserved.
func (*Connection) Objects ¶
func (c *Connection) Objects(container string, opts *ObjectsOpts) ([]Object, error)
Objects returns a slice of Object with information about each object in the container.
If Delimiter is set in the opts then PseudoDirectory may be set, with ContentType 'application/directory'. These are not real objects but represent directories of objects which haven't had an object created for them.
func (*Connection) ObjectsAll ¶
func (c *Connection) ObjectsAll(container string, opts *ObjectsOpts) ([]Object, error)
ObjectsAll is like Objects but it returns an unlimited number of Objects in a slice
It calls Objects multiple times using the Marker parameter
func (*Connection) ObjectsWalk ¶
func (c *Connection) ObjectsWalk(container string, opts *ObjectsOpts, walkFn ObjectsWalkFn) error
ObjectsWalk is uses to iterate through all the objects in chunks as returned by Objects or ObjectNames using the Marker and Limit parameters in the ObjectsOpts.
Pass in a closure `walkFn` which calls Objects or ObjectNames with the *ObjectsOpts passed to it and does something with the results.
Errors will be returned from this function ¶
It has a default Limit parameter but you may pass in your own
Example ¶
c, rollback := makeConnection(nil) defer rollback() objects := make([]string, 0) err := c.ObjectsWalk(container, nil, func(opts *swift.ObjectsOpts) (interface{}, error) { newObjects, err := c.ObjectNames(container, opts) if err == nil { objects = append(objects, newObjects...) } return newObjects, err }) fmt.Println("Found all the objects", objects, err)
Output:
func (*Connection) QueryInfo ¶ added in v1.0.19
func (c *Connection) QueryInfo() (infos SwiftInfo, err error)
Discover Swift configuration by doing a request against /info
func (*Connection) StaticLargeObjectCreate ¶ added in v1.0.26
func (c *Connection) StaticLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)
StaticLargeObjectCreate creates or truncates an existing static large object returning a writeable object. This sets opts.Flags to an appropriate value before calling StaticLargeObjectCreateFile
func (*Connection) StaticLargeObjectCreateFile ¶ added in v1.0.26
func (c *Connection) StaticLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)
StaticLargeObjectCreateFile creates a static large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passed to the largeObjectCreate method.
func (*Connection) StaticLargeObjectDelete ¶ added in v1.0.26
func (c *Connection) StaticLargeObjectDelete(container string, path string) error
StaticLargeObjectDelete deletes a static large object and all of its segments.
func (*Connection) StaticLargeObjectMove ¶ added in v1.0.26
func (c *Connection) StaticLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) error
StaticLargeObjectMove moves a static large object from srcContainer, srcObjectName to dstContainer, dstObjectName
func (*Connection) UnAuthenticate ¶
func (c *Connection) UnAuthenticate()
UnAuthenticate removes the authentication from the Connection.
func (*Connection) VersionContainerCreate ¶ added in v1.0.1
func (c *Connection) VersionContainerCreate(current, version string) error
VersionContainerCreate is a helper method for creating and enabling version controlled containers.
It builds the current object container, the non-current object version container, and enables versioning.
If the server doesn't support versioning then it will return Forbidden however it will have created both the containers at that point.
Example ¶
c, rollback := makeConnection(nil) defer rollback() // Use the helper method to create the current and versions container. if err := c.VersionContainerCreate("cds", "cd-versions"); err != nil { fmt.Print(err.Error()) }
Output:
func (*Connection) VersionDisable ¶ added in v1.0.1
func (c *Connection) VersionDisable(current string) error
VersionDisable disables versioning on the current container.
Example ¶
c, rollback := makeConnection(nil) defer rollback() // Disable versioning on a container. Note that this does not delete the versioning container. c.VersionDisable("movies")
Output:
func (*Connection) VersionEnable ¶ added in v1.0.1
func (c *Connection) VersionEnable(current, version string) error
VersionEnable enables versioning on the current container with version as the tracking container.
May return Forbidden if this isn't supported by the server
Example ¶
c, rollback := makeConnection(nil) defer rollback() // Build the containers manually and enable them. if err := c.ContainerCreate("movie-versions", nil); err != nil { fmt.Print(err.Error()) } if err := c.ContainerCreate("movies", nil); err != nil { fmt.Print(err.Error()) } if err := c.VersionEnable("movies", "movie-versions"); err != nil { fmt.Print(err.Error()) } // Access the primary container as usual with ObjectCreate(), ObjectPut(), etc. // etc...
Output:
func (*Connection) VersionObjectList ¶ added in v1.0.1
func (c *Connection) VersionObjectList(version, object string) ([]string, error)
VersionObjectList returns a list of older versions of the object.
Objects are returned in the format <length><object_name>/<timestamp>
type Container ¶
type Container struct { Name string // Name of the container Count int64 // Number of objects in the container Bytes int64 // Total number of bytes used in the container }
Container contains information about a container
type ContainersOpts ¶
type ContainersOpts struct { Limit int // For an integer value n, limits the number of results to at most n values. Prefix string // Given a string value x, return container names matching the specified prefix. Marker string // Given a string value x, return container names greater in value than the specified marker. EndMarker string // Given a string value x, return container names less in value than the specified marker. Headers Headers // Any additional HTTP headers - can be nil }
ContainersOpts is options for Containers() and ContainerNames()
type CustomEndpointAuthenticator ¶ added in v1.0.24
type CustomEndpointAuthenticator interface {
StorageUrlForEndpoint(endpointType EndpointType) string
}
type DynamicLargeObjectCreateFile ¶ added in v1.0.26
type DynamicLargeObjectCreateFile struct {
// contains filtered or unexported fields
}
DynamicLargeObjectCreateFile represents an open static large object
func (*DynamicLargeObjectCreateFile) Close ¶ added in v1.0.26
func (file *DynamicLargeObjectCreateFile) Close() error
Close satisfies the io.Closer interface
func (*DynamicLargeObjectCreateFile) Flush ¶ added in v1.0.27
func (file *DynamicLargeObjectCreateFile) Flush() error
func (*DynamicLargeObjectCreateFile) Seek ¶ added in v1.0.27
Seek sets the offset for the next write operation
type EndpointType ¶ added in v1.0.24
type EndpointType string
type Error ¶
Error - all errors generated by this package are of this type. Other error may be passed on from library functions though.
type Expireser ¶ added in v1.0.46
Expireser is an optional interface to read the expiration time of the token
type Headers ¶
Headers stores HTTP headers (can only have one of each header like Swift).
func (Headers) AccountMetadata ¶
AccountMetadata converts Headers from account to a Metadata.
The keys in the Metadata will be converted to lower case.
func (Headers) ContainerMetadata ¶
ContainerMetadata converts Headers from container to a Metadata.
The keys in the Metadata will be converted to lower case.
func (Headers) IsLargeObject ¶ added in v1.0.27
func (Headers) IsLargeObjectDLO ¶ added in v1.0.27
func (Headers) IsLargeObjectSLO ¶ added in v1.0.27
func (Headers) Metadata ¶
Metadata gets the Metadata starting with the metaPrefix out of the Headers.
The keys in the Metadata will be converted to lower case
func (Headers) ObjectMetadata ¶
ObjectMetadata converts Headers from object to a Metadata.
The keys in the Metadata will be converted to lower case.
type LargeObjectFile ¶ added in v1.0.27
type LargeObjectOpts ¶ added in v1.0.26
type LargeObjectOpts struct { Container string // Name of container to place object ObjectName string // Name of object Flags int // Creation flags CheckHash bool // If set Check the hash Hash string // If set use this hash to check ContentType string // Content-Type of the object Headers Headers // Additional headers to upload the object with ChunkSize int64 // Size of chunks of the object, defaults to 10MB if not set MinChunkSize int64 // Minimum chunk size, automatically set for SLO's based on info SegmentContainer string // Name of the container to place segments SegmentPrefix string // Prefix to use for the segments NoBuffer bool // Prevents using a bufio.Writer to write segments }
LargeObjectOpts describes how a large object should be created
type Metadata ¶
Metadata stores account, container or object metadata.
func (Metadata) AccountHeaders ¶
AccountHeaders converts the Metadata for the account.
func (Metadata) ContainerHeaders ¶
ContainerHeaders converts the Metadata for the container.
func (Metadata) GetModTime ¶
GetModTime reads a modification time (mtime) from a Metadata object
This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.
If an error is returned then time will be returned as the zero time.
func (Metadata) Headers ¶
Headers convert the Metadata starting with the metaPrefix into a Headers.
The keys in the Metadata will be converted from lower case to http Canonical (see http.CanonicalHeaderKey).
func (Metadata) ObjectHeaders ¶
ObjectHeaders converts the Metadata for the object.
func (Metadata) SetModTime ¶
SetModTime writes an modification time (mtime) to a Metadata object
This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.
type Object ¶
type Object struct { Name string `json:"name"` // object name ContentType string `json:"content_type"` // eg application/directory Bytes int64 `json:"bytes"` // size in bytes ServerLastModified string `json:"last_modified"` // Last modified time, eg '2011-06-30T08:20:47.736680' as a string supplied by the server LastModified time.Time // Last modified time converted to a time.Time Hash string `json:"hash"` // MD5 hash, eg "d41d8cd98f00b204e9800998ecf8427e" SLOHash string `json:"slo_etag"` // MD5 hash of all segments' MD5 hash, eg "d41d8cd98f00b204e9800998ecf8427e" PseudoDirectory bool // Set when using delimiter to show that this directory object does not really exist SubDir string `json:"subdir"` // returned only when using delimiter to mark "pseudo directories" ObjectType ObjectType // type of this object }
Object contains information about an object
type ObjectCreateFile ¶
type ObjectCreateFile struct {
// contains filtered or unexported fields
}
ObjectCreateFile represents a swift object open for writing
func (*ObjectCreateFile) Close ¶
func (file *ObjectCreateFile) Close() error
Close the object and checks the md5sum if it was required.
Also returns any other errors from the server (eg container not found) so it is very important to check the errors on this method.
func (*ObjectCreateFile) CloseWithError ¶ added in v1.0.51
func (file *ObjectCreateFile) CloseWithError(err error) error
CloseWithError closes the object, aborting the upload.
func (*ObjectCreateFile) Headers ¶ added in v1.0.29
func (file *ObjectCreateFile) Headers() (Headers, error)
Headers returns the response headers from the created object if the upload has been completed. The Close() method must be called on an ObjectCreateFile before this method.
type ObjectOpenFile ¶
type ObjectOpenFile struct {
// contains filtered or unexported fields
}
ObjectOpenFile represents a swift object open for reading
func (*ObjectOpenFile) Close ¶
func (file *ObjectOpenFile) Close() (err error)
Close the object and checks the length and md5sum if it was required and all the object was read
func (*ObjectOpenFile) Length ¶ added in v1.0.5
func (file *ObjectOpenFile) Length() (int64, error)
Length gets the objects content length either from a cached copy or from the server.
func (*ObjectOpenFile) Read ¶
func (file *ObjectOpenFile) Read(p []byte) (n int, err error)
Read bytes from the object - see io.Reader
func (*ObjectOpenFile) Seek ¶
func (file *ObjectOpenFile) Seek(offset int64, whence int) (newPos int64, err error)
Seek sets the offset for the next Read to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. Seek returns the new offset and an Error, if any.
Seek uses HTTP Range headers which, if the file pointer is moved, will involve reopening the HTTP connection.
Note that you can't seek to the end of a file or beyond; HTTP Range requests don't support the file pointer being outside the data, unlike os.File
Seek(0, 1) will return the current file pointer.
type ObjectType ¶ added in v1.0.26
type ObjectType int
ObjectType is the type of the swift object, regular, static large, or dynamic large.
const ( RegularObjectType ObjectType = iota StaticLargeObjectType DynamicLargeObjectType )
Values that ObjectType can take
type ObjectsOpts ¶
type ObjectsOpts struct { Limit int // For an integer value n, limits the number of results to at most n values. Marker string // Given a string value x, return object names greater in value than the specified marker. EndMarker string // Given a string value x, return object names less in value than the specified marker Prefix string // For a string value x, causes the results to be limited to object names beginning with the substring x. Path string // For a string value x, return the object names nested in the pseudo path Delimiter rune // For a character c, return all the object names nested in the container Headers Headers // Any additional HTTP headers - can be nil KeepMarker bool // Do not reset Marker when using ObjectsAll or ObjectNamesAll }
ObjectOpts is options for Objects() and ObjectNames()
type ObjectsWalkFn ¶
type ObjectsWalkFn func(*ObjectsOpts) (interface{}, error)
A closure defined by the caller to iterate through all objects
Call Objects or ObjectNames from here with the *ObjectOpts passed in
Do whatever is required with the results then return them
type RequestOpts ¶ added in v1.0.1
type RequestOpts struct { Container string ObjectName string Operation string Parameters url.Values Headers Headers ErrorMap errorMap NoResponse bool Body io.Reader Retries int // if set this is called on re-authentication to refresh the targetUrl OnReAuth func() (string, error) }
RequestOpts contains parameters for Connection.storage.
type StaticLargeObjectCreateFile ¶ added in v1.0.26
type StaticLargeObjectCreateFile struct {
// contains filtered or unexported fields
}
StaticLargeObjectCreateFile represents an open static large object
func (*StaticLargeObjectCreateFile) Close ¶ added in v1.0.26
func (file *StaticLargeObjectCreateFile) Close() error
func (*StaticLargeObjectCreateFile) Flush ¶ added in v1.0.27
func (file *StaticLargeObjectCreateFile) Flush() error
func (*StaticLargeObjectCreateFile) Seek ¶ added in v1.0.27
Seek sets the offset for the next write operation
type SwiftInfo ¶ added in v1.0.19
type SwiftInfo map[string]interface{}
SwiftInfo contains the JSON object returned by Swift when the /info route is queried. The object contains, among others, the Swift version, the enabled middlewares and their configuration
func (SwiftInfo) SLOMinSegmentSize ¶ added in v1.0.27
func (SwiftInfo) SupportsBulkDelete ¶ added in v1.0.27
func (SwiftInfo) SupportsSLO ¶ added in v1.0.27
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
This implements a very basic Swift server Everything is stored in memory This comes from the https://github.com/mitchellh/goamz and was adapted for Swift
|
This implements a very basic Swift server Everything is stored in memory This comes from the https://github.com/mitchellh/goamz and was adapted for Swift |