upload

package module
v0.0.0-...-f7ce013 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 19, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

README 

= caddyv2-upload
:toc:
:source-highlighter: rouge

This repo holds a simple caddyserver v2 upload handler

WARNING: You have to secure the upload URL as the nature of
  this handler/plugin is to save uploaded artifacts on the disc.

In this handler are the following modules in use.

* templates
* file_server
* github.com/caddyserver/jsonc-adapter
* upload :-)

== Configuration

IMPORTANT: The relative paths are relative to the current web root
  directory `http.vars.root`!

=== Environment Variables

.Parameters
[cols="2,6",options=header]
|===
|Name
|Description

|**APPPORT**
|The Port on which Caddyserver should listen. The format depends which Configuration file format you use. See the official Caddyserver documentation for the format https://caddyserver.com/docs/conventions#network-addresses[Network addresses]

|**SKIP_LOG**
|With this variable can you control if the health check will be logged or not. This feature was implemented as part of the vars handler.

|===

=== Parameters

.Parameters
[cols="2,6",options=header]
|===
|Name
|Description

|**dest_dir**
|The directory in which the files will be uploaded

|**root_dir**
|The root directory for this module. When not set will the root
directory be evaluated at runtime from this variable `http.vars.root`.

|**file_field_name**
|The field name for the multi-part form upload

|**fixed_file_name**
|When set, all uploaded files will be renamed to the provided file name.

|**response_template**
|The response template after a upload was successfully

|**max_form_buffer**
|The maximum buffer size for https://pkg.go.dev/net/http#Request.ParseMultipartForm[ParseMultipartForm]. It accepts all size values supported by https://pkg.go.dev/github.com/dustin/go-humanize#pkg-constants[go-humanize]. Here can also be used a https://caddyserver.com/docs/conventions#placeholders[Placeholder] +
JSON: `"{env.MAXBUFFSIZE}"` +
Caddyfile: `{$MAXBUFFSIZE}` +

The default size is **1G**.

|**max_form_buffer_int**
|The maximum buffer size for https://pkg.go.dev/net/http#Request.ParseMultipartForm[ParseMultipartForm]. +
The default size is **1G**.

|**max_filesize**
|is the maximum size in bytes allowed for the upload.
It accepts all size values supported by https://pkg.go.dev/github.com/dustin/go-humanize#pkg-constants[go-humanize]. Reads of more bytes will return an error with HTTP status 413. Here can also be used a https://caddyserver.com/docs/conventions#placeholders[Placeholder] +
JSON: `"{env.MAXFILESIZE}"` +
Caddyfile: `{$MAXFILESIZE}` +

The default size is **1G**.

|**max_filesize_int**
|is the maximum size in bytes allowed for the upload. Reads of more bytes will return an error with HTTP status 413. +
The default size is **1G**.

|**notify_url**
|After a successful upload will this URL be called. The only supported schema is **https**.

|**notify_method**
|The default method is `GET`. If you need to make a `POST` request please open a feature issue
  as for now is a request body handling not implemented

|**insecure**
|This boolean flag configure the `InsecureSkipVerify` in the  https://pkg.go.dev/crypto/tls#Config[tls-config] .
  Default is false which implies that a valid CA must be used

|**capath**
|This is the Parameter where you can define the CA filename for the **notify_url**.

|**create_uuid_dir**
|If set to `true`, each file will get a unique directory with a UUID as its name.

|===

=== Caddy Variables

The following Variables will be added to the request flow

.variables
[cols="2,6",options=header]
|===
|Name
|Description

|**http.upload.filename**
|The uploaded filename

|**http.upload.filesize**
|The uploaded filesize

|**http.upload.directory**
|The directory where the file is uploaded

|**http.upload.uuiddir**
|The name of the random directory created if `create_uuid_dir` is enabled

|===

=== JSON

Because I prefer the https://caddyserver.com/docs/json/[JSON Config ] 
will I write here the configuration snipplet in JSON Syntax.

[source,json]
----
	"handle": [
		{
			"MyTlsSetting": {},
			"dest_dir": "upload", <1>
			"handler": "upload",
			"max_filesize": "5GB", <2>
			"max_form_buffer_int": 1000000, <3>
			"response_template": "upload-resp.txt" <4>
		}
	]

----
<1> Destination Directory on the Server site
<2> Maximal possible upload size
<3> Maximal buffer for uploading
<4> the response template which will be used for response after upload

A full working example is in 
`docker-files/opt/webroot/config/Caddyfile-upload.json`

=== Caddyfile

Here a example Caddyfile which expects that the environment variable
`APPPORT` is set.

[source]
----
{
	order upload before file_server
	log {
		level DEBUG
	}
}

{$APPPORT} {
	root .

	file_server browse
	templates

	@mypost method POST
	upload @mypost {
		dest_dir tmp-upl
		max_form_buffer 1G
		max_filesize 4MB
		response_template templates/upload-resp-template.txt
	}

	log {
		output file access.log
	}
}
----

== build

=== local
[source,shell]
----
xcaddy build --with github.com/kirsch33/realip \
  --with github.com/git001/caddyv2-upload
----

=== docker
[source,shell]
----
buildah bud --tag caddyv2-upload .
# or
docker build --tag caddyv2-upload .
----

== run

=== cli

[source,shell]
----
APPPORT=:2011 ./caddy run \
  -config Caddyfile-upload.json 
----

=== docker

You can get this image from docker hub

The default listen port must be defined with this variable

`APPPORT=:2011`

https://hub.docker.com/r/me2digital/caddyv2-upload

[source,shell]
----
podman run --rm --network host --name caddy-test \
  --env APPPORT=:8888 -it \
  docker.io/me2digital/caddyv2-upload:latest
# or 
docker run --name caddy-test --rm \
  docker.io/me2digital/caddyv2-upload:latest
----

=== OpenShift / kubernetes

You can use the examples in the directory `openshift` to deploy the caddy uploader
into your OpenShift or kubernetes Cluster.

[source,shell]
----
oc new-project caddyupload
oc -n caddyupload apply -f openshift/
----

To adopt the used Caddyfile then can you easily do this with a configmap.
The commands below show the steps.

[source,shell]
----
# add configure change trigger
oc -n caddyupload \
  set triggers \
  deployment.apps/caddy-upload \
  --from-config

# create configmap
oc -n caddyupload \
  create configmap \
  caddyfile-json \
  --from-file=Caddyfile-upload.json=<PATH_TO_LOCAL>/Caddyfile-upload.json

# add configmap to caddy deployment
oc -n caddyupload \
  set volumes \
  deployment.apps/caddy-upload \
  --add --configmap-name=caddyfile-json \
  --mount-path=/opt/webroot/config/Caddyfile-upload.json \
  --name=caddyfile-json \
  --sub-path=Caddyfile-upload.json \
  --type=configmap
----

In case the first shot of the configuration file does not work can you use the following
commands to recreate the configuration file.

[source,shell]
----
oc -n caddyupload-nis delete configmap caddyfile-json \
&& oc -n caddyupload-nis create configmap caddyfile-json --from-file=Caddyfile-upload.json=<PATH_TO_LOCAL>/Caddyfile-upload.json

# output of the commands
configmap "caddyfile-json" deleted
configmap/caddyfile-json created
----

== Health probe

There is a builtin health handler with the path `/health` which just returns
`200` and `Okay`.
The log output can be controled via the environment variable `SKIP_LOG`.

== example cli

When you run the Image with port 8888 can you use curl or any other
tool to post (upload) files

It's not necessary to use `-X POST` as written in this Blog post
https://daniel.haxx.se/blog/2015/09/11/unnecessary-use-of-curl-x/[UNNECESSARY USE OF CURL -X]


Here a example call with curl

[source,shell]
----
curl -v --form myFile=@README.adoc http://localhost:8888/templates/upload-template.html
*   Trying 127.0.0.1:8888...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8888 (#0)
> POST /templates/upload-template.html HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 2492
> Content-Type: multipart/form-data; boundary=------------------------58b770bc61c0e691
> Expect: 100-continue
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 299
< Etag: "rbb1gx8b"
< Last-Modified: Tue, 03 May 2022 11:34:09 GMT
< Server: Caddy
< Date: Thu, 19 May 2022 21:45:07 GMT
< 

http.request.uri.path: {{placeholder "http.request.uri.path"}}

http.request.uuid {{placeholder "http.request.uuid" }}
http.request.host {{placeholder "http.request.host" }}

http.upload.filename: {{placeholder "http.upload.filename"}}
http.upload.filesize: {{placeholder "http.upload.filesize"}}
----

== Background informations

The **max_form_buffer** paramater will be directly passed to https://cs.opensource.google/go/go/+/refs/tags/go1.18.2:src/mime/multipart/formdata.go;l=34;drc=7791e934c882fd103357448aee0fd577b20013ce[readForm] function and is used to check if the uploaded file should be saved temporarly on disk or keep it in the memory. This have dicret impact into the performance and disk usage of that module. Keep in mind when this paramter is low and the upload is a big file then will be there a lot of disk io. +

INFO: The observation from https://github.com/etherwvlf in issue https://github.com/git001/caddyv2-upload/issues/2[Memory issues on large uploads] was that the initial memory usage is 7-8 times higher then the configured **max_form_buffer** size.

Documentation

Index

Constants

View Source 
const (
	Version = "0.17"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Upload 

type Upload struct {
	DestDir          string `json:"dest_dir,omitempty"`
	RootDir          string `json:"root_dir,omitempty"`
	FileFieldName    string `json:"file_field_name,omitempty"`
	FixedFileName    string `json:"fixed_file_name,omitempty"`
	MaxFilesize      int64  `json:"max_filesize_int,omitempty"`
	MaxFilesizeH     string `json:"max_filesize,omitempty"`
	MaxFormBuffer    int64  `json:"max_form_buffer_int,omitempty"`
	MaxFormBufferH   string `json:"max_form_buffer,omitempty"`
	ResponseTemplate string `json:"response_template,omitempty"`
	NotifyURL        string `json:"notify_url,omitempty"`
	NotifyMethod     string `json:"notify_method,omitempty"`
	CreateUuidDir    bool   `json:"create_uuid_dir,omitempty"`

	MyTlsSetting struct {
		InsecureSkipVerify bool   `json:"insecure,omitempty"`
		CAPath             string `json:"capath,omitempty"`
	}
	// contains filtered or unexported fields
}

Middleware implements an HTTP handler that writes the uploaded file to a file on the disk.

func (Upload) CaddyModule 

func (Upload) CaddyModule() caddy.ModuleInfo

CaddyModule returns the Caddy module information.

func (*Upload) Provision 

func (u *Upload) Provision(ctx caddy.Context) error

Provision implements caddy.Provisioner.

func (Upload) SendNotify 

func (u Upload) SendNotify(requuid string) error

func (Upload) ServeHTTP 

func (u Upload) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error

ServeHTTP implements caddyhttp.MiddlewareHandler.

func (*Upload) UnmarshalCaddyfile 

func (u *Upload) UnmarshalCaddyfile(d *caddyfile.Dispenser) error

UnmarshalCaddyfile implements caddyfile.Unmarshaler.

func (*Upload) Validate 

func (u *Upload) Validate() error

Validate implements caddy.Validator.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.