fsagent

README

fsagent

fsagent is a Golang Application to perform various standard actions triggered by various events. FSAgent is highly customizable.

Name

The name FSAgent was originally the shorthand for File System Agent.
With the support of additional triggers and data sources outside of file system events in the monitored folders, a new meaning for the letters F and S must be found. Currently, I prefer the definition Free Service Agent.

Why

WARNING: sad true reality

When planning and developing new (open source) applications / systems, you usually use the latest technologies (containerization (e.g. Docker), message queues (e.g. RabbitMQ, ZeroMQ, ActiveMQ, ...), databases (e.g. PostgreSQL, MariaDB, Redis, MongoDB, ...), ...), but if you're working for a non-startup Company, you often have to deal with old legacy enterprise applications.
These applications do not have modern interfaces, many are decades old. The most modern communication channels of these applications are mostly FTP uploads and emails. I really mean FTP, not those new and fancy SFTP Servers.
However, many of these applications do not even have the functionality to upload, but can only store files in directories and need other applications such as Bat, Blat or Outlook to upload. Bat or Blat is not fundamentally bad, but if your whole business depends on software like Blat, you have a big problem.
In my free time, I have written this program for replacing such "interfaces". It is not just a replacement for Bat or Outlook, it monitors directories and executes predefined actions for new files. The configuration is kept as simple as possible (and will become even easier) to be done by anyone in IT departments, not just programmers.
The first goal was the elimination of the biggest pain in the ass. Gradually, however, it is also planned to extend fsagent for the service composition/orchestration of other protocols such as HTTP(S), AMQP, WebSub (PubSubHubbub).

Install

fsagent can easily installed by the go get-command:

go get simonwaldherr.de/go/fsagent

Config

fsagent can do many things, these can be defined and configured with json files.

start the fsagent daemon with go run fsagent.go config.json or compile a binary (go build) and run it with ./fsagent config.json.

the config.json file could look like:

[
  {
    "verbose": true,
    "debounce": true,
    "folder": "/mnt/prod/Server/Transfer/701/%Y/%m/%d/",
    "trigger": "fsevent",
    "match": "^[0-9]+\\.[Tt][Xx][Tt]$",
    "action": [
      {
        "do": "sleep",
        "config": {
          "time": 500
        },
        "onSuccess": [
          {
            "do": "mail",
            "config": {
              "name": "mail",
              "subject": "Lorem Ipsum",
              "body": "dolor sit amet",
              "from": "notification@company.tld",
              "to": ["example@domain.tld"],
              "cc": ["example2@domain.tld"],
              "bcc": ["example3@domain.tld"],
              "user": "notification",
              "pass": "spring2018",
              "server": "webmail.domain.tld",
              "port": 587
            },
            "onSuccess": [
              {
                "do": "move",
                "config": {
                  "name": "success/$file_%Y%m%d%H%M%S"
                }
              }
            ],
            "onFailure": [
              {
                "do": "move",
                "config": {
                  "name": "error/$file_%Y%m%d%H%M%S"
                }
              }
            ]
          }
        ]
      }
    ]
  }
]

Trigger

Currently there are two triggers available, the most important trigger is filesystem event trigger based on fsnotify. If this is not possible (e.g. if you work on mounted drives and the fs event comes from a different system) you can use a ticker as trigger.

Trigger	Info
fsevent	file system event based on fsnotify
ticker	checks for new files at a customizable frequency

Actions

There are some ready-made actions, but you can easily create others yourself.

Action	Info
Copy	creates a copy of a given file at the specified destination
Delete	removes a file
Move	moves a file to a new location
Decompress	decompresses a file
Compress	compresses a file
HttpPostR.	sends the content of a file in a HTTP Post Request Body
SendMail	sends the file as mail attachment
Sleep	waits for a specified duration

Todo / Contribute

Informations about the license, how to contribute and a list of improvements to do are in separate files.

Documentation

Index

• Variables
• func FileLastModified(filename string) (*time.Time, error)
• type Action
• type Actionable
• type Config
• type Program
  • func (p *Program) Start(s service.Service) error
  • func (p *Program) Stop(s service.Service) error

Variables

var Actions = []Actionable{
	&modules.Mail{},
	&modules.HTTP{},
	&modules.Sleep{},
	&modules.Delete{},
	&modules.Move{},
	&modules.Decompress{},
	&modules.Compress{},
	&modules.IsFile{},
}

Actions is a list of Actionable types that can be added to.

var Done chan bool

var WG sync.WaitGroup

Functions

func FileLastModified

func FileLastModified(filename string) (*time.Time, error)

FileLastModified returns the Time a file was last modified.

Types

type Action

type Action []struct {
	Do        string          `json:"do"`
	Config    json.RawMessage `json:"config"`
	Onsuccess Action          `json:"onSuccess"`
	Onfailure Action          `json:"onFailure"`
}

Action is something that should be performed.

type Actionable

type Actionable interface {
	Name() string
	EmptyConfig() interface{}
	Perform(config interface{}, fileName string) error
}

Actionable defines a common set of methods each action should have.

type Config

type Config struct {
	Folder   string `json:"folder"`
	Trigger  string `json:"trigger"`
	Ticker   int    `json:"ticker"`
	Match    string `json:"match"`
	Action   Action `json:"action"`
	OnlyNew  bool   `json:"onlynew"`
	Verbose  bool   `json:"verbose"`
	Debounce bool   `json:"debounce"`
}

Config represents an element of the application configuration.

type Program

type Program struct {
	// contains filtered or unexported fields
}

func (*Program) Start

func (p *Program) Start(s service.Service) error

func (*Program) Stop

func (p *Program) Stop(s service.Service) error