watchman

README

Project Documentation · API Endpoints (Admin Endpoints) · Community · Blog

moov-io/watchman

Moov's mission is to give developers an easy way to create and integrate bank processing into their own software products. Our open source projects are each focused on solving a single responsibility in financial services and designed around performance, scalability, and ease of use.

Moov Watchman offers download, parse, and search functions over numerous trade sanction lists from the United States, agencies, and nonprofits for complying with regional laws. Also included is a web UI and an async webhook notification service to initiate processes on remote systems.

Lists included in search are:

• US Treasury - Office of Foreign Assets Control (OFAC)
  • Specially Designated Nationals (SDN)
    • Includes SDN, SDN Alternative Names, SDN Addresses
  • Sectoral Sanctions Identifications (SSI)
• US Department of Commerce - Bureau of Industry and Security (BIS)
  • Denied Persons List (DPL)
  • Entity List (EL)

All United States and European Union companies are required to comply with various regulations and sanction lists (such as the US Patriot Act requiring compliance with the BIS Denied Persons List). Moov's primary usage for this project is with ACH origination in our paygate project.

Table of Contents

• Project Status
• Usage
  • As an API
    • Docker (Config)
    • Google Cloud (Config)
    • Webhook Notifications
    • Data Persistence
  • As a Go Module
  • As an In-Browser Search Tool
• Useful Resources
• Getting Help
• Supported and Tested Platforms
• Contributing
• Related Projects

Project Status

Moov Watchman is actively used in multiple production environments. Please star the project if you are interested in its progress. If you have layers above Watchman to simplify tasks, perform business operations, or found bugs we would appreciate an issue or pull request. Thanks!

Usage

The Watchman project implements an HTTP server and Go library for searching, parsing, and downloading lists. We also have an example of the webhook service. Below, you can find a detailed list of features offered by Watchman:

• Download OFAC, BIS Denied Persons List (DPL), and various other data sources on startup
  • Admin endpoint to manually refresh OFAC and DPL data
• Index data for searches
• Async searches and notifications (webhooks)
• Manual overrides to mark a Company or Customer as unsafe (blocked) or exception (never blocked).
• Library for OFAC and BIS DPL data to download and parse their custom files

Docker

We publish a public Docker image moov/watchman from Docker Hub or use this repository. No configuration is required to serve on :8084 and metrics at :9094/metrics in Prometheus format. We also have Docker images for OpenShift published as quay.io/moov/watchman. Lastly, we offer a moov/watchman:static Docker image with files from 2019. This image can be useful for faster local testing or consistent results.

Pull & start the Docker image:

docker pull moov/watchman:latest
docker run -p 8084:8084 -p 9094:9094 moov/watchman:latest

Get information about a company using their entity ID:

curl "localhost:8084/ofac/companies/13374"

{
   "id":"13374",
   "sdn":{
      "entityID":"13374",
      "sdnName":"SYRONICS",
      "sdnType":"",
      "program":[
         "NPWMD"
      ],
      "title":"",
      "callSign":"",
      "vesselType":"",
      "tonnage":"",
      "grossRegisteredTonnage":"",
      "vesselFlag":"",
      "vesselOwner":"",
      "remarks":""
   },
   "addresses":[
      {
         "entityID":"13374",
         "addressID":"21360",
         "address":"Kaboon Street, PO Box 5966",
         "cityStateProvincePostalCode":"Damascus",
         "country":"Syria",
         "addressRemarks":""
      }
   ],
   "alts":[
      {
         "entityID":"13374",
         "alternateID":"15056",
         "alternateType":"aka",
         "alternateName":"SYRIAN ARAB CO. FOR ELECTRONIC INDUSTRIES",
         "alternateRemarks":""
      }
   ],
   "comments":null,
   "status":null
}

Google Cloud Run

To get started in a hosted environment you can deploy this project to the Google Cloud Platform.

From your Google Cloud dashboard create a new project and call it:

moov-watchman-demo

Enable the Container Registry API for your project and associate a billing account if needed. Then, open the Cloud Shell terminal and run the following Docker commands, substituting your unique project ID:

docker pull moov/watchman
docker tag moov/watchman gcr.io/<PROJECT-ID>/watchman
docker push gcr.io/<PROJECT-ID>/watchman

Deploy the container to Cloud Run:

gcloud run deploy --image gcr.io/<PROJECT-ID>/watchman --port 8084

Select your target platform to 1, service name to watchman, and region to the one closest to you (enable Google API service if a prompt appears). Upon a successful build you will be given a URL where the API has been deployed:

https://YOUR-WATCHMAN-APP-URL.a.run.app

Now you can ping the server:

curl https://YOUR-WATCHMAN-APP-URL.a.run.app/ping

You should get this response:

PONG

Configuration Settings

Environmental Variable	Description	Default
DATA_REFRESH_INTERVAL	Interval for data redownload and reparse. off disables this refreshing.	12h
INITIAL_DATA_DIRECTORY	Directory filepath with initial files to use instead of downloading. Periodic downloads will replace the initial files.	Empty
WEBHOOK_BATCH_SIZE	How many watches to read from database per batch of async searches.	100
LOG_FORMAT	Format for logging lines to be written as.	Options: json, plain - Default: plain
BASE_PATH	HTTP path to serve API and web UI from.	/
HTTP_BIND_ADDRESS	Address to bind HTTP server on. This overrides the command-line flag -http.addr.	Default: :8084
HTTP_ADMIN_BIND_ADDRESS	Address to bind admin HTTP server on. This overrides the command-line flag -admin.addr.	Default: :9094
HTTPS_CERT_FILE	Filepath containing a certificate (or intermediate chain) to be served by the HTTP server. Requires all traffic be over secure HTTP.	Empty
HTTPS_KEY_FILE	Filepath of a private key matching the leaf certificate from HTTPS_CERT_FILE.	Empty
DATABASE_TYPE	Which database option to use (Options: sqlite, mysql).	Default: sqlite
WEB_ROOT	Directory to serve web UI from.	Default: webui/

List Configurations

Environmental Variable	Description	Default
OFAC_DOWNLOAD_TEMPLATE	HTTP address for downloading raw OFAC files.	https://www.treasury.gov/ofac/downloads/%s
DPL_DOWNLOAD_TEMPLATE	HTTP address for downloading the DPL.	https://www.bis.doc.gov/dpl/%s
CSL_DOWNLOAD_TEMPLATE	HTTP address for downloading the Consolidated Screening List (CSL), which is a collection of US government sanctions lists.	https://api.trade.gov/consolidated_screening_list/%s
KEEP_STOPWORDS	Boolean to keep stopwords in names.	false
DEBUG_NAME_PIPELINE	Boolean to pring debug messages for each name (SDN, SSI) processing step.	false

Storage

Based on DATABASE_TYPE, the following environmental variables will be read to configure connections for a specific database.

MySQL

• MYSQL_ADDRESS: TCP address for connecting to the MySQL server. (example: tcp(hostname:3306))
• MYSQL_DATABASE: Name of database to connect into.
• MYSQL_PASSWORD: Password of user account for authentication.
• MYSQL_USER: Username used for authentication.

Refer to the MySQL driver documentation for connection parameters.

• MYSQL_TIMEOUT: Timeout parameter specified on (DSN) data source name. (Default: 30s)

SQLite

• SQLITE_DB_PATH: Local filepath location for the paygate SQLite database. (Default: watchman.db)

Refer to the SQLite driver documentation for connection parameters.

Webhook Notifications

When Watchman sends a webhook to your application, the body will contain a JSON representation of the Company or Customer model as the body to a POST request. You can see an example in Go.

An Authorization header will also be sent with the authToken provided when setting up the watch. Clients should verify this token to ensure authenticated communicated.

Webhook notifications are ran after the OFAC data is successfully refreshed, which is determined by the DATA_REFRESH_INTERVAL environmental variable.

Watching a specific Customer or Company by ID

Moov Watchman supports sending a webhook periodically when a specific Company or Customer is to be watched. This is designed to update another system about an OFAC entry's sanction status.

Watching a customer or company name

Moov Watchman supports sending a webhook periodically with a free-form name of a Company or Customer. This allows external applications to be notified when an entity matching that name is added to the OFAC list. The match percentage will be included in the JSON payload.

Prometheus Metrics

• http_response_duration_seconds: A histogram of HTTP response timings.
• last_data_refresh_success: Unix timestamp of when data was last refreshed successfully.
• last_data_refresh_count: Count of records for a given sanction or entity list.
• match_percentages A histogram which holds the match percentages with a label (type) of searches.
  • type: Can be address, q, remarksID, name, altName
• mysql_connections: Number of MySQL connections and their statuses.
• sqlite_connections: Number of SQLite connections and their statuses.

Data Persistence

By design, Watchman does not persist (save) any data about the search queries or actions created. The only storage occurs in memory of the process and upon restart Watchman will have no files or data saved. Also, no in-memory encryption of the data is performed.

Go Library

This project uses Go Modules and uses Go v1.14 or higher. See Golang's install instructions for help setting up Go. You can download the source code and we offer tagged and released versions as well. We highly recommend you use a tagged release for production.

$ git@github.com:moov-io/watchman.git

# Pull down into the Go Module cache
$ go get -u github.com/moov-io/watchman

$ go doc github.com/moov-io/watchman/client Search

In-Browser Watchman Search

Using our in-browser utility, you can instantly perform advanced Watchman searches. Simply fill search fields and generate a detailed report that includes match percentage, alternative names, effective/expiration dates, IDs, addresses, and other useful information. This tool is particularly useful for completing quick searches with the aid of a intutive interface.

Reporting Blocks to OFAC

OFAC requires annual reports of blocked entities and offers guidance for this report. Section 31 C.F.R. § 501.603(b)(2) requires this annual report.

Useful Resources

• OFAC Sanctions Search Page
• Subscribe for OFAC email updates
• When should I call the OFAC Hotline?
• BIS Denied Persons List with Denied US Export Privileges (DPL)
• BIS Entity List
• Sectoral Sanctions Identifications (SSI)

Getting Help

We maintain a runbook for common issues and configuration options. Also, if you've encountered a security issue please contact us at security@moov.io.

channel	info
Project Documentation	Our project documentation available online.
Twitter @moov_io	You can follow Moov.IO's Twitter feed to get updates on our project(s). You can also tweet us questions or just share blogs or stories.
GitHub Issue	If you are able to reproduce a problem please open a GitHub Issue under the specific project that caused the error.
moov-io slack	Join our slack channel to have an interactive discussion about the development of the project.

Supported and Tested Platforms

• 64-bit Linux (Ubuntu, Debian), macOS, and Windows

Note: 32-bit platforms have known issues and are not supported.

Contributing

Yes please! Please review our Contributing guide and Code of Conduct to get started! Checkout our issues for first time contributors for something to help out with.

This project uses Go Modules and uses Go 1.14 or higher. See Golang's install instructions for help setting up Go. You can download the source code and we offer tagged and released versions as well. We highly recommend you use a tagged release for production.

Releasing

To make a release of ach simply open a pull request with CHANGELOG.md and version.go updated with the next version number and details. You'll also need to push the tag (i.e. git push origin v1.0.0) to origin in order for CI to make the release.

Testing

We maintain a comprehensive suite of unit tests and recommend table-driven testing when a particular function warrants several very similar test cases. To run all test files in the current directory, use go test. Current overall coverage can be found on Codecov.

Related Projects

As part of Moov's initiative to offer open source fintech infrastructure, we have a large collection of active projects you may find useful:

• Moov Fed implements utility services for searching the United States Federal Reserve System such as ABA routing numbers, financial institution name lookup, and FedACH and Fedwire routing information.

• Moov Image Cash Letter implements Image Cash Letter (ICL) files used for Check21, X.9 or check truncation files for exchange and remote deposit in the U.S.

• Moov Wire implements an interface to write files for the Fedwire Funds Service, a real-time gross settlement funds transfer system operated by the United States Federal Reserve Banks.

• Moov ACH provides ACH file generation and parsing, supporting all Standard Entry Codes for the primary method of money movement throughout the United States.

• Moov Metro 2 provides a way to easily read, create, and validate Metro 2 format, which is used for consumer credit history reporting by the United States credit bureaus.

License

Apache License 2.0 - See LICENSE for details.

Documentation

Index

• Constants

Constants

const Version = "v0.16.2"