go-judge

README

go-judge

中文文档 | Documentation

Fast, Simple, Secure

Quick Start

Install & Run

Download compiled executable go-judge for your platform from Release and run.

Or, by docker

docker run -it --rm --privileged --shm-size=256m -p 5050:5050 --name=go-judge criyle/go-judge

REST API

A REST service to run program in restricted environment (Listening on localhost:5050 by default).

• POST /run execute program in the restricted environment
• GET /file list all cached file id to original name map
  • POST /file prepare a file in the go judge (in memory), returns fileId (can be referenced in /run parameter)
  • GET /file/:fileId downloads file from go judge (in memory), returns file content
  • DELETE /file/:fileId delete file specified by fileId
• /ws WebSocket version for /run
• /stream WebSocket for stream run. Supports streaming interface
• GET /version gets build git version (e.g. v1.9.0) together with runtime information (go version, os, platform)
  • GET /config gets some configuration (e.g. fileStorePath, runnerConfig) together with some supported features

REST API Interface

API Interface Structure Definition

Example Request & Response

Example Request & Response

Documentation

Prerequisite

• Linux Kernel Version >= 3.10
• Cgroup file system mounted at /sys/fs/cgroup. Usually done by systemd

Architecture

+----------------------------------------------------------------------------------+
| Transport Layer (HTTP / WebSocket / FFI / ...)                                   |
+----------------------------------------------------------------------------------+
| Sandbox Worker (Environment Pool w/ Environment Builder )                        |
+-----------------------------------------------------------+----------------------+
| EnvExec                                                   | File Store           |
+--------------------+----------------+---------------------+---------------+------+
| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | Shared Memory | Disk |
+--------------------+----------------+---------------------+---------------+------+

Configurations

Server:

• The default binding address for the go judge is localhost:5050. Can be specified with -http-addr flag.
• By default gRPC endpoint is disabled, to enable gRPC endpoint, add -enable-grpc flag.
  • The default binding address for the gRPC go judge is localhost:5051. Can be specified with -grpc-addr flag.
• The default log level is info, use -silent to disable logs or use -release to enable release logger (auto turn on if in docker).
• -auth-token to add token-based authentication to REST / gRPC
• By default, the GO debug endpoints (localhost:5052/debug) are disabled, to enable, specifies -enable-debug, and it also enables debug log
• By default, the prometheus metrics endpoints (localhost:5052/metrics) are disabled, to enable, specifies -enable-metrics
• Monitoring HTTP endpoint is enabled if metrics / debug is enabled, the default addr is localhost:5052 and can be specified by -monitor-addr

Sandbox:

• The default concurrency equal to number of CPU, Can be specified with -parallelism flag.
• -mount-conf specifies detailed mount configuration, please refer File System Mount as a reference (Linux only)
• -file-timeout specifies maximum TTL for file created in file store （e.g. 30m)
• The default file store is in memory(/dev/shm/), local cache can be specified with -dir flag.
• -output-limit specifies size limit of POSIX rlimit of output (default 256MiB)
• -copy-out-limit specifies the default file copy out max (default 64MiB)

You can find more available configuration here.

Run Terminal in the Container

Download go-judge-shell from Release and run. It will connect local go-judge, and open an interactive shell in the container for debugging purpose.

Return Status

• Accepted: Program exited with status code 0 within time & memory limits
• Memory Limit Exceeded: Program uses more memory than memory limits
• Time Limit Exceeded: (exitStatus usually have value 9 as killed by SIGKILL after timeout)
  • Program uses more CPU time than cpuLimit
  • Or, program uses more clock time than clockLimit
• Output Limit Exceeded:
  • Program output more than pipeCollector limits
  • Or, program output more than output-limit
• File Error:
  • CopyIn file is not existed
  • Or, CopyIn file too large for container file system
  • Or, CopyOut file is not existed after program exited
• Non Zero Exit Status: Program exited with non 0 status code within time & memory limits
• Signalled: Program exited with signal (e.g. SIGSEGV)
• Dangerous Syscall: Program killed by seccomp filter (not enabled by default)
• Internal Error:
  • Program is not exist
  • Or, container create not successful (e.g. not privileged docker)
  • Or, other errors

Container Root Filesystem

For linux platform, the default mounts points are bind mounting host's /lib, /lib64, /usr, /bin, /etc/ld.so.cache, /etc/alternatives, /etc/fpc.cfg, /dev/null, /dev/urandom, /dev/random, /dev/zero, /dev/full and mounts tmpfs at /w, /tmp and creates /proc.

To customize mount points, please look at example mount.yaml file.

If mount.yaml is not specified, the size of tmpfs for /w and /tmp is configured through -tmp-fs-param with default value size=128m,nr_inodes=4k

If a file named /.env exists in the container rootfs, the container will load the file as environment variable line by line.

If a bind mount is specifying a target within the previous mounted one, please ensure the target exists in the previous mount point.

Metrics Monitoring Endpoint

Prometheus Metrics Monitoring Endpoint

Notice

[!WARNING]
Window and macOS support are experimental and should not be used in production environments

cgroup usage

For cgroup v1, the go-judge need root privilege to create cgroup. Either creates sub-directory /sys/fs/cgroup/cpuacct/gojudge, /sys/fs/cgroup/memory/gojudge, /sys/fs/cgroup/pids/gojudge and make execution user readable or use sudo to run it.

For cgroup v2, systemd dbus will be used to create a transient scope for cgroup integration.

If no permission to create cgroup, the cgroup related limit will not be effective.

cgroup v2 support

The cgroup v2 is supported by go-judge now when running as root since more Linux distribution are enabling cgroup v2 by default (e.g. Ubuntu 21.10+, Fedora 31+). However, for kernel < 5.19, due to missing memory.max_usage_in_bytes in memory controller, the memory usage is now accounted by maxrss returned by wait4 syscall. Thus, the memory usage appears higher than those who uses cgroup v1. For kernel >= 5.19, memory.peak is being used.

When running in containers, the go-judge will migrate all processed into /api hierarchy to enable nesting support.

When running in Linux distributions powered by systemd, the go-judge will contact systemd via dbus to create a transient scope as cgroup root.

When running with kernel >= 5.7, the go-judge will try faster clone3(CLONE_INTO_CGROUP) and vfork method.

Memory Usage

The controller usually consumes 20M memory and each container usually consumes 20M + size of tmpfs 2 * 128M. For each request, it consumes as much as user program limit + extra limit (16k) + total copy out max. Notice that the cached file stores in the shared memory (/dev/shm) of the host, so please ensure enough size allocated.

For example, when concurrency = 4, the container itself can consume as much as 60 + (20+32) * 4M = 268M + 4 * total copy out + total max memory of requests.

Due to limitation of GO runtime, the memory will not return to OS automatically, which could lead to OOM killer. A background worker was introduced to checks heap usage and invokes GC when necessary.

• -force-gc-target default 20m, the minimal size to trigger GC
• -force-gc-interval default 5s, the interval to check memory usage