log2oms
A super tiny agent (binary 7MB, container 12MB) that pushs app logs to Azure Log Analytics (OMS)
Why we need this
I have been exploring options to push container logs to a remote storage like Log Analytics. A few available options are:
- Use OMS container (https://hub.docker.com/r/microsoft/oms/) to push logs. However, 1) this solution requires running the OMS container as privileged, so it won't work on services like ACI. 2) the size of the image (307MB)...isn't very nice.
- Install OMS agent into my app container. I tried, and realized 1) it comes with lots of dependencies, python etc. 2) it doesn't support alpine. 3) Size of just the installer (omsagent-1.4.4-210.universal.x64.sh 110MB), isn't very container fridenly. Since I only want to upload logs, most of the dependencies are really unnecessary.
So I implemented this tiny agent that uses Log Analytics data collector API (https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api), and make it container friendly.
How to use it
Sidecar
The docker image is published at https://hub.docker.com/r/yangl/log2oms/
The best way to use log2oms is by adopting the "sidecar" pattern. Having log2oms container (image yangl/log2oms
) run as a "sidecar" of your app container, and use a shared volume to read the app logs and uplaod to log2oms.
Take a nginx web server as example, you simply run yangl/log2oms
as another container and shares the nginx /var/log/nginx volume. log2oms will tail the nginx logs and upload to Log Analytics automatically.
+-----------------------------+
| | |
| NGINX | log2oms |
| | |
+-----------------------------+
| (shared volume) |
| /var/log/nginx |
+-----------------------------+
The log2oms container requires only 4 environment variables to run:
LOG2OMS_WORKSPACE_ID
This is the workspace ID of Log Analytics.
LOG2OMS_WORKSPACE_SECRET
This is the secret of your workspace, you can find it from "Advanced Settings" in Azure portal.
LOG2OMS_LOG_FILE
This is the log file to tail and upload. Right now only support 1 file, in nginx case, this will be access.log
LOG2OMS_LOG_TYPE
This is the table you want logs upload to. Note that LogAnalytics will add a postfix _CL
to this name. so if we have nginx
here, in LogAnalytics the table will be nginx_CL
.
And that's it. No changes needed from app container.
More flags:
LOG2OMS_METADATA_*
This is an environment variable prefix for log metadata. The metadata will be sent to Log Analytics for every log message. This is useful if you have multiple replicas sending logs and want to differentiate them. For example, set LOG2OMS_METADATA_Location=WestUS
and LOG2OMS_METADATA_Role=Frontend
, logs in Analytics will have 2 more columns Location
and Role
.
Sample for Kubernetes
samples/kubernetes/deploy.yaml
is a sample yaml how to deploy an nginx server with log2oms as a sidecar.
To try the sample:
- Run
kubectl create -f samples/kubernetes/deploy.yaml
to create the deployment.
- curl the pod IP on port 80 to generate a few lines of nginx logs.
kubectl logs {pod-name} log2oms
should show some agent logs like following
[LOG2OMS][2018-03-17T04:21:36Z] Start tail logs from: /logs/access.log
[2018-03-17T04:22:56Z] 10.244.0.1 - - [17/Mar/2018:04:22:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
[LOG2OMS][2018-03-17T04:23:01Z] Posted 1 messages.
- Wait a few minutes to let LogAnalytics process, then you can query
nginx_access_CL | take 100
in LogAnalytics to see the nginx access logs.
Sample for Azure Container Instances
- Click the deploy-to-azure button in ACI sample page to create an nginx container in ACI with the log2oms sidecar.
- After deployment succeed (should be a few seconds), access the container public IP address to generate a few lines of logs. Use Azure portal or Cloud Shell command
az container show -g {resource-group} -n {container-group-name}
to find out the public IP address.
- Wait a few minutes to let LogAnalytics process, then you can query
nginx_access_CL | take 100
in LogAnalytics to see the nginx access logs.
Future improvements
- Send a heartbeat signal to log analytics so you know when it is working / stop working.
- Handle SIGTERM to flush out logs before termination.
- Exit on a termination signal file. This will be useful for task containers so the sidecar can stop automatically.