k8s-pod-scheduler

README

Kubernetes(K8S) Pod Scheduler

A simple and lightweight go tool that can run on top of K8s to schedule and run long-running tasks with a simple gRPC interface.

It's something smaller and simpler than Apache Airflow for small or simple tasks. Instead of using a workflow management tool with a lot of overhead, this tool can be more fit for simple and small tasks that don't really require a lot of configurations to execute.

Use cases of this package

When a user wants to read jobs from a remote-store and schedule them accordingly.

User guide

Assuming you have a backend or a service that does some processes and output some jobs or recurring tasks that need to be scheduled. You can connect this service with k8s-pod-scheduler easily.

The recommended way is to use the proto interface provided at idl/job.proto and call it via gRPC. Feel free to add REST or CLI support if your use case needs it.

If you want to test the service without implementing a client, you can use any gRPC client like BloomRPC or gRPCurl.

As you can see in The service has only one method which is Add to add a new job to the scheduler. This job will accept these parameters:

• Name string
• Cron expression string. It accepts cron expressions with seconds. Also, "@daily" and "@every "

The service will add your job to the scheduler, run it as scheduled, and kill the pod when the job is done. Note that when the job fails, the scheduler will kill it as well.

Supported Fields:

Field	Type	Required
Name	String	✓
Cron	String	✓
Spec.image	String	✓
Spec.Args	List
Spec.Commands	List

Start the program

You can run it from your machine directly or use this image as an example from dockerhub ahmedmagdi/k8s-pod-scheduler.

docker pull ahmedmagdi/k8s-pod-scheduler:0.0.1
kubectl apply -f ./example/k8s-deployment.yaml

This will start a pod listening on port 30007. You can call it then using BloomRPC or similar tools.

Triggering a new job:

grpcurl -plaintext -import-path ./idl -proto job.proto -d '{"job": {"name":"my-first-job", "cron":"@every 0h0m30s", "spec":{"image":"ahmedmagdi/go-sample-task:1.0.0"}}}' \
    127.0.0.1:30007 job.JobService/Add

After that if you checked the logs of the running pod or watched the changes in k8s (kubectl get pods -n podscheduler -w) you will notice the following:

NAME                            READY   STATUS    RESTARTS   AGE
podscheduler-58dcffbdd7-zz2hg   1/1     Running   0          53m

When triggering a build

NAME                            READY   STATUS    RESTARTS   AGE
podscheduler-58dcffbdd7-zz2hg   1/1     Running   0          53m
my-first-jobw7qld               0/1     Pending   0          0s
my-first-jobw7qld               0/1     Pending   0          0s
my-first-jobw7qld               0/1     ContainerCreating   0          0s
my-first-jobw7qld               1/1     Running             0          1s
my-first-jobw7qld               0/1     Completed           0          6s
my-first-jobw7qld               0/1     Terminating         0          6s
my-first-jobw7qld               0/1     Terminating         0          6s

Local Development:

make setup

Will setup all the needed tools and dependencies.

TODO:

• Store registered jobs state in a persistent store instead of storing them in memory.
• Provide API to update, and remove tasks.