natun

README

Serverless feature engineering - from notebook to production

Build production-ready ML features, directly from your notebook, without being a software engineer.
Focus on data science, Natun will take care of the engineering overhead.

Explore the docs »

Getting started in 5 minutes · Report a Bug · Request a Feature

🧐 What is Natun?

With Natun, Data Scientists can write ML Features that can work in production directly in your notebook and deploy them to production on their own (yes! without going through the engineering team). It's like writing Lambda for Feature Engineering.

After developing your ML features, deploy them to your Kubernetes, and the Natun extension will take care of the production concerns (such as scale, high availability, authentication, caching, monitoring, etc.)

😍 Why people love Natun?

• Data Scientists love Natun because it's easy to use and make a use of their work. No more excuses from the engineering team!
• DevOps teams love Natun because it's built on top of Kubernetes, leveraging their standard stack, and it's ridiculously easy to deploy and manage (because of Kubernetes, duh?).

And what about the engineers? Well, it's finally taking away this "burden" from them.

⭐️ Features

• Same code for both training and production
  You can run the same Natun compatible features in training and production and prevent the training serving skew.
• Real-Time / On-demand feature calculation
  Natun is optimizing features to be calculated at the time of the request.
• Caching and storing
  Natun is utilizing an integrated Feature-Store to cache the calculation results and take snapshots of the data to cold storage for historical purposes (such as re-training).
• Pluggable
  Natun is highly-pluggable. Technically, you can even bring your own feature store implementation.
• Kubernetes Native and Standards
  Natun is implementing by-design functionalities of Kubernetes such as leader-election, scaling, health, auto-recovery, monitoring and logging.

(back to top)

💡 How does it work?

Natun is extending Kubernetes to serve Natun-compatible features in a production-grade manner. It's implementing the "Kubernetes Way," which advocates for writing the desired state and allows you to focus on writing the business logic.

Natun takes care of the engineering concerns of implementing the "desired state" by managing and controlling Kubernetes-native resources such as deployments to connect your production data-sources and to run your business logic at scale.

Behind the scenes, Natun is attaching the proper building block to implement the "production behavior" of your use-case (i.e., streaming, REST API, gRPC, expression, etc.).

You can read more about Natun's architecture in the docs.

⚡️ Quick start

The LabSDK is the quickest and most popular way to develop Natun compatible features.

The LabSDK allows you to write Natun-compatible features using Python and "convert" them to Kubernetes resources. This way, in most of the use-cases, you can iterate and play with your data.

Production Installation

Natun installation is not required for training purposes. You only need to install Natun when deploying to production (or staging).

Because Natun

Prerequisites

1. Kubernetes cluster

   (You can use Kind to install Natun locally)

2. kubectl installed and configured to your cluster.

3. Redis server

   ⚠️ Tip: You can also use a managed Redis solution such as AWS ElasticCache or GCP Memorystore, or via the K8s Redis Operator

Installation

The easiest way to install Natun is to use the OperatorHub Installation method.

(back to top)

🌍 "Hello World" feature

We know, you played with the quick starter already.. But let's paste here some "hello world" code for the lazy bastard out there 👹

For more examples, please refer to the Documentation

Option 1: Using LabSDK

@natun.register(str, freshness="1m", staleness="15m")
def hello_world(**req : NatunRequest):
  return "hello "+req["entity_id"]+"!"

Option 2: Natun compatible Feature (the Kubernetes native way)

The code below, is actually the results of Option 1 above 🤪 But in case you've decided to not use the LabSDK, you can write a Feature Definition like this:

apiVersion: k8s.natun.ai/v1alpha1
    kind: Feature
    metadata:
      name: hello_world
    spec:
      primitive: string
      freshness: 1m
      staleness: 15m
      builder:
        kind: expression
    pyexp: |
      def hello_world(**req):
          return 'hello ' + req['entity_id']) + '!'

(back to top)

🏔 Roadmap

• Redis state provider
• Snowflake historical storage plugin
• S3 historical storage plugins
  • S3 storing
  • S3 fetching data - Spark
• KServe Integration
• Seldon Integration
• Scheduler builder

See the open issues for a full list of proposed features (and known issues).

(back to top)

👷‍ Contributing

Contributions make the open-source community a fantastic place to learn, inspire, and create. Any contributions you make are greatly appreciated (not only code! but also documenting, blogging, or giving us feedback) 😍.

Please fork the repo and create a pull request if you have a suggestion. You can also simply open an issue and choose "Feature Request" to give us some feedback.

Don't forget to give the project a star! ⭐️

For more information about contributing code to the project, read the CONTRIBUTING.md file.

(back to top)

📃 License

Distributed under the Apache2 License. Read the LICENSE file for more information.

(back to top)

👫 Joining the community

(back to top)