edgefs

module
Version: v1.3.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 3, 2020 License: Apache-2.0

README

EdgeFS - decentralized data fabric layer for Edge/IoT, Edge/Fog and Cloud Computing

EdgeFS is high-performance, low-latency, small memory footprint, decentralized data fabric system released under Apache License v2.0 developed in C/Go.

EdgeFS is built around decentralized immutable metadata consistency to sustain network partitioning up to many days and yet provide consistent global data fabric namespace via unique versions reconciliation technique.

EdgeFS advanced globally immutable versioning with file-level granularity unlimited snapshots, global (LAN/WAN) data deduplication and geo-transparent data access enables variety of decentralized Edge/Fog/Cloud Computing use cases.

It integrates with Kubernetes and Docker Compose. Enables Edge/Fog/Cloud applications to use Amazon S3 compatible API with AI/ML S3X enhancements, built-in NoSQL Database, scale-out File via NFS, scale-out block via iSCSI/NBD.

edgefs-multicloud.png

EdgeFS is capable of spanning unlimited number of geographically distributed sites (Geo-site), connected with each other as one global name space data fabric running on top of Kubernetes platform, providing persistent, fault-tolerant and high-performance fully compatible S3 Object API and CSI volumes for stateful Kubernetes Applications.

At each Edge location, EdgeFS segment nodes deployed as containers (Kubernetes StatefulSet or Docker Compose) on physical or virtual nodes, pooling available storage capacity and presenting it via compatible S3/NFS/iSCSI/etc storage emulated protocols for cloud-native applications running on the same or dedicated servers.

EdgeFS can run on top of any underlying storage architecture - high-performance NVMe Fabric, local raw devices, existing filesytems or high-performance key-value databases.

How it works, in a Nutshell?

Essentially it is high-performance versioning system that is capable of achieving 1,000,000+ versions per second rates, where each version is universally identifieable and location independent.

If you familiar with "git", where all modifications are fully versioned and globally immutable, it is highly likely you already know how it works at its core. Think of it as a world-scale copy-on-write technique. Now, if we can make a parallel for you to understand it better - what EdgeFS does, it expands "git" paradigm to object storage and making Kubernetes Persistent Volumes accessible via emulated storage standard protocols e.g. File, Object, NoSQL databases and even block devices, in a high-performance and low-latency ways. With fully versioned modifications, fully immutable metadata and data, users data can be transparently replicated, distributed and dynamically pre-fetched across many Geo-sites.

Developer Guides, Services and APIs

Build it (Production Image)

Make sure Docker package is installed on your build server with version >= 17.05 with support for staged builds. Execute the following command:

git clone git@github.com:Nexenta/edgefs.git
cd edgefs
docker build -t edgefs .

Quick Starts

Deployments:

Configurations:

Join our growing community!

Community Meeting

A regular community meeting takes place monthly on every first Wednesday at 9:00 AM PT (Pacific Time). Convert to your local timezone.

Any changes to the meeting schedule will be added to the agenda doc and posted to Slack #users and the edgefs-users mailing list.

Anyone who wants to discuss the direction of the project, design and implementation reviews, or general questions with the broader community is welcome and encouraged to join.

Directories

Path Synopsis
src
grpc-efsproxy
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/cluster/clusterImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/efsutil
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/object/objectImpl
* Copyright (c) 2015-2019 Nexenta Systems, Inc.
* Copyright (c) 2015-2019 Nexenta Systems, Inc.
grpc-efsproxy/service/serviceImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/snapshot/snapshotImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/system/systemImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/tenant/bucketImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/tenant/tenantImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-efsproxy/tenant/userImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-iscsi
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-iscsi/lun/lunImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-iscsi/status/statusImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-nfs
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-nfs/export
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-s3
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-s3/status/statusImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-s3x
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-s3x/status/statusImpl
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
grpc-smb
* Copyright (c) 2020 Nexenta Systems, Inc.
* Copyright (c) 2020 Nexenta Systems, Inc.
grpc-smb/export
* Copyright (c) 2020 Nexenta Systems, Inc.
* Copyright (c) 2020 Nexenta Systems, Inc.
grpc-sql
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
* Copyright (c) 2015-2018 Nexenta Systems, Inc.
efscli Module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL