README
¶
What is lakeFS
lakeFS is an open-source tool that transforms your object storage into a Git-like repository. It enables you to manage your data lake the way you manage your code.
With lakeFS you can build repeatable, atomic, and versioned data lake operations - from complex ETL jobs to data science and analytics.
lakeFS supports AWS S3, Azure Blob Storage, and Google Cloud Storage as its underlying storage service. It is API compatible with S3 and works seamlessly with all modern data frameworks such as Spark, Hive, AWS Athena, Presto, etc.
For more information, see the official lakeFS documentation.
Capabilities
In Development
- Experiment - try new tools, upgrade versions, and evaluate code changes in isolation. You can get an isolated snapshot to run and compare experiments by creating a new branch of the data, while others are not exposed.
- Debug - checkout specific commits in a repository’s commit history to materialize consistent, historical versions of your data.
- Collaborate - leverage isolated branches managed by metadata (not copies of files) to work in parallel.
During Deployment
- Version Control - deploy data safely with CI/CD workflows borrowed from software engineering best practices. Ingest new data onto an isolated branch, perform data validations, then add to production through a merge operation.
- Test - define pre-merge and pre-commit hooks to run tests that enforce schema and validate properties of the data to catch issues before they reach production.
In Production
- Roll Back - recover from errors by instantly reverting data to a former, consistent snapshot of the data lake. Choose any commit in a repository’s commit history to revert in one atomic action.
- Troubleshoot - investigate production errors by starting with a snapshot of the inputs to the failed process.
- Cross-collection Consistency - provide consumers multiple synchronized collections of data in one atomic, revertable action.
Getting Started
Using Docker
Use this section to learn about lakeFS. For a production-suitable deployment, see the docs.
-
Ensure you have Docker installed on your computer.
-
Run the following command:
docker run --pull always --name lakefs -p 8000:8000 treeverse/lakefs run --local-settings -
Open http://127.0.0.1:8000/ in your web browser to set up an initial admin user. You will use this user to log in and send API requests.
Other quickstart methods
You can try lakeFS:
- Without installing, using the lakeFS Playground.
- On Kubernetes.
- By running the binary directly.
Setting up a repository
Once lakeFS is installed, you are ready to create your first repository!
Community
Stay up to date and get lakeFS support via:
- Share your lakeFS experience and get support on our Slack.
- Follow us and join the conversation on Twitter.
- Learn from video tutorials on our YouTube channel.
- Feel free to contact us about anything else.
More information
- Read the documentation.
- See the contributing guide.
- Take a look at our roadmap to peek into the future of lakeFS.
Licensing
lakeFS is completely free and open-source and licensed under the Apache 2.0 License.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
pkg
|
|
|
api/helpers
Package helpers provides useful wrappers for clients using the lakeFS OpenAPI.
|
Package helpers provides useful wrappers for clients using the lakeFS OpenAPI. |
|
gateway/sig
Package sig This file implements helper functions to validate Streaming AWS Signature Version '4' authorization header.
|
Package sig This file implements helper functions to validate Streaming AWS Signature Version '4' authorization header. |
|
kv/mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |
|
mock
Package mock is a generated GoMock package.
|
Package mock is a generated GoMock package. |