sebase

module
Version: v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 9, 2021 License: MIT

README

Schibsted Search Engineering Base Libraries

This repository contains a set of base utilities used throughout the years by the Search Engineering team and others at Schibsted.

Documentation is currently somewhat lacking, we hope to improve it reasonably soon, depending on some priorities.

Interfaces are provided for C and Go. Some code might be lacking Go interfaces.

Navigation

The repository is split in several submodules each residing in a different directory.

Vendor

This directory contains some third party sources we deemed important enough to vendor. See the separate README for more information. This directory has no outside dependencies. It can sometimes be replaced by using system libraries.

Util

Contains basic utilities such as AVL trees, socket utilties and encryption wrapper functions. This submodule depends on the vendored libraries and will also link with Libcurl, OpenSSL and pcre.

Vtree

A variable/config tree used in the core submodule. Can be initialized from "bconf" files, json or manually in the code. This submodule depends on util as well as libyajl version 2 or higher.

Plog

This submodule contains a log aggregator for structured logging and program state. It has client libaries and a daemon co-process expected to run on each instance. Log lines are stored in the daemon where it's structured into a log object that is logged once fully completed or connection interrupted.

It depends on the util submodule and uses protobuf to communicate.

Core

Contains code for Service Discovery registration and lookup, client side load balancing, program startup/shutdown and a few more utilities depending on vtree.

It depends on all the other submodules. It can also optionally link with libicu.

Building and testing

Dependencies

In total, the external library dependencies are

  • curl
  • icu (optional)
  • openssl
  • pcre
  • protobuf-c
  • yajl

Additionally a few tools are needed to build:

Running tests additionally requires

  • GNU make
  • python
Building

Currently this repository can only built with the sebuild tool. In the future we hope to add more traditional packaging as well.

Tests can after compiling be run simply by running the regress-runner binary which is compiled as part of the tree. It scans the directory it's invoked from for regress-runner.mk files which contains a print-tests make target indicating how to run tests.

To use this code in other projects with sebuild, you need to include this repository as a subdirectory of your code. This can be done with e.g. git subtree or git submodule. Then you add the directory containing this code as a COMPONENT in your top level Builddesc.

Building Go packages

The Go packages should work with go get etc. This repo is go module compatible.

You can optionally have to Go packages call the matching C libraries. This enables you to shares e.g. an fd pool between Go and C code.

To do that you will have add -tags sebuild_cgo when using go build. The easiest way to do this in a consistent way is probably to use sebuild and a configvars file to set gobuild_flags="-tags sebuild_cgo" there.

Maintainance Notice

Please note that most of the code in util, vtree and core are not currently activly maintained, they're provided with the hope that someone will find it useful. We do look at Pull Requests and Issues if reported.

If you wish to change the code or contribute, see CONTRIBUTING.md.

LICENSE

Copyright 2018 Schibsted

Licensed under the MIT License, you may not use this code except in compliance with the License. The full license is included in the file LICENCE.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Directories

Path Synopsis
core
bin/acl-proxy
A privilege separation HTTP proxy The privilege it's guarding is the permission to talk to the upstream server.
A privilege separation HTTP proxy The privilege it's guarding is the permission to talk to the upstream server.
bin/cert-sleep
Will sleep until the expiration any of the PEM encoded certificates read from all files given as arguments.
Will sleep until the expiration any of the PEM encoded certificates read from all files given as arguments.
pkg/fd_pool
Go fd pool implementation, providing client side load balancing.
Go fd pool implementation, providing client side load balancing.
pkg/sd/etcd
Etcd driver for the SD registry package.
Etcd driver for the SD registry package.
plog
pkg/plog
Package plog is used for logging and sending logs to a log server, where they'll be aggregated into log objects and kept safe even if the client program exists or crashes.
Package plog is used for logging and sending logs to a log server, where they'll be aggregated into log objects and kept safe even if the client program exists or crashes.
util
pkg/sbalance
This package allows you to iterate over a set of service nodes, either in sequence or randomly.
This package allows you to iterate over a set of service nodes, either in sequence or randomly.
pkg/slog
Package slog is used to log from the other sebase packages.
Package slog is used to log from the other sebase packages.
vtree
pkg/vtree
Package vtree is an interface for C compatible vtrees used in sebase.
Package vtree is an interface for C compatible vtrees used in sebase.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL