Schibsted Search Engineering Base Libraries
This repository contains a set of base utilities used throughout the years by the Search Engineering team and others at Schibsted.
Documentation is currently somewhat lacking, we hope to improve it reasonably soon, depending on some priorities.
Interfaces are provided for C and Go. Some code might be lacking Go interfaces.
The repository is split in several submodules each residing in a different directory.
This directory contains some third party sources we deemed important enough to vendor. See the separate README for more information. This directory has no outside dependencies. It can sometimes be replaced by using system libraries.
Contains basic utilities such as AVL trees, socket utilties and encryption wrapper functions. This submodule depends on the vendored libraries and will also link with Libcurl, OpenSSL and pcre.
A variable/config tree used in the core submodule. Can be initialized from "bconf" files, json or manually in the code. This submodule depends on util as well as libyajl version 2 or higher.
This submodule contains a log aggregator for structured logging and program state. It has client libaries and a daemon co-process expected to run on each instance. Log lines are stored in the daemon where it's structured into a log object that is logged once fully completed or connection interrupted.
It depends on the util submodule and uses protobuf to communicate.
Contains code for Service Discovery registration and lookup, client side load balancing, program startup/shutdown and a few more utilities depending on vtree.
It depends on all the other submodules. It can also optionally link with libicu.
Building and testing
In total, the external library dependencies are
- icu (optional)
Additionally a few tools are needed to build:
- Ninja (often called ninja-build or similar)
- Sebuild https://github.com/schibsted/sebuild
- Gcc or clang, including C++ support.
Running tests additionally requires
- GNU make
Currently this repository can only built with the sebuild tool. In the future we hope to add more traditional packaging as well.
Tests can after compiling be run simply by running the
which is compiled as part of the tree. It scans the directory it's invoked from
for regress-runner.mk files which contains a print-tests make target indicating
how to run tests.
To use this code in other projects with sebuild, you need to include this
repository as a subdirectory of your code. This can be done with e.g.
git subtree or
git submodule. Then you add the directory containing this code as
COMPONENT in your top level Builddesc.
Building Go packages
The Go packages should work with go get etc. This repo is go module compatible.
You can optionally have to Go packages call the matching C libraries. This enables you to shares e.g. an fd pool between Go and C code.
To do that you will have add
-tags sebuild_cgo when using go build. The easiest
way to do this in a consistent way is probably to use sebuild and a configvars
file to set
gobuild_flags="-tags sebuild_cgo" there.
Please note that most of the code in util, vtree and core are not currently activly maintained, they're provided with the hope that someone will find it useful. We do look at Pull Requests and Issues if reported.
If you wish to change the code or contribute, see CONTRIBUTING.md.
Copyright 2018 Schibsted
Licensed under the MIT License, you may not use this code except in compliance with the License. The full license is included in the file LICENCE.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
A privilege separation HTTP proxy The privilege it's guarding is the permission to talk to the upstream server.
|A privilege separation HTTP proxy The privilege it's guarding is the permission to talk to the upstream server.|
Will sleep until the expiration any of the PEM encoded certificates read from all files given as arguments.
|Will sleep until the expiration any of the PEM encoded certificates read from all files given as arguments.|
Go fd pool implementation, providing client side load balancing.
|Go fd pool implementation, providing client side load balancing.|
Etcd driver for the SD registry package.
|Etcd driver for the SD registry package.|
Package plog is used for logging and sending logs to a log server, where they'll be aggregated into log objects and kept safe even if the client program exists or crashes.
|Package plog is used for logging and sending logs to a log server, where they'll be aggregated into log objects and kept safe even if the client program exists or crashes.|
This package allows you to iterate over a set of service nodes, either in sequence or randomly.
|This package allows you to iterate over a set of service nodes, either in sequence or randomly.|
Package slog is used to log from the other sebase packages.
|Package slog is used to log from the other sebase packages.|
Package vtree is an interface for C compatible vtrees used in sebase.
|Package vtree is an interface for C compatible vtrees used in sebase.|