README
¶
If you like the project, please ★ star this repository to show your support! 🤩
A Machine Learning library written in pure Go designed to support relevant neural architectures in Natural Language Processing.
spaGO is self-contained, in that it uses its own lightweight computational graph framework for both training and inference, easy to understand from start to finish.
Features
Natural Language Processing
- Memory-efficient Word Embeddings (with badger key–value store)
- Character Language Models
- Recurrent Sequence Labeler with CRF on top ( e.g. Named Entities Recognition)
- Transformer models:
- Masked Language Model
- Next sentence prediction
- Tokens Classification
- Text Classification (e.g. Sentiment Analysis)
- Question Answering
- Natural Language Inference (i.e. Textual Entailment)
- Zero-Shot Text Classification
- Text Similarity
- Machine Translation
Internal Machine Learning Framework
-
Automatic differentiation:
- Define-by-Run (default, just like PyTorch does)
- Define-and-Run (similar to the static graph of TensorFlow)
-
Optimization methods:
- Gradient descent (Adam, RAdam, RMS-Prop, AdaGrad, SGD)
- Differential Evolution
-
Neural networks:
- Feed-forward models (Linear, Highway, Convolution, ...)
- Recurrent models (LSTM, GRU, BiLSTM...)
- Attention mechanisms (Self-Attention, Multi-Head Attention, ...)
- Recursive auto-encoders
Additional features
spaGO is compatible with pre-trained state-of-the-art neural models:
- Hugging Face Transformers
- Flair sequence labeler architecture
Usage
Requirements:
Clone this repo or get the library:
go get -u github.com/nlpodyssey/spago
spaGO supports two main use cases, which are explained more in detail in the following.
CLI mode
Several programs can be leveraged to tour the current NLP capabilities in spaGO. A list of the demos now follows.
- Hugging Face Importer
- Masked Language Model
- Question Answering
- Machine Translation
- Named Entities Recognition
The Docker image can be built like this.
docker build -t spago:main . -f Dockerfile
Library mode
You can access the core functionality of spaGO, i.e. optimizing mathematical expressions by back-propagating gradients through a computational graph, in your own code by using spaGO in library mode.
At a high level, it comprises four main modules:
- Matrix
- Graph
- Model
- Optimizer
To get started, look at the implementation of built-in neural models, such as the LSTM. Don't be afraid, it is straightforward Go code. The idea is that you could have written spaGO :)
You may find a Feature Source Tree useful for a quick overview of the library's package organization.
There is also a repo with handy examples, such as MNIST classification.
Current Status
We're not at a v1.0.0 yet, so spaGO is currently work-in-progress.
However, it has been running smoothly for a quite a few months now in a system that analyzes thousands of news items a day!
Besides, it's pretty easy to get your hands on through, so you might want to use it in your real applications.
Early adopters may make use of it for production use today as long as they understand and accept that spaGO is not fully tested and that APIs might change.
Known Limits
Sadly, at the moment, spaGO is not GPU friendly by design.
Contributing
We're glad you're thinking about contributing to spaGO! If you think something is missing or could be improved, please open issues and pull requests. If you'd like to help this project grow, we'd love to have you!
To start contributing, check the Contributing Guidelines.
Contact
We encourage you to write an issue. This would help the community grow.
If you really want to write to us privately, please email Matteo Grella with your questions or comments.
Projects Using spaGO
Below is a list of known projects that use spaGO:
- Golem - A batteries-included implementation of "TabNet: Attentive Interpretable Tabular Learning".
- WhatsNew - A simple tool to collect and process quite a few web news from multiple sources.
- Translator - A simple self-hostable Machine Translation service.
Other Links
Acknowledgments
spaGO is a personal project that is part of the open-source NLP Odyssey initiative initiated by members of the EXOP team. I would therefore like to thank EXOP GmbH here, which is providing full support for development by promoting the project and giving it increasing importance.
Sponsors
We appreciate contributions of all kinds. We especially want to thank spaGO fiscal sponsors who contribute to ongoing project maintenance.
Our aim is simplifying people's life by making lending easy, fast and accessible, leveraging Open Banking and Artificial Intelligence. https://www.faire.ai/
We work on Artificial Intelligence based hardware and software systems, declining them in areas such as Energy Management, Personal Safety, E-Health and Sports equipment. https://hype-design.it/
Professional services in the IT sector for Local Administrations, Enterprises and Local Authorities. https://www.boxxapps.com/
See our Open Collective page if you too are interested in becoming a sponsor.
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
ner
This is the first attempt to launch a sequence labeling server from the command line.
|
This is the first attempt to launch a sequence labeling server from the command line. |
|
embeddings
|
|
|
store/diskstore
Module
|
|
|
graphviz
module
|
|
|
nn
|
|
|
approxlinear
Module
|
|
|
pkg
|
|
|
mat32/internal/asm/f32
Package f32 provides float32 vector primitives.
|
Package f32 provides float32 vector primitives. |
|
mat64/internal/asm/f64
Package f64 provides float64 vector primitives.
|
Package f64 provides float64 vector primitives. |
|
ml/ag/encoding/dot
Package dot creates a graphviz compatible version of the ag.Graph.
|
Package dot creates a graphviz compatible version of the ag.Graph. |
|
ml/encoding/fofe
Package fofe provides an implementation of the Fixed-size Ordinally-Forgetting Encoding (FOFE) method.
|
Package fofe provides an implementation of the Fixed-size Ordinally-Forgetting Encoding (FOFE) method. |
|
ml/nn/attention/lshattention
Package lshattention provides an implementation of the LSH-Attention model, as describe in `Reformer: The Efficient Transformer` by N. Kitaev, Ł. Kaiser, A. Levskaya (https://arxiv.org/pdf/2001.04451.pdf).
|
Package lshattention provides an implementation of the LSH-Attention model, as describe in `Reformer: The Efficient Transformer` by N. Kitaev, Ł. Kaiser, A. Levskaya (https://arxiv.org/pdf/2001.04451.pdf). |
|
ml/nn/attention/syntheticattention
Package syntheticattention provides an implementation of the Synthetic Attention described in: "SYNTHESIZER: Rethinking Self-Attention in Transformer Models" by Tay et al., 2020.
|
Package syntheticattention provides an implementation of the Synthetic Attention described in: "SYNTHESIZER: Rethinking Self-Attention in Transformer Models" by Tay et al., 2020. |
|
ml/nn/birnncrf
Package birnncrf provides an implementation of a Bidirectional Recurrent Neural Network (BiRNN) with a Conditional Random Fields (CRF) on tom.
|
Package birnncrf provides an implementation of a Bidirectional Recurrent Neural Network (BiRNN) with a Conditional Random Fields (CRF) on tom. |
|
ml/nn/bls
Package bls provides an implementation of the Broad Learning System (BLS) described in "Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture" by C. L. Philip Chen and Zhulin Liu, 2017.
|
Package bls provides an implementation of the Broad Learning System (BLS) described in "Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture" by C. L. Philip Chen and Zhulin Liu, 2017. |
|
ml/nn/conv1x1
Package conv1x1 implements a 1-dimensional 1-kernel convolution model
|
Package conv1x1 implements a 1-dimensional 1-kernel convolution model |
|
ml/nn/gmlp
Package gmlp implements a model composed by basic MLP layers with gating mechanism.
|
Package gmlp implements a model composed by basic MLP layers with gating mechanism. |
|
ml/nn/gnn/slstm
Package slstm implements a Sentence-State LSTM graph neural network.
|
Package slstm implements a Sentence-State LSTM graph neural network. |
|
ml/nn/gnn/startransformer
Package startransformer provides a variant implementation of the Star-Transformer model introduced by Qipeng Guo, Xipeng Qiu et al.
|
Package startransformer provides a variant implementation of the Star-Transformer model introduced by Qipeng Guo, Xipeng Qiu et al. |
|
ml/nn/normalization/adanorm
Package adanorm implements the Adaptive Normalization (AdaNorm) method.
|
Package adanorm implements the Adaptive Normalization (AdaNorm) method. |
|
ml/nn/normalization/fixnorm
Package fixnorm implements the fixnorm normalization method.
|
Package fixnorm implements the fixnorm normalization method. |
|
ml/nn/normalization/layernorm
Package layernorm implements the Layer Normalization (LayerNorm) i method.
|
Package layernorm implements the Layer Normalization (LayerNorm) i method. |
|
ml/nn/normalization/layernormsimple
Package layernormsimple implements a simple version of LayerNorm (LayerNorm-simple).
|
Package layernormsimple implements a simple version of LayerNorm (LayerNorm-simple). |
|
ml/nn/normalization/rmsnorm
Package rmsnorm implements the Root Mean Square Layer Normalization method.
|
Package rmsnorm implements the Root Mean Square Layer Normalization method. |
|
ml/nn/rae
Package rae provides an implementation of the recursive auto-encoder strategy described in "Towards Lossless Encoding of Sentences" by Prato et al., 2019.
|
Package rae provides an implementation of the recursive auto-encoder strategy described in "Towards Lossless Encoding of Sentences" by Prato et al., 2019. |
|
ml/nn/rc
Package rc contains built-in Residual Connections (RC).
|
Package rc contains built-in Residual Connections (RC). |
|
ml/nn/recurrent/horn
Package horn provides an implementation of Higher Order Recurrent Neural Networks (HORN).
|
Package horn provides an implementation of Higher Order Recurrent Neural Networks (HORN). |
|
ml/nn/recurrent/lstmsc
Package lstmsc provides an implementation of LSTM enriched with a PolicyGradient to enable Dynamic Skip Connections.
|
Package lstmsc provides an implementation of LSTM enriched with a PolicyGradient to enable Dynamic Skip Connections. |
|
ml/nn/recurrent/mist
Package mist provides an implementation of the MIST (MIxed hiSTory) recurrent network as described in "Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies" by Di Pietro et al., 2018 (https://arxiv.org/pdf/1702.07805.pdf).
|
Package mist provides an implementation of the MIST (MIxed hiSTory) recurrent network as described in "Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies" by Di Pietro et al., 2018 (https://arxiv.org/pdf/1702.07805.pdf). |
|
ml/nn/recurrent/nru
Package nru provides an implementation of the NRU (Non-Saturating Recurrent Units) recurrent network as described in "Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies" by Chandar et al., 2019.
|
Package nru provides an implementation of the NRU (Non-Saturating Recurrent Units) recurrent network as described in "Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies" by Chandar et al., 2019. |
|
ml/nn/recurrent/rla
Package rla provides an implementation of RLA (Recurrent Linear Attention).
|
Package rla provides an implementation of RLA (Recurrent Linear Attention). |
|
ml/nn/recurrent/srnn
Package srnn implements the SRNN (Shuffling Recurrent Neural Networks) by Rotman and Wolf, 2020.
|
Package srnn implements the SRNN (Shuffling Recurrent Neural Networks) by Rotman and Wolf, 2020. |
|
ml/nn/sgu
Package sgu implements the Spatial Gating Unit (SGU).
|
Package sgu implements the Spatial Gating Unit (SGU). |
|
nlp/charlm
Package charlm provides an implementation of a character-level language model that uses a recurrent neural network as its backbone.
|
Package charlm provides an implementation of a character-level language model that uses a recurrent neural network as its backbone. |
|
nlp/contextualstringembeddings
Package contextualstringembeddings provides an implementation of the "Contextual String Embeddings" of words (Akbik et al., 2018).
|
Package contextualstringembeddings provides an implementation of the "Contextual String Embeddings" of words (Akbik et al., 2018). |
|
nlp/evolvingembeddings
Package evolvingembeddings provides a word embedding model that evolves by dynamically aggregating contextual embeddings over time during inference.
|
Package evolvingembeddings provides a word embedding model that evolves by dynamically aggregating contextual embeddings over time during inference. |
|
nlp/sequencelabeler
Package sequencelabeler provides an implementation of a sequence labeling architecture composed by Embeddings -> BiRNN -> Scorer -> CRF.
|
Package sequencelabeler provides an implementation of a sequence labeling architecture composed by Embeddings -> BiRNN -> Scorer -> CRF. |
|
nlp/stackedembeddings
Package stackedembeddings provides convenient types to stack multiple word embedding representations by concatenating them.
|
Package stackedembeddings provides convenient types to stack multiple word embedding representations by concatenating them. |
|
nlp/tokenizers
Package tokenizers is an interim solution while developing `gotokenizers` (https://github.com/nlpodyssey/gotokenizers).
|
Package tokenizers is an interim solution while developing `gotokenizers` (https://github.com/nlpodyssey/gotokenizers). |
|
nlp/tokenizers/basetokenizer
Package basetokenizer provides an implementation of a very simple tokenizer that splits by white-spaces (and alike) and punctuation symbols.
|
Package basetokenizer provides an implementation of a very simple tokenizer that splits by white-spaces (and alike) and punctuation symbols. |
|
nlp/tokenizers/sentencepiece/internal/sentencepiece
Package sentencepiece implements the SentencePiece encoder (Kudo and Richardson, 2018).
|
Package sentencepiece implements the SentencePiece encoder (Kudo and Richardson, 2018). |
|
nlp/transformers/bart
Package bart implements the transformer model introduced by Mike et al., 2019.
|
Package bart implements the transformer model introduced by Mike et al., 2019. |
|
nlp/transformers/bert
Package bert provides an implementation of BERT model (Bidirectional Encoder Representations from Transformers).
|
Package bert provides an implementation of BERT model (Bidirectional Encoder Representations from Transformers). |
|
nlp/transformers/generation
Package generation implements a generation search algorithm for conditional generation.
|
Package generation implements a generation search algorithm for conditional generation. |