anton

README

Anton

This project is an open-source tool that extracts and organizes data from the TON blockchain, efficiently storing it in PostgreSQL and ClickHouse databases.

Overview

Before you start, take a look at the official docs.

Consider an arbitrary contract. It has a state that is updated with any transaction on the contract's account. Each state has the contract code and data. The contract data can be complex, but developers typically provide get-methods in the contract, which can be executed to retrieve the necessary data. The TON has standard contracts (such as TEP-62, TEP-74), and they have predefined get-method names. Therefore, you can attempt to match accounts found in the network to these standards by checking the presence of the get-methods. Contract standards also specify TL-B constructor tags (or operation ids) for each acceptable message to contract, defined as the first 32 bits of the parsed message payload cell. So you if you know standard of a given contract, you can determine the type of message to it (for example, NFT item transfer) by parsing the first 32 bits of message body.

Anton allows you to define the contract interface in just one JSON schema. Format of every schema is described in detail in abi/README.md. Every schema comprises contract get-methods, as well as incoming and outgoing message schemas for the contract. Once contract interfaces are defined and stored in the database, Anton begins scanning new blocks on the network. The tool stores every account state, transaction, and message in the database. For get-methods without arguments in the contract interface, Anton emulates these methods and saves the returned values to the database. When a message is sent to a known contract interface, Anton attempts to match the message to a known schema by comparing the parsed operation ID. If the message is successfully parsed using the identified schema, Anton also stores the parsed data.

To explore contract interfaces known to this project, visit the abi/known directory. This will provide you with an understanding of the various contract interfaces already supported and serve as examples for adding your own.

Currently, Anton offers a REST API for retrieving filtered and aggregated data from the databases. To see example queries, refer to the API.md file.

To explore how Anton stores data, visit the migrations' directory.

Project structure

Folder	Description
abi	get-methods and tlb cell parsing
abi/known	contract interfaces known to this project
api/http	JSON API Swagger documentation
docs	only API query examples for now
config	custom postgresql configuration
migrations	database migrations
internal	database repositories and services implementation

Internal directory structure

Folder	Description
core	contains project domain
core/rndm	generates random domain structures
core/filter	describes filters
core/aggregate	describes aggregation metrics
core/repository	implements database repositories with filters and aggregation
app	contains all services interfaces and their configs
app/parser	service determines contract interfaces, parse contract data and message payloads
app/fetcher	service concurrently fetches data from blockchain
app/indexer	service scans blocks and save parsed data to databases
app/query	service aggregates database repositories
api/http	implements the REST API

Starting it up

Cloning repository

git clone https://github.com/tonindexer/anton
cd anton

Running tests

Run tests on abi package:

go test -p 1 $(go list ./... | grep /abi) -covermode=count

Run repositories tests:

# start databases up
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d postgres clickhouse

go test -p 1 $(go list ./... | grep /internal/core) -covermode=count

Running linter

Firstly, install golangci-lint.

golangci-lint run

Configuration

Installation requires some environment variables.

cp .env.example .env
nano .env

Name	Description	Default	Example
DB_NAME	Database name		idx
DB_USERNAME	Database username		user
DB_PASSWORD	Database password		pass
DB_CH_URL	Clickhouse URL to connect to		clickhouse://clickhouse:9000/db_name?sslmode=disable
DB_PG_URL	PostgreSQL URL to connect to		postgres://username:password@postgres:5432/db_name?sslmode=disable
FROM_BLOCK	Master chain seq_no to start from	1	23532000
WORKERS	Number of indexer workers	4	8
LITESERVERS	Lite servers to connect to		135.181.177.59:53312 aF91CuUHuuOv9rm2W5+O/4h38M3sRm40DtSdRxQhmtQ=
DEBUG_LOGS	Debug logs enabled	false	true

Building

# building it locally
go build -o anton .

# build local docker container via docker cli
docker build -t anton:latest .
# or via compose
docker compose -f docker-compose.yml -f docker-compose.dev.yml build

# pull public images
docker compose pull

Running

We have several options for compose run via override files:

• base (docker-compose.yml) - allows to run services with near default configuration;
• dev (docker-compose.dev.yml) - allows to rebuild Anton image locally and exposes databases ports;
• prod (docker-compose.prod.yml) - allows to configure and backup databases, requires at least 128GB RAM.

You can combine it by your own. Also, there are optional profiles:

• migrate - runs optional migrations service.

Take a look at the following run examples:

# run base compose
docker compose up -d

# run dev compose (build docker image locally)
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d

# run prod compose
# WARNING: requires at least 128GB RAM
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

To run Anton, you need at least one defined contract interface. There are some known interfaces in the abi/known directory. You can add them through this command:

docker compose exec web sh -c "anton contract /var/anton/known/*.json"

Schema migration

# run migrations service on running compose
docker compose run migrations

Reading logs

docker compose logs -f

Taking a backup

# starting up databases and API service
docker compose                      \
    -f docker-compose.yml           \
    -f docker-compose.prod.yml      \
        up -d postgres clickhouse web

# stop indexer
docker compose stop indexer

# create backup directories
mkdir backups backups/pg backups/ch

# backing up postgres
docker compose exec postgres pg_dump -U user db_name | gzip > backups/pg/1.pg.backup.gz

# backing up clickhouse (available only with docker-compose.prod.yml)
## connect to the clickhouse
docker compose exec clickhouse clickhouse-client
## execute backup command
# :) BACKUP DATABASE default TO File('/backups/1/');

# execute migrations through API service
docker compose exec web anton migrate up

# start up indexer
docker compose                      \
    -f docker-compose.yml           \
    -f docker-compose.prod.yml      \
        up -d indexer

Using

Show archive nodes from global config

docker run tonindexer/anton archive [--testnet]

Insert contract interface

# add from stdin
cat abi/known/tep81_dns.json | docker compose exec -T web anton contract --stdin
# add from file
docker compose exec web anton contract "/var/anton/known/tep81_dns.json"

Delete contract interface

docker compose exec web anton contract delete "dns_nft_item"

Add address label

docker compose exec web anton label "EQDj5AA8mQvM5wJEQsFFFof79y3ZsuX6wowktWQFhz_Anton" "anton.tools"

# known tonscan labels
docker compose exec web anton label --tonscan