fnlnews

README

fnl-news

The story

As a big fan of the Alania Vladikavkaz football club, which plays in the second Russian league (FNL), I want to get all news about the team from all sports sites in one place. I guess, there are other fans, who want to get news about their teams in FNL. So the FNL-news is a small service, which parses several most popular Russian sports sites, added team-tags to the articles (#Alania if FC Alania mentioned in an article, or #Torpedo if FC Torpedo mentioned), and posts the links in a telegram channel.

Run

Config

The service needs PostgreSQL database DSN, telegram bot access token, and telegram channel id. Default values are in the .env files, which can be replaced by values from .env.local. Just copy the file and change the values

cp .env .env.local

Run test

 go test ./tests/...

Build

go build -o fnl_news cmd/main.go

Run

./fnl_news

Database

I was trying to keep it light and simple, so there is only one table in the database: articles. It will be created when you first start the service if the database DSN is correct.

If you don't have installed PostgreSQL, you can install it from docker.

docker compose up

will create a container with PostgreSQL database. The DSN for the database

DATABASE_DSN=host=localhost user=fnluser password=123456 dbname=fnl port=5432 sslmode=disable

Note that there is no service container, only database. Use it only for developing, not in production.

Modules and packages

Modules

• Resty for HTTP requests. To get articles and send links to the channel.
• Logrus as a logger
• Viper for dealing with config files
• dig for dependency injection
• GORM as an ORM
• Testify for unit tests

Packages

• pkg/config - Wrapper for the Viper module
• pkg/httpclient - Wrapper for the Resty module

Internal packages

• internal/domain - Domain entities. Only article entity for now.
• internal/parser - The parser. Parses sources and returns the articles list. Also contains Tag Matcher which defines team-tags for an article
• internal/publisher - The publishers. There is only the Telegram publisher for now. All Publishers must implement the publisher.Publisher interface.
• internal/repository - Article Repository. Knows how to query articles from the database.
• internal/source - The sport sites sources. All must implement source.Source interface. Each Source contains regular expressions for extracting data from its HTML, and methods to modify and check the data.

Result

Telegram channel

Todo

• Domain entity Article depends on Gorm module as far as contains gorm descriptions for the fields.
• Parsers for different sources are described manually (internal/parser/ParsersFactory), maybe it is a good idea to automate it somehow
• ArticleRepository knows how to mark an article as sent for a concrete publisher(Telegram publisher). I guess we don't want to edit the article repository every time when a new publisher was added.
• httpClientResty and httpClientResty1251 have similar Post method.
• Add tests

Documentation

Index

• func DiCreateArticleRepository(db *gorm.DB) repository.Article
• func DiCreateConfig() *config.Config
• func DiCreateDB(cfg *config.Config) *gorm.DB
• func DiCreateHTTPClient() httpclient.Client
• func DiCreateParsers(matcher parser.TagMatcher, client httpclient.Client) []*parser.Parser
• func DiCreatePublishers(articleRepository repository.Article, config *config.Config, ...) []publisher.Publisher
• func DiCreateTagMatcher() parser.TagMatcher
• type App
  • func DiCreateApp(cfg *config.Config, db *gorm.DB, articleRepository repository.Article, ...) *App
  • func (app App) Start()

Functions

func DiCreateArticleRepository

func DiCreateArticleRepository(db *gorm.DB) repository.Article

DiCreateArticleRepository di function for article repository

func DiCreateConfig

func DiCreateConfig() *config.Config

DiCreateConfig di function for config

func DiCreateDB

func DiCreateDB(cfg *config.Config) *gorm.DB

DiCreateDB di function for database

func DiCreateHTTPClient

func DiCreateHTTPClient() httpclient.Client

DiCreateHTTPClient di function http client

func DiCreateParsers

func DiCreateParsers(matcher parser.TagMatcher, client httpclient.Client) []*parser.Parser

DiCreateParsers di function for parsers list

func DiCreatePublishers

func DiCreatePublishers(
	articleRepository repository.Article,
	config *config.Config,
	client httpclient.Client,
) []publisher.Publisher

DiCreatePublishers di function for publishers lis

func DiCreateTagMatcher

func DiCreateTagMatcher() parser.TagMatcher

DiCreateTagMatcher di function for tag matcher

Types

type App

type App struct {
	Cfg               *config.Config
	Db                *gorm.DB
	ArticleRepository repository.Article
	// contains filtered or unexported fields
}

App struct

func DiCreateApp

func DiCreateApp(
	cfg *config.Config,
	db *gorm.DB,
	articleRepository repository.Article,
	publishers []publisher.Publisher,
	parsers []*parser.Parser,
) *App

DiCreateApp di function for app

func (App) Start

func (app App) Start()

Start the app