lescatit

README

Lescatit _{(Let's categorized it)}

Lescatit is a project developed in go, mongo, docker, and kubernetes technologies, providing URL crawling and categorization functionality.

Table of Contents

• Features
• Setup
• Usage
• License

Features

Lescatit offers the following features:

• Getting user(s) information
• Deleting a user
• Changing user roles
• Updating user passwords
• Updating user email addresses
• Updating usernames
• Getting content of URL(s)
• Crawling URL(s)
• Categorizing URL(s)
• Generating a classification model
• Getting the classification model
• Updating the classification model
• Deleting classification model(s)
• Listing all classification models
• Getting URL categories
• Updating URL categories
• Reporting miscategorization
• Adding URL addresses
• Deleting URL(s)
• Listing all URLs

Setup

To set up Lescatit, follow these steps:

1. Download the latest version:

   LATEST_VERSION=$(wget -qO - https://api.github.com/repos/mtnmunuklu/Lescatit/releases/latest \
   | grep tag_name \
   | cut -d  '"' -f 4)
   
   curl -LJO https://github.com/mtnmunuklu/Lescatit/archive/refs/tags/$LATEST_VERSION.tar.gz
   

2. Extract the downloaded file:

   FILE_NAME=Lescatit-$(echo $LATEST_VERSION | cut -d 'v' -f 2)
   tar -xvf $FILE_NAME.tar.gz
   

3. Execute the setup scripts:

   cd $FILE_NAME/scripts
   # Execute on worker and control plane servers.
   bash setup_tools.sh
   bash setup_k8s.sh
   # Execute only on the first control plane server.
   # It will create setup_k8s_control_plane.sh and setup_k8s_worker.sh files.
   # Control plane and worker scripts are for joining the Kubernetes cluster.
   # You can use these scripts on new nodes when you add new nodes as control plane or worker.
   bash setup_k8s_first_control_plane.sh
   

Usage

Lescatit consists of 5 different services: authentication, crawler, categorizer, categorization, and api. All incoming requests are first forwarded to the API service. Afterwards, the API service decides to which service the incoming request will be forwarded. The requested URL plays a role in the decision-making process.

To understand the features of each service, the available endpoints, how to make requests, and the expected responses, refer to the api.pdf file under the docs folder.

You can also access the documents describing the software structure of each service under the docs folder.

License

This project is licensed under the terms of the MIT license.

You can check out the full license here