license_plate_recognition

README

License Plate Recognition with go-darknet

Table of Contents

• About
• Requirements
• Installation
  • Get source code
  • Protobuf generation
  • Neural net weights
  • Custom handler
• Usage
  • Server
  • Client

About

This is a gRPC server which accepts image and can make license plate recognition (using YOLOv3 or YOLOv4 neural network).

Neural networks were trained on dataset of russian license plates. But you can train it on another dataset - read about process here https://github.com/AlexeyAB/darknet#how-to-train-to-detect-your-custom-objects

Server tries to find license plate at first. Then it does OCR (if it's possible) and provides output which can be represented later as follows:

Sample plate #1	Sample plate #2

Images has been taken near my house

Darknet architecture for finding license plates - Yolo V3

Darknet architecture for doing OCR stuff - Yolo V4

gRPC server accepts this data struct according proto3 specification

// Essential information to process
message LPRRequest{
    // Bytes representation of image (PNG)
    bytes image = 1;
    // Optional information about image. Could be usefull if client-side already knows where license plate should be located (due some object detections technique)
    BBox bbox = 2;
}

// Reference information about detection rectangle
message BBox{
    int32 x_left = 1;
    int32 y_top = 2;
    int32 height = 3;
    int32 width = 4;
}

The gRPC server response is:

// Response from server
message LPRResponse{
    // Set of found license plates with corresponding information
    repeated LPRInfo license_plates = 1;
    // Number of seconds has taken to proccess license plate detections and OCR
    float elapsed = 2;
    // Optional message from server
    string message = 3;
    // Optional warning message from server. If it is not empty you probably should investiage such behavior
    string warning = 4;
    // Optional error message from server. If it is not empty you should investiage the error
    string error = 5;
}

// Information about single license plate
message LPRInfo {
    // License plate location
    BBox bbox = 1;
    // License plate OCR bounding bboxes. Bounding bboxes are sorted by horizontal line
    // Warning: those coordinates are relative to license plate bounding box, not the parent image!
    repeated BBox ocr_bboxes = 2;
    // License plate text
    string text = 3;
}

Full gRPC documentation is here: HTML or Markdown

Requirements

$\textcolor{red}{\textsf{No OpenCV installation is needed!}}$

Please follow instructions from go-darknet. There you will know how to install AlexeyAB's darknet and Go-binding for it.

Instalation

Get source code

Notice: we are using Go-modules

go get https://github.com/LdDl/license_plate_recognition

Download weights and configuration

Notice: please read source code of *.sh script before downloading. This script MAY NOT fit yours needs.

cd cmd/
chmod +x download_data_RU.sh
./download_data_RU.sh

Usage

Start server

• Navigate to folder with server application source code

  cd cmd/server
  

• Build source code of server application to executable

  go build -o recognition_server main.go
  

• Run server application

  ./recognition_server --cfg conf.toml
  

  Note: Please see conf.toml description for correct usage

• On server's side the directory './detected' will appear if you provide save_detected = true in TOML configuration. Detected license plates with OCR annotations will be stored there.

Test Client-Server

Notice: server should be started

• Navigate to folder with server application source code

  cd cmd/client
  

• Build source code of client application to executable

  go build -o client_app main.go
  

• Run client application

  ./client_app --host=localhost --port=50051 --file=sample.jpg -x 0 -y 0 --width=4032 --height=3024
  

• Check, if server can handle error (like negative height parameter):

  ./client_app --host=localhost --port=50051 --file=sample.jpg -x 0 -y 0 --width=42 --height=-24
  

• On client's side there will be output something like this:

  Elapsed seconds: 0.17216872
  Detections num: 2
  Detection #0:
          Text: A100CX777
          Plate bbox: x_left:2027 y_top:2027 height:304 width:646
          OCR bboxes: [x_left:109 y_top:109 height:71 width:71 x_left:186 y_top:186 height:77 width:59 x_left:252 y_top:252 height:79 width:66 x_left:323 y_top:323 height:80 width:73 x_left:404 y_top:404 height:77 width:77 x_left:485 y_top:485 height:70 width:83 x_left:584 y_top:584 height:72 width:71 x_left:657 y_top:657 height:76 width:68 x_left:731 y_top:731 height:76 width:66]
  Detection #1:
          Text: M288HO199
          Plate bbox: x_left:262 y_top:262 height:186 width:391
          OCR bboxes: [x_left:56 y_top:56 height:28 width:25 x_left:86 y_top:86 height:36 width:24 x_left:112 y_top:112 height:36 width:25 x_left:139 y_top:139 height:39 width:27 x_left:168 y_top:168 height:28 width:27 x_left:196 y_top:196 height:28 width:31 x_left:232 y_top:232 height:30 width:21 x_left:253 y_top:253 height:31 width:21 x_left:276 y_top:276 height:30 width:23]
  

Documentation

Index

• Variables
• type Configuration
• type Detections
  • func (r Detections) Len() int
  • func (r Detections) Less(i, j int) bool
  • func (r Detections) Swap(i, j int)
• type LPRQueue
  • func NewLPRQueue(network *YOLONetwork, queueLimit int, saveDetected bool) *LPRQueue
  • func (q *LPRQueue) SendToQueue(req *QueueRequest) (*QueueResponse, error)
  • func (q *LPRQueue) WaitRequests()
• type PlateResponse
  • func (presp *PlateResponse) String() string
• type QueueRequest
  • func NewQueueRequest(ctx context.Context, image *image.NRGBA) QueueRequest
• type QueueResponse
• type YOLONetwork
  • func NewYOLONetwork(platesCfg, platesWeights, ocrCfg, ocrWeights string, ...) (*YOLONetwork, error)
  • func (net *YOLONetwork) ReadLicensePlates(imgSrc image.Image, saveCrop bool) (*YOLOResponse, error)
• type YOLOResponse
  • func (resp *YOLOResponse) String() string

Variables

var (
	ErrMaxTimeout = fmt.Errorf("max timeout is 1 minute")
)

Types

type Configuration

type Configuration struct {
	ServerConf serverInstanceConfiguration `toml:"server"`
	YOLOPlates yoloConfiguration           `toml:"yolo_plates"`
	YOLOOCR    yoloConfiguration           `toml:"yolo_ocr"`
}

type Detections

type Detections []*darknet.Detection

Detections slice of image.Rectangle (for sorting)

func (Detections) Len

func (r Detections) Len() int

func (Detections) Less

func (r Detections) Less(i, j int) bool

func (Detections) Swap

func (r Detections) Swap(i, j int)

type LPRQueue

type LPRQueue struct {
	// contains filtered or unexported fields
}

LPRQueue is a queue to handle asynchronous requests to neural network

func NewLPRQueue

func NewLPRQueue(network *YOLONetwork, queueLimit int, saveDetected bool) *LPRQueue

func (*LPRQueue) SendToQueue

func (q *LPRQueue) SendToQueue(req *QueueRequest) (*QueueResponse, error)

SendToQueue is wrapper around request and its context

func (*LPRQueue) WaitRequests

func (q *LPRQueue) WaitRequests()

WaitRequests is endless loop for waiting frames

type PlateResponse

type PlateResponse struct {
	Text          string
	Probability   float64
	Rect          image.Rectangle
	CroppedNumber *image.NRGBA
	OCRClassesIDs []int
	OCRRects      []image.Rectangle
}

PlateResponse Detected license plate information

func (*PlateResponse) String

func (presp *PlateResponse) String() string

type QueueRequest

type QueueRequest struct {
	ImageData *image.NRGBA
	// contains filtered or unexported fields
}

QueueRequest is wrapping around image and response channel

func NewQueueRequest

func NewQueueRequest(ctx context.Context, image *image.NRGBA) QueueRequest

type QueueResponse

type QueueResponse struct {
	Resp  *YOLOResponse
	Error error
}

QueueResponse is just response from YOLO

type YOLONetwork

type YOLONetwork struct {
	LicensePlates *darknet.YOLONetwork
	OCR           *darknet.YOLONetwork
}

YOLONetwork Aggregate two neural networks: one is for finding license plates, another is for OCR

func NewYOLONetwork

func NewYOLONetwork(platesCfg, platesWeights, ocrCfg, ocrWeights string, platesThreshold, ocrThreshold float32) (*YOLONetwork, error)

NewYOLONetwork Return pointer to YOLONetwork

func (*YOLONetwork) ReadLicensePlates

func (net *YOLONetwork) ReadLicensePlates(imgSrc image.Image, saveCrop bool) (*YOLOResponse, error)

ReadLicensePlates Returns found license plates with information about each one

type YOLOResponse

type YOLOResponse struct {
	Plates  []PlateResponse
	Elapsed time.Duration
}

YOLOResponse Neural net's response

func (*YOLOResponse) String

func (resp *YOLOResponse) String() string