caire

README

Caire is a content aware image resize library based on Seam Carving for Content-Aware Image Resizing paper.

How does it work

• An energy map (edge detection) is generated from the provided image.
• The algorithm tries to find the least important parts of the image taking into account the lowest energy values.
• Using a dynamic programming approach the algorithm will generate individual seams accrossing the image from top to down, or from left to right (depending on the horizontal or vertical resizing) and will allocate for each seam a custom value, the least important pixels having the lowest energy cost and the most important ones having the highest cost.
• Traverse the image from the second row to the last row and compute the cumulative minimum energy for all possible connected seams for each entry.
• The minimum energy level is calculated by summing up the current pixel with the lowest value of the neighboring pixels from the previous row.
• Traverse the image from top to bottom and compute the minimum energy level. For each pixel in a row we compute the energy of the current pixel plus the energy of one of the three possible pixels above it.
• Find the lowest cost seam from the energy matrix starting from the last row and remove it.
• Repeat the process.

The process illustrated:

Original image	Energy map	Seams applied

Features

Key features which differentiates this library from the other existing open source solutions:

• Customizable command line support
• Support for both shrinking or enlarging the image
• Resize image both vertically and horizontally
• Can resize all the images from a directory
• Does not require any third party library
• Use of sobel threshold for fine tuning
• Use of blur filter for increased edge detection
• Square the image with a single command
• Support for proportional scaling
• Face detection to avoid face deformation
• Support for multiple output image type

Face detection

The library is capable detecting human faces prior resizing the images via https://github.com/esimov/pigo, which does not require to have OpenCV installed.

The image below illustrates the application capabilities to detect human faces prior resizing. It's clearly visible from the image that with face detection activated the algorithm will avoid cropping pixels inside faces, retaining the face zone unaltered.

Original image	With face detection	Without face detection

Sample image source

Install

First, install Go, set your GOPATH, and make sure $GOPATH/bin is on your PATH.

$ export GOPATH="$HOME/go"
$ export PATH="$PATH:$GOPATH/bin"

Next download the project and build the binary file.

$ go get -u -f github.com/esimov/caire/cmd/caire
$ go install

MacOS (Brew) install

The library now can be installed via Homebrew. The only thing you need is to run the commands below.

$ brew tap esimov/caire
$ brew install caire

Usage

$ caire -in input.jpg -out output.jpg

To detect faces prior rescaling use the -face flag and provide the face clasification binary file included in the data folder. The sample code below will rescale the provided image with 20% but will check for human faces prior rescaling.

For the face detection related arguments check the Pigo documentation.

$ caire -in input.jpg -out output.jpg -face=1 -cc="data/facefinder" -perc=1 -width=20

Supported commands:

$ caire --help

The following flags are supported:

Flag	Default	Description
in	n/a	Input file
out	n/a	Output file
width	n/a	New width
height	n/a	New height
perc	false	Reduce image by percentage
square	false	Reduce image to square dimensions
scale	false	Proportional scaling
blur	1	Blur radius
sobel	10	Sobel filter threshold
debug	false	Use debugger
face	false	Use face detection
angle	float	Plane rotated faces angle
cc	string	Cascade classifier

In case you wish to scale down the image by a specific percentage, it can be used the -perc boolean flag. For example to reduce the image dimension by 20% both horizontally and vertically you can use the following command:

$ caire -in input/source.jpg -out ./out.jpg -perc=1 -width=20 -height=20 -debug=false

Also the library supports the -square option. When this option is used the image will be resized to a squre, based on the shortest edge.

The -scale option will resize the image proportionally. First the image is scaled down preserving the image aspect ratio, then the seam carving algorithm is applied only to the remaining points. Ex. : given an image of dimensions 2048x1536 if we want to resize to the 1024x500, the tool first rescale the image to 1024x768, then will remove only the remaining 268px. Using this option will drastically reduce the processing time.

The CLI command can process all the images from a specific directory too.

$ caire -in ./input-directory -out ./output-directory

Caire integrations

• Caire can be used as a servesless function via OpenFaaS: https://github.com/esimov/caire-openfaas
• Caire can also be used as a snap function (https://snapcraft.io/caire): $ snap run caire --h

Results

Shrunk images

Original	Shrunk

Enlarged images

Original	Extended

Useful resources

• https://en.wikipedia.org/wiki/Seam_carving
• https://inst.eecs.berkeley.edu/~cs194-26/fa16/hw/proj4-seamcarving/imret.pdf
• http://pages.cs.wisc.edu/~moayad/cs766/download_files/alnammi_cs_766_final_report.pdf
• https://stacks.stanford.edu/file/druid:my512gb2187/Zargham_Nassirpour_Content_aware_image_resizing.pdf

Author

• Endre Simo (@simo_endre)

License

Copyright © 2018 Endre Simo

This project is under the MIT License. See the LICENSE file for the full license text.

Documentation

Overview

Package caire is a content aware image resize library, which can rescale the source image seamlessly both vertically and horizontally by eliminating the less important parts of the image.

The package provides a command line interface, supporting various flags for different types of rescaling operations. To check the supported commands type:

$ caire --help

In case you wish to integrate the API in a self constructed environment here is a simple example:

package main

import (
	"fmt"
	"github.com/esimov/caire"
)

func main() {
	p := &caire.Processor{
		// Initialize struct variables
	}

	if err := p.Process(in, out); err != nil {
		fmt.Printf("Error rescaling image: %s", err.Error())
	}
}

Index

• Constants
• Variables
• func Grayscale(src *image.NRGBA) *image.NRGBA
• func RemoveTempImage(tmpImage string)
• func Resize(s SeamCarver, img *image.NRGBA) (image.Image, error)
• func SobelFilter(img *image.NRGBA, threshold float64) *image.NRGBA
• func StackBlur(img *image.NRGBA, radius uint32) *image.NRGBA
• type ActiveSeam
• type Carver
  • func NewCarver(width, height int) *Carver
  • func (c *Carver) AddSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA
  • func (c *Carver) ComputeSeams(img *image.NRGBA, p *Processor)
  • func (c *Carver) FindLowestEnergySeams() []Seam
  • func (c *Carver) RemoveSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA
  • func (c *Carver) RotateImage270(src *image.NRGBA) *image.NRGBA
  • func (c *Carver) RotateImage90(src *image.NRGBA) *image.NRGBA
• type Processor
  • func (p *Processor) Process(r io.Reader, w io.Writer) error
  • func (p *Processor) Resize(img *image.NRGBA) (image.Image, error)
• type Seam
• type SeamCarver
• type UsedSeams

Constants

const MaxResizeWithoutScaling = 2000

Variables

var TempImage = fmt.Sprintf("%d.jpg", time.Now().Unix())

TempImage temporary image file.

Functions

func Grayscale

func Grayscale(src *image.NRGBA) *image.NRGBA

Grayscale converts the image to grayscale mode.

func RemoveTempImage

func RemoveTempImage(tmpImage string)

RemoveTempImage removes the temporary image generated during face detection process.

func Resize

func Resize(s SeamCarver, img *image.NRGBA) (image.Image, error)

Resize implements the Resize method of the Carver interface. It returns the concrete resize operation method.

func SobelFilter

func SobelFilter(img *image.NRGBA, threshold float64) *image.NRGBA

SobelFilter detects image edges. See https://en.wikipedia.org/wiki/Sobel_operator

func StackBlur

func StackBlur(img *image.NRGBA, radius uint32) *image.NRGBA

StackBlur applies a blur filter to the provided image. The radius defines the bluring average.

Types

type ActiveSeam

type ActiveSeam struct {
	Seam
	Pix color.Color
}

ActiveSeam contains the current seam position and color.

type Carver

type Carver struct {
	Width  int
	Height int
	Points []float64
}

Carver is the main entry struct having as parameters the newly generated image width, height and seam points.

func NewCarver

func NewCarver(width, height int) *Carver

NewCarver returns an initialized Carver structure.

func (*Carver) AddSeam

func (c *Carver) AddSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA

AddSeam add new seam.

func (*Carver) ComputeSeams

func (c *Carver) ComputeSeams(img *image.NRGBA, p *Processor)

ComputeSeams compute the minimum energy level based on the following logic:

• traverse the image from the second row to the last row and compute the cumulative minimum energy M for all possible connected seams for each entry (i, j).

• the minimum energy level is calculated by summing up the current pixel value with the minimum pixel value of the neighboring pixels from the previous row.

func (*Carver) FindLowestEnergySeams

func (c *Carver) FindLowestEnergySeams() []Seam

FindLowestEnergySeams find the lowest vertical energy seam.

func (*Carver) RemoveSeam

func (c *Carver) RemoveSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA

RemoveSeam remove the least important columns based on the stored energy (seams) level.

func (*Carver) RotateImage270

func (c *Carver) RotateImage270(src *image.NRGBA) *image.NRGBA

RotateImage270 rotate the image by 270 degree counter clockwise.

func (*Carver) RotateImage90

func (c *Carver) RotateImage90(src *image.NRGBA) *image.NRGBA

RotateImage90 rotate the image by 90 degree counter clockwise.

type Processor

type Processor struct {
	SobelThreshold int
	BlurRadius     int
	NewWidth       int
	NewHeight      int
	Percentage     bool
	Square         bool
	Debug          bool
	Scale          bool
	FaceDetect     bool
	FaceAngle      float64
	Classifier     string
}

Processor options

func (*Processor) Process

func (p *Processor) Process(r io.Reader, w io.Writer) error

Process is the main function having as parameters an input reader and an output writer. We are using the io package, because this way we can provide different types of input and output source, as long as they implement the io.Reader and io.Writer interface.

func (*Processor) Resize

func (p *Processor) Resize(img *image.NRGBA) (image.Image, error)

Resize method takes the source image and rescales it using the parameters provided. The new image can be rescaled either horizontally or vertically (or both). Depending on the provided parameters the image can be either reduced or enlarged.

type Seam

type Seam struct {
	X int
	Y int
}

Seam struct contains the seam pixel coordinates.

type SeamCarver

type SeamCarver interface {
	Resize(*image.NRGBA) (image.Image, error)
}

SeamCarver interface defines the Resize method. This has to be implemented by every struct which declares a Resize method.

type UsedSeams

type UsedSeams struct {
	ActiveSeam []ActiveSeam
}

UsedSeams contains the already generated seams.