caire

README

Caire is a content aware image resize library based on Seam Carving for Content-Aware Image Resizing paper.

How does it work

• An energy map (edge detection) is generated from the provided image.
• The algorithm tries to find the least important parts of the image taking into account the lowest energy values.
• Using a dynamic programming approach the algorithm will generate individual seams across the image from top to down, or from left to right (depending on the horizontal or vertical resizing) and will allocate for each seam a custom value, the least important pixels having the lowest energy cost and the most important ones having the highest cost.
• We traverse the image from the second row to the last row and compute the cumulative minimum energy for all possible connected seams for each entry.
• The minimum energy level is calculated by summing up the current pixel value with the lowest value of the neighboring pixels obtained from the previous row.
• We traverse the image from top to bottom and compute the minimum energy level. For each pixel in a row we compute the energy of the current pixel plus the energy of one of the three possible pixels above it.
• Find the lowest cost seam from the energy matrix starting from the last row and remove it.
• Repeat the process.

The process illustrated:

Original image	Energy map	Seams applied

Features

Key features which differentiates this library from the other existing open source solutions:

• GUI progress indicator
• Customizable command line support
• Support for both shrinking or enlarging the image
• Resize image both vertically and horizontally
• Face detection to avoid face deformation
• Support for multiple output image type (jpg, jpeg, png, bmp)
• Support for stdin and stdout pipe commands
• Can process whole directories recursively and concurrently
• Use of sobel threshold for fine tuning
• Use of blur filter for increased edge detection
• Support for squaring the image with a single command
• Support for proportional scaling
• Support for protective mask
• Support for removal mask
• GUI debug mode support

Install

First, install Go, set your GOPATH, and make sure $GOPATH/bin is on your PATH.

$ go install github.com/esimov/caire/cmd/caire@latest

MacOS (Brew) install

The library can also be installed via Homebrew.

$ brew install caire

Usage

$ caire -in input.jpg -out output.jpg

Supported commands:

$ caire --help

The following flags are supported:

Flag	Default	Description
in	-	Input file
out	-	Output file
width	n/a	New width
height	n/a	New height
preview	true	Show GUI window
perc	false	Reduce image by percentage
square	false	Reduce image to square dimensions
blur	4	Blur radius
sobel	2	Sobel filter threshold
debug	false	Use debugger
face	false	Use face detection
angle	float	Plane rotated faces angle
mask	string	Mask file path
rmask	string	Remove mask file path
color	string	Seam color (default #ff0000)
shape	string	Shape type used for debugging: circle,line (default circle)

Face detection

The library is capable of detecting human faces prior resizing the images by using the lightweight Pigo (https://github.com/esimov/pigo) face detection library.

The image below illustrates the application capabilities for human face detection prior resizing. It's clearly visible that with face detection activated the algorithm will avoid cropping pixels inside the detected faces, retaining the face zone unaltered.

Original image	With face detection	Without face detection

Sample image source

GUI progress indicator

A GUI preview mode is also incorporated into the library for in time process visualization. The Gio GUI library has been used because of its robustness and modern architecture. Prior running it please make sure that you have installed all the required dependencies noted in the installation section (https://gioui.org/#installation) .

The preview window is activated by default but you can deactivate it any time by setting the -preview flag to false. When the images are processed concurrently from a directory the preview mode is deactivated.

Face detection to avoid face deformation

In order to detect faces prior rescaling, use the -face flag. There is no need to provide a face classification file, since it's already embedded into the generated binary file. The sample code below will resize the provided image with 20%, but checks for human faces in order tot avoid face deformations.

For face detection related settings please check the Pigo documentation.

$ caire -in input.jpg -out output.jpg -face=1 -perc=1 -width=20

Support for stdin and stdout pipe commands

You can also use stdin and stdout with -:

$ cat input/source.jpg | caire -in - -out - >out.jpg

in and out default to - so you can also use:

$ cat input/source.jpg | caire >out.jpg
$ caire -out out.jpg < input/source.jpg

You can provide also an image URL for the -in flag or even use curl or wget as a pipe command in which case there is no need to use the -in flag.

$ caire -in <image_url> -out <output-folder>
$ curl -s <image_url> | caire > out.jpg

Process multiple images from a directory concurrently

The library can also process multiple images from a directory concurrently. You have to provide only the source and the destination folder and the new width or height in this case.

$ caire -in <input_folder> -out <output-folder>

Support for multiple output image type

There is no need to define the output file type, just use the correct extension and the library will encode the image to that specific type.

Other options

In case you wish to scale down the image by a specific percentage, it can be used the -perc boolean flag. In this case the values provided for the width and height are expressed in percentage and not pixel values. For example to reduce the image dimension by 20% both horizontally and vertically you can use the following command:

$ caire -in input/source.jpg -out ./out.jpg -perc=1 -width=20 -height=20 -debug=false

Also the library supports the -square option. When this option is used the image will be resized to a square, based on the shortest edge.

When an image is resized on both the X and Y axis, the algorithm will first try to rescale it prior resizing, but also will preserve the image aspect ratio. The seam carving algorithm is applied only to the remaining points. Ex. : given an image of dimensions 2048x1536 if we want to resize to the 1024x500, the tool first rescale the image to 1024x768 and then will remove only the remaining 268px.

Masks support:

• -mask: The path to the protective mask. The mask should be in binary format and have the same size as the input image. White areas represent regions where no seams should be carved.
• -rmask: The path to the removal mask. The mask should be in binary format and have the same size as the input image. White areas represent regions to be removed.

Mask	Mask removal

Caire integrations

• Caire can be used as a serverless function via OpenFaaS: https://github.com/esimov/caire-openfaas
• Caire can also be used as a snap function (https://snapcraft.io/caire): $ snap run caire --h

Results

Shrunk images

Original	Shrunk

Enlarged images

Original	Extended

Useful resources

• https://en.wikipedia.org/wiki/Seam_carving
• https://inst.eecs.berkeley.edu/~cs194-26/fa16/hw/proj4-seamcarving/imret.pdf
• http://pages.cs.wisc.edu/~moayad/cs766/download_files/alnammi_cs_766_final_report.pdf
• https://stacks.stanford.edu/file/druid:my512gb2187/Zargham_Nassirpour_Content_aware_image_resizing.pdf

Author

• Endre Simo (@simo_endre)

License

Copyright © 2018 Endre Simo

This project is under the MIT License. See the LICENSE file for the full license text.

Documentation

Overview

Package caire is a content aware image resize library, which can rescale the source image seamlessly both vertically and horizontally by eliminating the less important parts of the image.

The package provides a command line interface, supporting various flags for different types of rescaling operations. To check the supported commands type:

$ caire --help

In case you wish to integrate the API in a self constructed environment here is a simple example:

package main

import (
	"fmt"
	"github.com/esimov/caire"
)

func main() {
	p := &caire.Processor{
		// Initialize struct variables
	}

	if err := p.Process(in, out); err != nil {
		fmt.Printf("Error rescaling image: %s", err.Error())
	}
}

Index

• func Resize(s SeamCarver, img *image.NRGBA) (image.Image, error)
• type C
• type Carver
  • func NewCarver(width, height int) *Carver
  • func (c *Carver) AddSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA
  • func (c *Carver) ComputeSeams(p *Processor, img *image.NRGBA) (*image.NRGBA, error)
  • func (c *Carver) FindLowestEnergySeams(p *Processor) []Seam
  • func (c *Carver) RemoveSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA
  • func (c *Carver) SobelDetector(img *image.NRGBA, threshold float64) *image.NRGBA
  • func (c *Carver) StackBlur(img *image.NRGBA, radius uint32) *image.NRGBA
• type D
• type Gui
  • func NewGUI(width, height int) *Gui
  • func (g *Gui) AddHudControl(hudControlType hudControlType, title string, enabled bool)
  • func (g *Gui) DrawSeam(shape ShapeType, x, y, thickness float32)
  • func (g *Gui) Run() error
• type Image
• type Processor
  • func (p *Processor) Execute(img *Image)
  • func (p *Processor) Process(r io.Reader, w io.Writer) error
  • func (p *Processor) Resize(img *image.NRGBA) (image.Image, error)
• type Seam
• type SeamCarver
• type ShapeType

Functions

func Resize

func Resize(s SeamCarver, img *image.NRGBA) (image.Image, error)

Types

type C

type C = layout.Context

type Carver

type Carver struct {
	Points []float64
	Seams  []Seam
	Width  int
	Height int
}

Carver is the main entry struct having as parameters the newly generated image width, height and seam points.

func NewCarver

func NewCarver(width, height int) *Carver

NewCarver returns an initialized Carver structure.

func (*Carver) AddSeam

func (c *Carver) AddSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA

AddSeam add a new seam.

func (*Carver) ComputeSeams

func (c *Carver) ComputeSeams(p *Processor, img *image.NRGBA) (*image.NRGBA, error)

ComputeSeams compute the minimum energy level based on the following logic:

• traverse the image from the second row to the last row and compute the cumulative minimum energy M for all possible connected seams for each entry (i, j).

• the minimum energy level is calculated by summing up the current pixel value with the minimum pixel value of the neighboring pixels from the previous row.

func (*Carver) FindLowestEnergySeams

func (c *Carver) FindLowestEnergySeams(p *Processor) []Seam

FindLowestEnergySeams find the lowest vertical energy seam.

func (*Carver) RemoveSeam

func (c *Carver) RemoveSeam(img *image.NRGBA, seams []Seam, debug bool) *image.NRGBA

RemoveSeam remove the least important columns based on the stored energy (seams) level.

func (*Carver) SobelDetector

func (c *Carver) SobelDetector(img *image.NRGBA, threshold float64) *image.NRGBA

SobelDetector uses the sobel filter operator for detecting image edges. See https://en.wikipedia.org/wiki/Sobel_operator

func (*Carver) StackBlur

func (c *Carver) StackBlur(img *image.NRGBA, radius uint32) *image.NRGBA

StackBlur applies a blur filter to the provided image. The radius defines the bluring average.

type D

type D = layout.Dimensions

type Gui

type Gui struct {
	// contains filtered or unexported fields
}

Gui is the basic struct containing all of the information needed for the UI operation. It receives the resized image transferred through a channel which is called in a separate goroutine.

func NewGUI

func NewGUI(width, height int) *Gui

NewGUI initializes the Gio interface.

func (*Gui) AddHudControl

func (g *Gui) AddHudControl(hudControlType hudControlType, title string, enabled bool)

AddHudControl adds a new hud control for debugging.

func (*Gui) DrawSeam

func (g *Gui) DrawSeam(shape ShapeType, x, y, thickness float32)

DrawSeam visualizes the seam carver in action when the preview mode is activated. It receives as parameters the shape type, the seam (x,y) coordinates and it's thickness.

func (*Gui) Run

func (g *Gui) Run() error

Run is the core method of the Gio GUI application. This updates the window with the resized image received from a channel and terminates when the image resizing operation completes.

type Image

type Image struct {
	Src, Dst, PipeName string
	Workers            int
}

type Processor

type Processor struct {
	FaceAngle      float64
	SeamColor      string
	MaskPath       string
	RMaskPath      string
	ShapeType      ShapeType
	SobelThreshold int
	BlurRadius     int
	NewWidth       int
	NewHeight      int
	FaceDetector   *pigo.Pigo
	Spinner        *utils.Spinner
	Mask           *image.NRGBA
	RMask          *image.NRGBA
	DebugMask      *image.NRGBA
	Percentage     bool
	Square         bool
	Debug          bool
	Preview        bool
	FaceDetect     bool
	// contains filtered or unexported fields
}

Processor options

func (*Processor) Execute

func (p *Processor) Execute(img *Image)

Execute executes the image resizing process. In case the preview mode is activated it will be invoked in a separate goroutine in order to avoid blocking the main OS thread. Otherwise it will be called normally.

func (*Processor) Process

func (p *Processor) Process(r io.Reader, w io.Writer) error

Process encodes the resized image into an io.Writer interface. We are using the io package, since we can provide different input and output types, as long as they implement the io.Reader and io.Writer interface.

func (*Processor) Resize

func (p *Processor) Resize(img *image.NRGBA) (image.Image, error)

Carve is the main entry point for the image resize operation. The new image can be resized either horizontally or vertically (or both). Depending on the provided options the image can be either reduced or enlarged.

type Seam

type Seam struct {
	X int
	Y int
}

Seam struct contains the seam pixel coordinates.

type SeamCarver

type SeamCarver interface {
	Resize(*image.NRGBA) (image.Image, error)
}

SeamCarver defines the Carve interface method, which have to be implemented by the Processor struct.

type ShapeType

type ShapeType string

const (
	Circle ShapeType = "circle"
	Line   ShapeType = "line"
)