yapscan

README

yapscan

Yapscan is a YAra based Process SCANner, aimed at giving more control about what to scan and giving detailed reports on matches.

Features

You can use yapscan to selectively scan the memory of running processes as well as files in local hard drives and/or mounted shares. The most notable differences to stock yara are (see section Usage),

• Supports loading yara-rules from an encrypted zip file to prevent anti-virus software from detecting rules as malicious.
• Multiple yara rules can also be loaded recursively from a directory.
• Can suspend processes to be scanned (use with care, may crash your system).
• Allows for filtering of memory segments to be scanned based on size, type (image, mapped, private), state (commit, free, reserve), and permissions.
• Allows for easy scanning of all running processes, local drives and/or mounted shares.
• Comes with extensive reporting features to allow later analysis of efficacy of rules.
• Matched memory segments can even be automatically dumped and stored as part of the

Other quality-of-life features include

• Statically built, dependency free exe for Windows
• Listing running processes
• Listing and dumping memory segments of a specific process
• Compiling yara rules and compressing them into an encrypted zip.
• Provides an "executable DLL" for locked down environments such as VDIs (see section Executable DLL)

Yapscan comes with support for both Windows and Linux, however Windows is the primary and most thoroughly tested target OS.

Usage

I'll write a proper man page soon. For now, use yapscan --help and yapscan <command> --help for usage information.

COMMANDS:
   list-processes, ps, lsproc  lists all running processes
   list-process-memory, lsmem  lists all memory segments of a process
   dump                        dumps memory of a process
   scan                        scans processes or paths with yara rules
   zip-rules                   creates an encrypted zip containing compiled yara rules
   join                        joins dumps with padding
   help, h                     Shows a list of commands or help for one command

> yapscan scan --help
NAME:
   yapscan scan - scans processes or paths with yara rules

USAGE:
   yapscan scan [command options] [pid/path...]

OPTIONS:
   --rules value, -r value, -C value                   path to yara rules file or directory, if it's a file it can be a yara rules file or a zip containing a rules file encrypted with password "infected"
   --rules-recurse, --recurse-rules, --rr              if --rules specifies a directory, compile rules recursively (default: false)
   --all-processes, --all-p                            scan all running processes (default: false)
   --all-drives, --all-d                               scan all files in all local drives, implies --recurse (default: false)
   --all-shares, --all-s                               scan all files in all mounted net-shares, implies --recurse (default: false)
   --file-extensions value, -e value                   list of file extensions to scan, use special extension "-" as no extension, use --file-extensions "" to allow any (default: "-", "so", "exe", "dll", "sys")
   --threads value, -t value                           number of threads (goroutines) used for scanning files (default: 12)
   --full-report                                       create a full report (default: false)
   --store-dumps                                       store dumps of memory regions that match rules, implies --full-report, the report will be encrypted with --password (default: false)
   --keep                                              keep the temporary report directory, by default it will be deleted; ignored without --full-report (default: false)
   --password value                                    the password of the encrypted report, ignored unless --store-dumps is set (default: "infected")
   --filter-permissions value, --f-perm value          only consider segments with the given permissions or more, examples: "rw" includes segments with rw, rc and rwx
   --filter-permissions-exact value, --f-perm-e value  comma separated list of permissions to be considered, supported permissions: r, rw, rc, rwx, rcx
   --filter-type value, --f-type value                 comma separated list of considered types, supported types: image, mapped, private
   --filter-state value, --f-state value               comma separated list of considered states, supported states: free, commit, reserve (default: "commit")
   --filter-size-max value, --f-size-max value         maximum size of memory segments to be considered, can be absolute (e.g. "1.5GB"), percentage of total RAM (e.g. "10%T") or percentage of free RAM (e.g. "10%F") (default: "10%F")
   --filter-size-min value, --f-size-min value         minimum size of memory segments to be considered
   --suspend, -s                                       suspend the process before reading its memory (default: false)
   --force, -f                                         don't ask before suspending a process (default: false)
   --help, -h                                          show help (default: false)

Here are some additional example usages

# Create rules zip (optional)
yapscan zip-rules --output rules.zip rules.yara

# Scan a process with PID 423 with default filters
yapscan scan -r rules.zip 423
# Scan all processes with default filters
yapscan scan -r rules.zip --all-processes
# Scan all processes and all local drives with default filters
yapscan scan -r rules.zip --all-processes --all-drives
# Scan everything with default filters
yapscan scan -r rules.zip --all-processes --all-drives --all-shares

# Only scan memory segments with execute permission or more
yapscan scan -r rules.zip --filter-permissions x --all-processes
# Only scan memory segments with exactly read and execute (not write)
yapscan scan -r rules.zip --filter-permissions-exact rx --all-processes

# Enable logging, reporting and auto dumping of matched segments
yapscan --log-level debug --log-path yapscan.log scan -r rules.zip --full-report --store-dumps --all-processes

Executable DLL

The DLL built by this project is not a usual DLL, meant for importing functions from. Instead it acts similarly to the exe with two exported, high-level entry points:

// start acts same as you would expect a main function to act.
// It assumes a terminal with stdout/stderr and stdin has already
// been allocated.
extern int start(int argc, char** argv);

// run is meant for use with rundll32.
// It opens a new console window via AllocConsole(), then parses the
// lpCmdLine to extract the arguments and calls starts yapscan
// with the extracted arguments. 
extern void run(HWND hWnd, HINSTANCE hInst, LPTSTR lpCmdLine, int nCmdShow);

Some environments like VDIs (Virtual Desktop Infrastructure) may prevent the execution of arbitrary exe-files but still allows for use of arbitrary DLLs. If you gain access to a command line terminal in such an environment you can call yapscan via the built DLL like so.

rundll32.exe yapscan.dll,run scan -r rules.zip --all-processes

NOTE: This feature is still experimental! There very likely are quirks with the argument parsing.

State of this project

BETA, FeatureFreeze

Right now yapscan is in a beta period. I am currently working on a stable 1.0 release. This release will have a defined and documented stable API and cli interface that will remain backwards compatible for all 1.x versions. Before releasing 1.0 I also want to have at least a reasonable amount of test-coverage.

Feel free to use and test it and open issues for any bugs you may find. If you would like to have some additional features you can also open an issue with a feature request, but until the 1.0 release I will not be working on or merging any new features.

Contributing

Found a bug or want a feature, but you can't code or don't have the time? Feel free to open an issue! Please look for duplicates first.

If you want to contribute code, be it fixes or features, please open a pull request on the develop branch and mention any related issues.

In the rapid-development phase I have only used the master branch, but will switch very shortly to the git-flow workflow. This means the master branch will represent the latest stable release at any point in time and any work-in-progress is to be merged into the develop branch.

Scanning Technique

The actual scanning is left to the yara library. In case of file scanning, the high level yara library function yr_rules_scan_file is used. This function memory-maps the given file. Scanning process memory, on the other hand, is done on a lower level. Yapscan copies one memory segment at a time into a buffer in its own memory and then uses yr_rules_scan_mem in order to scan this buffer.

Building Yapscan

To build natively on Linux, for Linux you need install Go and the yara library. Once you have installed the dependencies it's as easy as:

# Install Golang and libyara
git clone https://github.com/fkie-cad/yapscan
cd yapscan/cmd/yapscan
go build

If you want to build on Linux for Windows, all you need installed is docker.

# Install docker
git clone https://github.com/fkie-cad/yapscan
cd yapscan/cicd/
./crossBuildForWindows.sh

The resulting binaries will be placed in cicd/build/.

Building natively on Windows, using MSYS2 follow these instructions

1. Install Go
2. Install MSYS2 and follow the first steps on the MSYS2 Website of updating via pacman.
3. Install build dependencies pacman --needed -S base-devel git autoconf automake libtool mingw-w64-{x86_64,i686}-{gcc,make,pkgconf}
4. Open PowerShell in the cicd/ directory and execute .\buildOnWindows.ps1 -MsysPath <msys_path> -BuildDeps where <msys_path> is the install directory for MSYS2, default is C:\msys64. NOTE: You'll have to press Enter on the MSYS window, once the dependencies are finished.
5. Enjoy the built files in cicd/build/

If you want to run tests on Windows, you have to run .\cicd\buildOnWindows.ps1 -BuildDeps only once. Then you open PowerShell and execute .\cicd\enableMingw.ps1 -MsysPath <msys_path> to set the appropriate environment variables. Now it's as easy as go test -tags yara_static ./.... The -tags yara_static is necessary if you use the build scripts, as they do not install any windows DLLs but only the static libraries.

NOTE: You might get inexplicable failures with -race on Windows with golagn >=1.14. According to the golang release notes for 1.14, the new pointer arithmetic checks are somewhat overzealous on Windows. In Golang v1.15 it seems this may not have been fixed, but the checks are enabled automatically. You can deactivate them like this go test -tags yara_static -race -gcflags=all=-d=checkptr=0 ./....

You don't have to rely on the powershell/bash scripts, but they are intended to make things as easy as possible at the cost of control over the compilation. If you want more control, take a look at the scripts use and modify them or execute the commands individually. The scripts perform the following tasks.

1. Start "MSYS2 MinGW 64-bit"
   1. Download OpenSSL from github
   2. Static-Build OpenSSL and install the development files
   3. Download libyara from github
   4. Static-Build libyara and install it
2. Set some environment variables in powershell, to allow the use of the mingw toolchain
3. Call the go build command with the appropriate build tag for static builds

Thanks to @hillu (author of go-yara), for pointing me in the right direction for building natively on windows. See #7 and the links therein if you want some more details.

Documentation

Overview

Package yapscan provides high-level features on top of the popular virus-scanner yara.

Index

• Constants
• Variables
• func AddressesFromMatches(matches []yara.MatchString, offset uint64) []uint64
• func FormatSlice(format string, slice interface{}, args ...interface{}) []string
• func IsYaraRulesFile(name string) bool
• func Join(parts []string, defaultGlue, finalGlue string) string
• func LoadYaraRules(path string, recurseIfDir bool) (*yara.Rules, error)
• type FilterMatch
• type MemoryScanProgress
• type MemoryScanner
• type MemorySegmentFilter
  • func NewAndFilter(filters ...MemorySegmentFilter) MemorySegmentFilter
  • func NewFilterFromFunc(filter MemorySegmentFilterFunc, parameter interface{}, reasonTemplate string) MemorySegmentFilter
  • func NewMaxSizeFilter(size uintptr) MemorySegmentFilter
  • func NewMinSizeFilter(size uintptr) MemorySegmentFilter
  • func NewPermissionsFilter(perm procio.Permissions) MemorySegmentFilter
  • func NewPermissionsFilterExact(perms []procio.Permissions) MemorySegmentFilter
  • func NewStateFilter(states []procio.State) MemorySegmentFilter
  • func NewTypeFilter(types []procio.Type) MemorySegmentFilter
• type MemorySegmentFilterFunc
• type ProcessScanner
  • func NewProcessScanner(proc procio.Process, filter MemorySegmentFilter, scanner MemoryScanner) *ProcessScanner
  • func (s *ProcessScanner) Scan() (<-chan *MemoryScanProgress, error)
• type Rules
• type YaraScanner
  • func NewYaraScanner(rules Rules) (*YaraScanner, error)
  • func (s *YaraScanner) ScanFile(filename string) ([]yara.MatchRule, error)
  • func (s *YaraScanner) ScanMem(buf []byte) ([]yara.MatchRule, error)

Examples

• Join

Constants

const RulesZIPPassword = "infected"

RulesZIPPassword is the password yapscan uses to de-/encrypt the rules zip file.

Variables

var DefaultYaraRulesNamespace = ""

DefaultYaraRulesNamespace is the default namespace when compiling rules.

var ErrSkipped = errors.New("skipped")

ErrSkipped is returned, when a memory segment is skipped due to the applied filter.

var YaraRulesFileExtensions = []string{
	".yar",
	".yara",
}

YaraRulesFileExtensions are the file extensions yapscan expects rules files to have. This is used when loading files from a directory.

Functions

func AddressesFromMatches

func AddressesFromMatches(matches []yara.MatchString, offset uint64) []uint64

AddressesFromMatches returns one value for each given yara.MatchString. The returned values are equal to the given offset plus the Offset field of each yara.MatchString.

func FormatSlice

func FormatSlice(format string, slice interface{}, args ...interface{}) []string

FormatSlice calls fmt.Sprintf(format, element, args...) for each element in the given slice. The returned string slice contains the formatted output.

func IsYaraRulesFile

func IsYaraRulesFile(name string) bool

IsYaraRulesFile returns true, if the given filename has one of the extensions in YaraRulesFileExtensions.

func Join

func Join(parts []string, defaultGlue, finalGlue string) string

Join joins all elements of a string slice, using the defaultGlue for all but the last two elements.

Example

parts := []string{"life", "the universe", "everything"}
fmt.Println(Join(parts, ", ", " and "))

Output:

life, the universe and everything

func LoadYaraRules

func LoadYaraRules(path string, recurseIfDir bool) (*yara.Rules, error)

LoadYaraRules loads yara.Rules from a file (or files) and compiles if necessary. The given path can be a path to a directory, a compiled rules-file, a plain text file containing rules, or an encrypted zip file containing rules.

If the path is a directory, all files with one of the file extensions in YaraRulesFileExtensions are loaded (recursively if recurseIfDir is true). All files are assumed to be uncompiled and will be compiled. Loading multiple already compiled files into one yara.Rules object is not supported. Each file will be compiled with the namespace equal to its filename, relative to the given path.

If the path is a single file, it may be compiled, uncompiled or a zip file. An uncompiled file will be compiled with the namespace `DefaultYaraRulesNamespace+"/"+filename`. A zip file will be opened and decrypted with the RulesZIPPassword. The contents of the zip file will be treated similar to the way a directory is treated (see above), however *all* files are assumed to be rules-files, recursion is always enabled and there may be either a single compiled file or arbitrarily many uncompiled files in the zip.

Types

type FilterMatch

type FilterMatch struct {
	Result bool
	MSI    *procio.MemorySegmentInfo
	Reason string // Reason for filter mismatch, if Result is false
}

FilterMatch contains information about the matching of a MemorySegmentFilter.

type MemoryScanProgress

type MemoryScanProgress struct {
	// Process contains information about the process being scanned.
	Process procio.Process
	// MemorySegment contains information about the specific memory segment which was just scanned.
	MemorySegment *procio.MemorySegmentInfo
	// Dump contains the raw contents of the memory segment.
	Dump []byte
	// Matches contains the yara.MatchRule results.
	Matches []yara.MatchRule
	// Error contains the encountered error or nil, if no error was encountered.
	Error error
}

MemoryScanProgress contains all information, generated during scanning.

type MemoryScanner

type MemoryScanner interface {
	ScanMem(buf []byte) (results []yara.MatchRule, err error)
}

MemoryScanner is a yara.Rules compatible interface, defining the subset of functions required for scanning memory buffers.

type MemorySegmentFilter

type MemorySegmentFilter interface {
	Filter(info *procio.MemorySegmentInfo) *FilterMatch
}

MemorySegmentFilter describes an interface, capable of filtering *procio.MemorySegmentInfo instances.

func NewAndFilter

func NewAndFilter(filters ...MemorySegmentFilter) MemorySegmentFilter

NewAndFilter creates a new filter, which is the logical AND-combination of all given MemorySegmentFilter instances.

func NewFilterFromFunc

func NewFilterFromFunc(filter MemorySegmentFilterFunc, parameter interface{}, reasonTemplate string) MemorySegmentFilter

NewFilterFromFunc creates a new filter from a given MemorySegmentFilterFunc.

func NewMaxSizeFilter

func NewMaxSizeFilter(size uintptr) MemorySegmentFilter

NewMaxSizeFilter creates a new filter, matching *procio.MemorySegmentInfo with the given maximum size.

func NewMinSizeFilter

func NewMinSizeFilter(size uintptr) MemorySegmentFilter

NewMinSizeFilter creates a new filter, matching *procio.MemorySegmentInfo with the given minimum size.

func NewPermissionsFilter

func NewPermissionsFilter(perm procio.Permissions) MemorySegmentFilter

NewPermissionsFilter creates a new filter, matching *procio.MemorySegmentInfo with procio.Permissions equal to or more permissive than the given perm.

func NewPermissionsFilterExact

func NewPermissionsFilterExact(perms []procio.Permissions) MemorySegmentFilter

NewPermissionsFilterExact creates a new filter, matching *procio.MemorySegmentInfo with procio.Permissions exactly equal to one of the given perms.

func NewStateFilter

func NewStateFilter(states []procio.State) MemorySegmentFilter

NewStateFilter creates a new filter, matching *procio.MemorySegmentInfo with a procio.State equal to one of the given states.

func NewTypeFilter

func NewTypeFilter(types []procio.Type) MemorySegmentFilter

NewTypeFilter creates a new filter, matching *procio.MemorySegmentInfo with a procio.Type equal to one of the given types.

type MemorySegmentFilterFunc

type MemorySegmentFilterFunc func(info *procio.MemorySegmentInfo) bool

MemorySegmentFilterFunc is a callback, used to filter *procio.MemorySegmentInfo instances.

type ProcessScanner

type ProcessScanner struct {
	// contains filtered or unexported fields
}

ProcessScanner implements scanning of memory segments, allocated by a process. This scanning is done using an underlying MemoryScanner on segments, matching a MemorySegmentFilter.

func NewProcessScanner

func NewProcessScanner(proc procio.Process, filter MemorySegmentFilter, scanner MemoryScanner) *ProcessScanner

NewProcessScanner create a new ProcessScanner with for the given procio.Process. It uses the given MemoryScanner in order to scan memory segments of the process, which match the given MemoryScanner.

func (*ProcessScanner) Scan

func (s *ProcessScanner) Scan() (<-chan *MemoryScanProgress, error)

Scan starts an asynchronous scan. The returned unbuffered channel will yield MemoryScanProgress instances every time a memory segment has been processed. The channel will be closed when all segments have been processed.

type Rules

type Rules interface {
	ScanFile(filename string, flags yara.ScanFlags, timeout time.Duration, cb yara.ScanCallback) (err error)
	ScanMem(buf []byte, flags yara.ScanFlags, timeout time.Duration, cb yara.ScanCallback) (err error)
}

Rules are a yara.Rules compatible interface, defining the functions required by yapscan. The choice of an interface over the concrete struct yara.Rules is mostly to make testing easier.

type YaraScanner

type YaraScanner struct {
	// contains filtered or unexported fields
}

YaraScanner is a wrapper for yara.Rules, with a more go-like interface.

func NewYaraScanner

func NewYaraScanner(rules Rules) (*YaraScanner, error)

NewYaraScanner creates a new YaraScanner from the given yara.Rules.

func (*YaraScanner) ScanFile

func (s *YaraScanner) ScanFile(filename string) ([]yara.MatchRule, error)

ScanFile scans the file with the given filename. This function simply calls ScanFile on the underlying yara.Rules object.

func (*YaraScanner) ScanMem

func (s *YaraScanner) ScanMem(buf []byte) ([]yara.MatchRule, error)

ScanMem scans the given buffer. This function simply calls ScanMem on the underlying yara.Rules object.