c4t

module
Version: v0.0.0-...-0d16973 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 10, 2021 License: MIT

README

c4t

c4t ('C4 tester') is the top-level testing framework for the 'C Compiler Concurrency Checker' (C4) project. It sits on top of c4f and herdtools7 and allows running of multi-compiler, multi-machine testing campaigns.

c4t is written in Go (making it fairly portable and cross-compilable, perhaps unlike c4f), and is free software under the MIT licence.

Components

The main entry point into c4t is c4t, which sets up and executes multiple parallel testing cycles for fully automated use. Many parts of c4t's testing infrastructure, as well as other utilities, are also available in separate binaries.

The main object of work in c4t is a test plan, a bulky (and often gzipped) JSON file that centralises information about a testing campaign for one machine. Many of its components operate on one or more test plans, either implicitly or explicitly.

At time of writing, c4t has the following components:

The main test cycle
  • c4t-plan, which reads the tester config and creates an initial test plan over some Litmus tests and compilers for a single machine;
  • c4t-perturb, which perturbs a plan configuration, choosing a random sample of plans as well as randomised parameters to the compiler;
  • c4t-fuzz, which runs c4f over a test plan to create a more useful plan;
  • c4t-lift, which runs backends like litmus7 over a test plan to lift subjects to harnesses;
  • c4t-invoke (on the machine running c4t) and c4t-mach (on the target machine), which communicate with each other through SSH and perform the compilation and running phases of a test plan;
  • c4t, which combines the above into a looping test campaign over multiple machines.
Analysing things
  • c4t-analyse, which performs some basic analysis over a test plan and prints reports on failures, compiler warnings, etc.;
  • c4t-obs, which parses and pretty-prints information from backend observation JSON records (such as those produced by c4t-backend and nested inside plan files).
Utilities
  • c4t-backend, for running backends separately from test cycles;
  • c4t-coverage, which produces coverage testbeds (work in progress);
  • c4t-gccnt (GCCn't), a wrapper over gcc that can inject compiler failures when certain parameters are triggered (useful for testing that the workflow handles such issues);
  • c4t-setc, which overrides compiler parameters in an existing plan (useful for exploring particular optimisation levels).

Use

Note that c4t is still pretty rough around the edges - please feel free to file issues about its user experience and documentation.

  • Install using the usual go tools: for example, go get github.com/c4-project/c4t/cmd/.... All commands are in the cmd directory.
  • Make sure that the c4f tools are in PATH on the test-running machine (eg run make install)
  • Make sure that at least c4t-mach is installed on any remote machine you wish to use for testing.
  • Create a tester.toml file in UserConfigDir/c4t (see tester-example.toml).
  • The easiest way to check if c4t is working is c4t path/to/c4f/examples/c_litmus/memalloy.

Acknowledgements

c4t is part of work carried out under the umbrella of the IRIS project.

Directories

Path Synopsis
cmd
c4t
internal
app/config
Package config implements the c4f-config app.
Package config implements the c4f-config app.
app/coverage
Package coverage contains the app definition for c4t-coverage.
Package coverage contains the app definition for c4t-coverage.
app/director
Package director contains the app definition for c4t ('the director').
Package director contains the app definition for c4t ('the director').
app/fuzz
Package fuzz contains the app definition for c4t-fuzz.
Package fuzz contains the app definition for c4t-fuzz.
app/gccnt
Package gccnt contains the app definition for c4t-gccnt.
Package gccnt contains the app definition for c4t-gccnt.
app/invoke
Package invoker contains the app definition for c4t-invoke.
Package invoker contains the app definition for c4t-invoke.
app/lift
Package lift contains the app definition for c4t-lift.
Package lift contains the app definition for c4t-lift.
app/mach
Package mach contains the app definition for c4t-mach.
Package mach contains the app definition for c4t-mach.
app/mkdoc
Package mkdoc contains the app definition for c4t-mkdoc.
Package mkdoc contains the app definition for c4t-mkdoc.
app/perturb
Package perturb contains the app definition for c4t-perturb.
Package perturb contains the app definition for c4t-perturb.
app/plan
Package plan contains the app definition for c4t-plan.
Package plan contains the app definition for c4t-plan.
app/setc
Package setc contains the app definition for c4t-setc.
Package setc contains the app definition for c4t-setc.
c4f
Package c4f contains wrappers for the external c4f programs.
Package c4f contains wrappers for the external c4f programs.
config
Package config describes the top-level tester configuration.
Package config describes the top-level tester configuration.
config/pretty
Package pretty contains pretty-printers for configuration.
Package pretty contains pretty-printers for configuration.
copier
Package copier provides a mockable interface for network-transparent(ish) file copying, and implementations thereof.
Package copier provides a mockable interface for network-transparent(ish) file copying, and implementations thereof.
coverage
Package coverage contains helpers for building compiler coverage reports.
Package coverage contains helpers for building compiler coverage reports.
director
Package director contains the top-level C4 test director, which manages a full testing campaign.
Package director contains the top-level C4 test director, which manages a full testing campaign.
director/pathset
Package pathset contains the various path-sets for the director.
Package pathset contains the various path-sets for the director.
helper/errhelp
Package errhelp contains error helper functions.
Package errhelp contains error helper functions.
helper/srvrun
Package srvrun contains various ways to run, or pretend to run, services.
Package srvrun contains various ways to run, or pretend to run, services.
helper/stringhelp
Package stringhelp contains various helpers for strings and collections thereof.
Package stringhelp contains various helpers for strings and collections thereof.
helper/testhelp
Package testhelp contains test helpers.
Package testhelp contains test helpers.
id
Package id describes C4's dot-delimited IDs.
Package id describes C4's dot-delimited IDs.
machine
Package machine contains models for compiler-bearing machines.
Package machine contains models for compiler-bearing machines.
model/filekind
Package filekind contains types for dealing with the various 'kinds' of file present in a subject.
Package filekind contains types for dealing with the various 'kinds' of file present in a subject.
model/litmus
Package litmus contains model structs and associated functions for Litmus test entries.
Package litmus contains model structs and associated functions for Litmus test entries.
model/recipe
Package recipe contains 'recipes': instructions to the machine node on how to compile a subject.
Package recipe contains 'recipes': instructions to the machine node on how to compile a subject.
model/service
Package service contains models for backends and compilers (external services, hence the name).
Package service contains models for backends and compilers (external services, hence the name).
model/service/backend
Package backend contains style-to-backend resolution.
Package backend contains style-to-backend resolution.
model/service/compiler
Package compiler contains types for compilers, which are a particular type of service.
Package compiler contains types for compilers, which are a particular type of service.
model/service/compiler/optlevel
Package optlevel contains types that capture information about compiler optimisation levels.
Package optlevel contains types that capture information about compiler optimisation levels.
model/service/fuzzer
Package fuzzer considers fuzzers as services.
Package fuzzer considers fuzzers as services.
mutation
Package mutation contains support for mutation testing using c4t.
Package mutation contains support for mutation testing using c4t.
observing
Package observing contains plumbing for the tester's implementation of the observer pattern.
Package observing contains plumbing for the tester's implementation of the observer pattern.
plan
Package plan contains the Plan type, as well as various parts of plans that don't warrant their own packages.
Package plan contains the Plan type, as well as various parts of plans that don't warrant their own packages.
plan/analysis
Package analysis handles analysing a Corpus and filing its subjects into categorised sub-corpora.
Package analysis handles analysing a Corpus and filing its subjects into categorised sub-corpora.
plan/stage
Package stage contains plan metadata describing which stages of a test cycle have occurred.
Package stage contains plan metadata describing which stages of a test cycle have occurred.
quantity
Package quantity contains the quantity sets for various parts of the tester.
Package quantity contains the quantity sets for various parts of the tester.
remote
Package remote provides helpers for dealing with remote test operations over SSH and SCP.
Package remote provides helpers for dealing with remote test operations over SSH and SCP.
serviceimpl
Package serviceimpl contains implementations of compiler and backend services, as well as hard-coded logic for resolving them from style IDs.
Package serviceimpl contains implementations of compiler and backend services, as well as hard-coded logic for resolving them from style IDs.
serviceimpl/backend/delitmus
Package delitmus lifts the c4f delitmusifier into a backend.
Package delitmus lifts the c4f delitmusifier into a backend.
serviceimpl/backend/herdstyle
Package herdstyle contains backends that c4f in a similar way to the Herd memory simulator.
Package herdstyle contains backends that c4f in a similar way to the Herd memory simulator.
serviceimpl/backend/herdstyle/herd
Package herd contains the parts of a Herdtools backend specific to herd7.
Package herd contains the parts of a Herdtools backend specific to herd7.
serviceimpl/backend/herdstyle/litmus
Package litmus contains the parts of a Herdtools backend specific to litmus7.
Package litmus contains the parts of a Herdtools backend specific to litmus7.
serviceimpl/backend/herdstyle/parser
Package parser contains logic for parsing Herd-style observations.
Package parser contains logic for parsing Herd-style observations.
serviceimpl/backend/herdstyle/rmem
Package rmem implements rudimentary backend support for RMEM.
Package rmem implements rudimentary backend support for RMEM.
serviceimpl/compiler
Package compiler contains style-to-compiler resolution.
Package compiler contains style-to-compiler resolution.
stage/analyser
Package analyser represents the stage of the tester that takes a plan, performs various statistics on it, and outputs reports.
Package analyser represents the stage of the tester that takes a plan, performs various statistics on it, and outputs reports.
stage/analyser/csvdump
Package csvdump handles outputting of analysis data as CSVs.
Package csvdump handles outputting of analysis data as CSVs.
stage/analyser/pretty
Package pretty provides a pretty-printer for analyses.
Package pretty provides a pretty-printer for analyses.
stage/analyser/saver
Package saver contains the part of the analyser that uses the analyser to save failing tests.
Package saver contains the part of the analyser that uses the analyser to save failing tests.
stage/fuzzer
Package fuzzer contains a test-plan batch fuzzer.
Package fuzzer contains a test-plan batch fuzzer.
stage/invoker
Package invoker handles invoking the 'mach' stage in a network-transparent manner with the c4t-mach binary.
Package invoker handles invoking the 'mach' stage in a network-transparent manner with the c4t-mach binary.
stage/invoker/runner
Package runner contains low-level code for running the machine node via SSH and locally.
Package runner contains low-level code for running the machine node via SSH and locally.
stage/lifter
Package lifter contains the part of the tester framework that lifts litmus tests to compilable C.
Package lifter contains the part of the tester framework that lifts litmus tests to compilable C.
stage/mach
Package mach contains the 'machine-dependent' stage of the tester.
Package mach contains the 'machine-dependent' stage of the tester.
stage/mach/compiler
Package compiler contains a test-plan batch compiler.
Package compiler contains a test-plan batch compiler.
stage/mach/forward
Package forward describes the JSON-based protocol used to 'forward' messages and errors from a machine node to its invoker, potentially over SSH.
Package forward describes the JSON-based protocol used to 'forward' messages and errors from a machine node to its invoker, potentially over SSH.
stage/mach/interpreter
Package interpreter contains the recipe interpreter for the machine node.
Package interpreter contains the recipe interpreter for the machine node.
stage/mach/observer
Package observer contains interfaces and types for machine node observers.
Package observer contains interfaces and types for machine node observers.
stage/mach/runner
Package runner contains the part of c4t that runs compiled test binaries and interprets their output.
Package runner contains the part of c4t that runs compiled test binaries and interprets their output.
stage/perturber
Package perturber contains the logic for the test perturber.
Package perturber contains the logic for the test perturber.
stage/planner
Package planner contains the logic for the test planner.
Package planner contains the logic for the test planner.
stat
Package stat implements c4t's persistent statistics support.
Package stat implements c4t's persistent statistics support.
subject/compilation
Package compilation contains types and functions for dealing with 'compilations': subject/compiler products.
Package compilation contains types and functions for dealing with 'compilations': subject/compiler products.
subject/corpus
Package corpus concerns test corpora (collections of named subjects).
Package corpus concerns test corpora (collections of named subjects).
subject/corpus/builder
Package builder describes a set of types and methods for building corpora asynchronously.
Package builder describes a set of types and methods for building corpora asynchronously.
subject/normaliser
Package normaliser provides utilities for archiving and transferring plans, corpora, and subjects.
Package normaliser provides utilities for archiving and transferring plans, corpora, and subjects.
subject/normpath
Package normpath contains various 'normalised path' fragments, and functions for constructing them.
Package normpath contains various 'normalised path' fragments, and functions for constructing them.
subject/obs
Package obs concerns 'observations': the end result of running a test on a particular machine.
Package obs concerns 'observations': the end result of running a test on a particular machine.
tabulator
Package tabulator provides a generic interface for tabulating data (to CSV, to tabwriter, etc).
Package tabulator provides a generic interface for tabulating data (to CSV, to tabwriter, etc).
timing
Package timing contains helpers for tracking and recording timings.
Package timing contains helpers for tracking and recording timings.
tool/gccnt
Package gccnt implements the "GCCn't" wrapper over GCC-style compilers.
Package gccnt implements the "GCCn't" wrapper over GCC-style compilers.
ux
Package ux contains functions and types for human-readable views and observers on parts of c4t, as well as various common 'user experience' functions such as standard flag parsers.
Package ux contains functions and types for human-readable views and observers on parts of c4t, as well as various common 'user experience' functions such as standard flag parsers.
ux/dash
Package dash contains the c4t console dashboard.
Package dash contains the c4t console dashboard.
ux/directorobs
Package directorobs contains a pre-packaged observer set for the test director.
Package directorobs contains a pre-packaged observer set for the test director.
ux/singleobs
Package singleobs contains observer implementations for use in the 'single-shot' c4t commands.
Package singleobs contains observer implementations for use in the 'single-shot' c4t commands.
ux/stdflag
Package stdflag contains standard command-line flag code for c4t binaries.
Package stdflag contains standard command-line flag code for c4t binaries.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL