Redo implementation in Bourne Shell

I have implemented the build system redo as designed by DJB in Bourne Shell. To understand how redo can be simpler, more flexible, more powerful and more reliable than make, read Introduction to redo and/or redo: a top-down software build system. The current version of redo is `$(redo-ifchange ./redo; ./redo --version 2>&1 | head -n1)`.

redo
  the main program
redo-always
  marks the current target as always needing to be rebuilt
redo-dot
  prints redo dependency graph in DOT format (example output, example rendering)
redo-ifchange
  adds dependencies for the current target (if a dependency changes, the target will be rebuilt)
redo-ifcreate
  adds non-existence dependencies for the current target (if a non-existence dependency is created, the target will be rebuilt)
redo-ood
  prints a list of all target files that are out of date
redo-targets
  prints a list of all target files that exist (a target is a file redo can build)
redo-sources
  prints a list of all source files that exist (a source is a dependency that is not a target)
redo-stamp
  detects if the current target has changed (see apenwarr's documentation)
redo-whichdo
  prints search paths for dofiles to build a redo target

# Installation

To install redo, copy the redo executables and man pages (detached PGP signature) to a directory in your \$PATH. For example, if `/usr/local/bin` exists, is listed in your \$PATH and can be written to by the current user, the following commands will install redo:

`

    wget http://news.dieweltistgarnichtso.net/bin/archives/redo-sh.tar.gz
    tar -moxzf redo-sh.tar.gz -C /usr/local

`

This website is hosted in Germany and German police is legally allowed to spread malware. I therefore strongly recommend to verify the PGP signature before installation to ensure that the redo release archive has not been tampered with. If you have gpg installed and trust my PGP signing key, the following commands will install redo only if the signature matches:

`

    wget http://news.dieweltistgarnichtso.net/bin/archives/redo-sh.tar.gz http://news.dieweltistgarnichtso.net/bin/archives/redo-sh.tar.gz.sig
    gpg --verify redo-sh.tar.gz.sig redo-sh.tar.gz && tar -moxzf redo-sh.tar.gz -C /usr/local

`

This implementation of redo depends only on GNU Core Utilities or BusyBox.

# Frequently Asked Questions

# How do I build a file with redo?

To build a file `target`, you have to create a dofile `target.do` with shell commands that write the intended result of the build either to standard output or to its parameter \$3. Then run `redo target`.

The default target is `all`: `redo` without arguments executes commands from `all.do`.

# What do the redo parameters \$1, \$2, \$3 mean?

When redo runs a dofile, it gives it three parameters:

\$1 | filename of target
\$2 | basename of target, without extension if default dofile is used
\$3 | filename of temporary output file that is renamed on build success

# How does the extension removal in parameter \$2 work?

Redo removes the extension that it gets from the default dofile filename:

Target | Dofile | Parameter \$2
--- | --- | ---
`a.b.c` | `a.b.c.do` | a.b.c
`a.b.c` | `default.do` | a.b.c
`a.b.c` | `default.c.do` | a.b
`a.b.c` | `default.b.c.do` | a

# How do I declare dependencies with redo?

Use the redo-ifchange command in a dofile: `redo-ifchange dependency` inside `target.do` means: If the `target` is built, the `dependency` is built if it does not exist and recorded as a dependency. On subsequent builds, if `dependency` does not exist or has changed since the last build, both `dependency` and `target` are rebuilt.

# How does this redo implementation check dependencies?

For dependency checking, this implementation of redo checks the dependencies' ctime against the stored ctime. If the ctime differs, it checks the dependencies' md5sum against the stored md5sum. This is arguably more useful than just using ctime.

# How do I declare non-existence dependencies with redo?

Use the redo-ifcreate command in a dofile: `redo-ifcreate ne_dependency` inside `target.do` means: If `target` is built, the non-existing file `ne_dependency` is recorded as a non-existence dependency. If `ne_dependency` exists on subsequent builds, `target` is rebuilt.

# What are command line options for this redo implementation?

Short option | Long option | Effect
--- | --- | ---
-d | --debug | print dependency checks as they happen
| --debug-jobs | print messages about job management
| --debug-locks | print messages about file locking
-h | --help | print usage instructions and exit
-j [n] | --jobs [n] | execute at most [n] dofiles in parallel
| --version | print version information and exit
-x | --xtrace | print commands as they are executed (variables expanded)

# How can I use this redo implementation to build in parallel?

If the `-j` or `--jobs` option is given followed by an integer, an invocation of redo or redo-ifchange with multiple arguments builds targets in parallel. The integer argument specifies the maximum number of targets allowed to be built in parallel. Targets are locked, so if targets built in parallel share a dependency, redo only builds it once, unless the dependency dofile uses redo-always to rebuild every time, or the dependency becomes out of date during a build in some other way.

# How can I make redo build everything in a different directory?

You can use union mounts to combine multiple directories. For example, on Linux with unionfs-fuse, the command `unionfs -o cow 'output=RW:source=RO' /tmp/build` combines the `output` and `source` directories so that `/tmp/build` contains all files from the `source` directory, but any newly created file will show up only in the `output` directory.

# Why can a target be built twice during a run?

Some build processes require several builds of the same target. Naïve implementors of build systems disregard this possibility and do not check the validity of a target again if it was built during a run.

- A target that has a dofile that invokes redo-always will be rebuilt several more times, if several targets invoke redo-ifchange for it.
- A dofile can use a loop involving inotifywait to wait for changes to dependencies and immediately rebuild targets if a dependency changes.
- TeX documents can contain internal references. Page and equation numbers are written to an external file needed for a second build.
- TemplateHaskell with profiling needs an executable compiled twice.
- The technique detailed in David A. Wheeler's Fully Countering Trusting Trust through Diverse Double-Compiling requires two builds of a target.

# Why does redo-ifchange not add a dependency if the invoked dofile does not generate a target?

redo rebuilds target files when source files have changed. Files that do not exist can never change – therefore, a non-existent file is always up to date.

# Did you write tests for your redo implementation?

Yes.

# How does this implementation of redo compare with other implementations?

The following table compares redo implementations. Differences from the DJB redo design and incompatibilities with existing dofiles are emphasized.

Implementation | Programming Language | Modification Time Check | File Content Hash Check | Parallel Build | Notes, Bugs, and Incompatibilities
--- | --- | --- | --- | --- | ---
Nils Dagsson Moskopp's redo | Bourne Shell | ✓ Yes | ✓ Yes | ✓ Yes | few dependencies
Alan Grosskurth's redo | Bourne Shell | ✕ No | ✓ Yes | ✕ No | few dependencies dofiles must be shell scripts
Avery Pennarun's do | Bourne Shell | ✕ No | ✕ No | ✕ No | few dependencies each target is always rebuilt
Avery Pennarun's redo | Python | ✓ Yes | ✕ No | ✓ Yes | targets are not rebuilt if modified
Christian Neukirchen's redo | C | ✓ Yes | ✓ Yes | ✓ Yes | standard output is not written to target
Jonathan de Boyne Pollard's redo | Bourne Again Shell | ✓ Yes | ✓ Yes | ✕ No | dofiles must be executable and begin with `#!` special files are never hashed
Gyepi Sam's redux | Go | ✓ Yes | ✓ Yes | ✕ No | wrong handling of \$2 parameter
jekor's redo | Haskell | ✕ No | ✓ Yes | ✕ No | \$1 parameter is not provided no redo-ifcreate implementation
Jonathan de Boyne Pollard's redo | C++ | ✓ Yes | ✓ Yes | ✓ Yes | special files are never hashed
Shanti Bouchez-Mongardé's redo | Python | ✓ Yes | ✕ No | ✓ Yes | dofiles are also looked for in `do/` directory standard output is not written to target targets are not rebuilt if modified
Tharre's redo | C | ✕ No | ✓ Yes | ✕ No | targets are only built once

#