Parchive - ArchWiki

Jump to content

# Parchive

1 language

- 日本語

From ArchWiki

Related articles

- List of applications/Security#Hash checkers

Parchive (Parity archive) is a file verification and repair tool using PAR2 files to detect damage in data files and repair them if necessary. It can be used with any kind of file.

## Installation

Install the par2cmdline package. The commands `par2`, `par2create`, `par2verify` and `par2repair` are now available.

## How it works

`par2create` takes the input file(s) and interprets the input as a certain number of data blocks. Based on the data blocks, `par2create` then creates recovery blocks with the help of the w:Reed–Solomon error correction code. Later, you can trade any recovery block for any corrupted data block in order to repair the source data. You need as much recovery blocks as data blocks have gone corrupted in order to repair the file(s) successfully.

Let us say you want to calculate 30% of recovery information for a precious file:

    $ par2create -r30 file

Parchive now has created the `file.par2` index file which is essentially not needed for recovery. Additionally, it has created the recovery blocks and has spread them into multiple files. If you created, say, 592 recovery blocks, you will find the files

    file.vol000+001.par2
    file.vol001+002.par2
    file.vol003+004.par2
    file.vol007+008.par2
    file.vol015+016.par2
    file.vol031+032.par2
    file.vol063+064.par2
    file.vol127+128.par2
    file.vol255+256.par2
    file.vol511+081.par2

The number left of the plus sign is the index of the first recovery block in the particular file and the number on the right the number of recovery blocks the file provides.

In the early days where no integrity check was done on link level these files proved useful since one could select recovery files based on the number of data blocks gone corrupted. If your download left you with 43 corrupted data blocks, you could convert the number to binary and instantly see what recovery files you had to fetch:

        32 16  8  4  2  1
    43 = 1  0  1  0  1  1

    → download *+032.par2, *+008.par2, *+002.par2 and *+001.par2.

This is very efficient in terms of bandwidth usage. One would finally call

    $ par2repair file*.par2

in order to repair the downloaded file(s). You can ignore the index file since Parchive can handle a missing index file. Sometimes it is useful to include the `-N` parameter in order to reach an intact data block which would otherwise go undetected.

## Tips and tricks

### Batch-protecting your files

It may be the case that you do not want to serve clients recovery files, but want to ensure additional integrity of your files. In times of helium filled hard disk drives, shingled-bit technology, dense capacity per square centimeter, transfer losses etc. the probability of bit rot is high. Remember that you should always have (automated) backups of your data, but a little additional protection does not hurt, especially since we have so much storage available today. By creating `par2` files you have a much more convenient way to verify the integrity of your data and restore the data than running application programs over the files and sieving for error outputs. Bit rot now can happen in both the original file(s) AND the recovery file(s), and you still can repair the original file(s).

As a consequence, one recovery file containing all recovery packets is sufficient (parameter `-n1`). This also reduces the amount of recovery data. An important question to answer is the percentage of redundancy you want to have. Especially for smaller files (<1 MiB) the amount of recovery data does not really correlate with the original file size:

Original data | percentage | Recovery data (without index)
--- | --- | ---
184.8 KiB | 5 | 287.8 KiB
184.8 KiB | 100 | 743.1 KiB
3.4 MiB | 5 | 458.8 KiB
3.4 MiB | 30 | 1.5 MiB
3.4 MiB | 100 | 4 MiB
1.7 GiB | 5 | 87.6 MiB

5% is a reasonable amount of recovery data but you can go up to 100% recovery data for really important files. 100% recovery files can restore your file if you accidentally deleted the original one and you are too lazy to search for the file in your backup (you have one, have you? :).

Here is a simple script which runs over the current directory recursively and batch-protects the files:

Tip: You may have to make sure that your `bin/` folder is in the search path:

    $ export PATH=$PATH:$HOME/bin/

Put this line in your init file in your home directory to make it permanent.

    ~/bin/batchprotect.sh

    #!/bin/bash
    readonly REDUNDENCY_PERCENTAGE=5
    readonly PRINT_ON_SUCCESS=1

    create_par2() {

      local current_file="${1}"
      if [ ! -e "${current_file}-5p.par2" ] && [ -s "${current_file}" ]; then
        if par2create -n1 -q -q -r"${REDUNDENCY_PERCENTAGE}" "${current_file}"; then

          if [ "${PRINT_ON_SUCCESS}" = 1 ]; then
            printf '\033[1;37m%s \033[1;34m%s \033[1;32m%s \033[0m\n' \
                    'Creating par2 on' "${current_file}" 'successful'
          fi

          rm "${current_file}.par2"
          mv "${current_file}".vol*par2 "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2"
          return 0

        else
          printf '\033[1;37m%s \033[1;34m%s \033[1;31m%s \033[0m\n' \
                  'Creating par2 on' "${current_file}" 'failed'
          return 1
        fi
      fi
    }

    verify_par2() {

      local current_file="${1}"
      if [ -f "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2" ]; then
        if par2verify -q -q "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2"; then
          if [ "${PRINT_ON_SUCCESS}" = 1 ]; then
            printf '\033[1;37m%s \033[1;34m%s \033[1;32m%s \033[0m\n' \
                    'Verifying par2 on' "${current_file}" 'successful'
          fi
          return 0
        else
          printf '\033[1;37m%s \033[1;34m%s \033[1;31m%s \033[0m\n' \
                  'Verifying par2 on' "${current_file}" 'failed'
          return 1
        fi
      fi
    }

    repair_par2() {

      local current_file="${1}"
      if [ -f "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2" ]; then
        if par2repair -p -q -q "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2"; then
          if [ "${PRINT_ON_SUCCESS}" = 1 ]; then
            printf '\033[1;37m%s \033[1;34m%s \033[1;32m%s \033[0m\n' \
                    'Repairing' "${current_file}" 'successful'
          fi
          return 0
        else
          printf '\033[1;37m%s \033[1;34m%s \033[1;31m%s \033[0m\n' \
                  'Repairing' "${current_file}" 'failed'
          return 1
        fi
      fi
    }

    main() {

      while read -r current_file; do
        # Create or verify
        if [ ! -e "${current_file}-${REDUNDENCY_PERCENTAGE}p.par2" ]; then
          create_par2 "$current_file"
        else
          if ! verify_par2 "$current_file"; then
            repair_par2 "$current_file"
          fi
        fi
      # We ignore the file itself, any par2/sig extension and any .git directory and print without the leading ./
      done < <(find . -type f ! -name '*par2loop.sh' ! -name '*par2' ! -name '*.sig' ! -path '*.git*' -prune -a -printf '%P\n' | shuf)
    }

    main

You would then call `batchprotect.sh` in order to have all files in the current directory recursively protected by Parchive. Additionally, you do not have to maintain list of checksums since gpg can do that for you, too.

### Verification

    $ par2verify file-5.par2
    $ cfv file-5.par2
    $ gpg --verify file.sig

And, if you changed the path of the original file:

    $ par2verify -B / file-5.par2 /new/path/to/fileRenamed
    $ gpg --verify file.sig /new/path/to/fileRenamed

You can also change the path of the `par2`/`sig` file.

Categories:

- Data compression and archiving
- System recovery