MIRRORING-PORTS(7) Miscellaneous Information Manual MIRRORING-PORTS(7)

mirroring-ports — how to build a mirror for ports distfiles

The OpenBSD Ports Collection offers some powerful tools to mirror software sources.

dpb(1) features a -F option which is explicitly designed for mirroring distfiles.

If run with -F jobs, dpb will

  • Limit itself to fetching distfiles, and not build any packages.
  • Disregard any architecture or broken annotation, and try to fetch every distfile.
  • Fetch files to a temporary copy named some_file.part using ftp -C to resume interrupted downloads.
  • Keep a global list of sha256 checksums as ${DISTDIR}/distinfo, and use that to refetch files when the ports tree records a changing checksum.
  • Produces a log of old distfiles in ${DISTDIR}/history,
  • Create sha256 links under ${DISTDIR}/by_cipher/sha256 as per link-checksum(1)'s former duties.

For partial distfiles collections, dpb can also be run with -DHISTORY_ONLY to scan the full ports tree and update ${DISTDIR}/history without fetching anything.

${DISTDIR}/distinfo
a cache of known distfiles with their respective checksums.
${DISTDIR}/history
List of files appearing in ${DISTDIR}/distinfo that seem to no longer be required by the ports tree. dpb(1) will append to this file each time it is run on the whole tree (option -a) and only if the ports tree scan finishes without error. Each line is of the form 
timestamp SHA256 (file) = sha

denoting the first time a file/sha entry was no longer seen in the ports tree.

clean-old-distfiles(1), dpb(1), ports(7)

The new integrated -F option to dpb(1) was introduced in OpenBSD 5.1, replacing the original infrastructure introduced in OpenBSD 2.7.

Changing checksums is a recurring problem that is outside the direct control of the OpenBSD Project. Some software distributors change distribution files without warning, without changing the file name proper. Once the problem has been identified, the port maintainer should usually contact the software author to fix the problem or, if the software author is unresponsive, the maintainer should use DIST_SUBDIR to provide some state to guard against shifting checksums.

However, a more robust approach is also needed, so that ports users can depend on distfiles mirrors to carry what they need irrespective of those synchronization issues. The ${DISTFILES}/by-cipher/sha256 directory provides more persistent access to the distfiles, indexed through the actual checksums that the files should match. Provided mirroring is run sufficiently often, two versions of the same distfile with respective checksums cksum1 and cksum2 will be available under the names ${DISTFILES}/sha256/c1/cksum1/distfile and ${DISTFILES}/sha256/c2/cksum2/distfile.

If REFETCH is set to true, bsd.port.mk(5) will try to retrieve files under that naming scheme as a last resort.

August 27, 2020 OpenBSD 7.5