Bug 709558

Summary:	sys-cluster/er - High-level distributed erasure coding library combining functionality of shuffile and redset
Product:	Gentoo Linux	Reporter:	Alessandro Barbieri <lssndrbarbieri>
Component:	New packages	Assignee:	Default Assignee for New Packages <maintainer-wanted>
Status:	UNCONFIRMED ---
Severity:	normal	Keywords:	InOverlay, PullRequest
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	Linux
URL:	https://github.com/ECP-VeloC/er
See Also:	https://github.com/gentoo/gentoo/pull/13407
Whiteboard:
Package list:		Runtime testing required:	---
Bug Depends on:	709554, 709562, 709564
Bug Blocks:	709566

Description Alessandro Barbieri 2020-02-14 02:56:50 UTC

LICENSE: MIT

This module combines the funcitonality of shuffile and redset, which lets both migrate and rebuild files within an MPI job. It is useful for moving and rebuilding data files on distributed storage, e.g., node-local storage, particularly when MPI ranks may be running on different compute nodes than when they originally created their files.

To encode, one defines a redundancy scheme and then applies the redundancy scheme to a list of files also providing a name for the encoded set. To rebuild, one specifies the name of the encoded set. Additionally, there is a function to remove the encoding information, which is needed when removing a dataset.

Currently, er assumes the group of processes participating is the same as MPI_COMM_WORLD.

Usage is documented in src/er.h