Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 709558 - sys-cluster/er - High-level distributed erasure coding library combining functionality of shuffile and redset
Summary: sys-cluster/er - High-level distributed erasure coding library combining func...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Default Assignee for New Packages
URL: https://github.com/ECP-VeloC/er
Whiteboard:
Keywords: InOverlay, PullRequest
Depends on: 709554 709562 709564
Blocks: 709566
  Show dependency tree
 
Reported: 2020-02-14 02:56 UTC by Alessandro Barbieri
Modified: 2020-11-07 12:06 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alessandro Barbieri 2020-02-14 02:56:50 UTC
LICENSE: MIT

This module combines the funcitonality of shuffile and redset, which lets both migrate and rebuild files within an MPI job. It is useful for moving and rebuilding data files on distributed storage, e.g., node-local storage, particularly when MPI ranks may be running on different compute nodes than when they originally created their files.

To encode, one defines a redundancy scheme and then applies the redundancy scheme to a list of files also providing a name for the encoded set. To rebuild, one specifies the name of the encoded set. Additionally, there is a function to remove the encoding information, which is needed when removing a dataset.

Currently, er assumes the group of processes participating is the same as MPI_COMM_WORLD.

Usage is documented in src/er.h