According to the RedHat summary:
The vulnerability exists when the library is decoding data out of a dataset encoded with the H5Z_NBIT decoding. When calculating the precision that a BCD number is encoded as, the library will fail to ensure that the precision is within the bounds of the size. Due to this, the library will calculate an index outside the bounds of the space allocated for the BCD number. Whilst decoding this data, the library will then write outside the bounds of the buffer leading to a heap-based buffer overflow. This can lead to code execution under the context of the application using the library.
When decoding data out of a dataset encoded with the H5Z_NBIT decoding, the
HDF5 1.8.16 library will fail to ensure that the precision is within the
bounds of the size leading to arbitrary code execution.
Created attachment 454952 [details, diff]
* This bug affects 1.8.17 as well, and is release fixed with 1.8.18 which is not in the tree
* The commits in the URL are incomplete fixes as they reference functions not defined in the patch
* Attached is a patch generated by a diff of the file in question between 1.8.17 and 1.8.18.
* The code differences between 1.8.17 and 1.8.14 (current stable) are enough to where I can't reliably backport
* With that in mind this fix should be combined with bug #601420 and 1.8.17 stablereq'ed
This issue was resolved and addressed in
GLSA 201701-13 at https://security.gentoo.org/glsa/201701-13
by GLSA coordinator Thomas Deutschmann (whissi).