Hello, I'm not sure that this bug might really be exploitable in some malicious ways. Yet ImageMagick had a very similar bug, which was later assigned a CVE number (https://github.com/ImageMagick/ImageMagick/issues/6341) and therefore I'm posting this as a possible vulnerability here. I was writing C-function supporting an R package and used an ASAN/UBSAN instrumented version of clang to look for potential memory holes. Instead of any hole I steadily got an "undefined behaviour" message when running a memory and gc-intense test suite manually, i.e when running all tests interactively within a single R-session, but not when R CMD check runs them file by file: memory.c:1204:28: runtime error: 2.63525e+19 is outside the range of representable values of type 'unsigned long' SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior memory.c:1204:28 I still don't know what the actual cause is, but it certainly implies an act of negligence to cast doubles to integers and not care for an obvious overflow within a memory management routine being central to R. The relevant code section is this: 1179 static void AdjustHeapSize(R_size_t size_needed) 1180 { 1181 R_size_t R_MinNFree = (R_size_t)(orig_R_NSize * R_MinFreeFrac); 1182 R_size_t R_MinVFree = (R_size_t)(orig_R_VSize * R_MinFreeFrac); 1183 R_size_t NNeeded = R_NodesInUse + R_MinNFree; 1184 R_size_t VNeeded = R_SmallVallocSize + R_LargeVallocSize 1185 + size_needed + R_MinVFree; 1186 double node_occup = ((double) NNeeded) / R_NSize; 1187 double vect_occup = ((double) VNeeded) / R_VSize; 1188 1189 if (node_occup > R_NGrowFrac) { 1190 R_size_t change = 1191 (R_size_t)(R_NGrowIncrMin + R_NGrowIncrFrac * R_NSize); 1192 1193 /* for early adjustments grow more aggressively */ 1194 static R_size_t last_in_use = 0; 1195 static int adjust_count = 1; 1196 if (adjust_count < 50) { 1197 adjust_count++; 1198 1199 /* estimate next in-use count by assuming linear growth */ 1200 R_size_t next_in_use = R_NodesInUse + (R_NodesInUse - last_in_use); 1201 last_in_use = R_NodesInUse; 1202 1203 /* try to achieve and occupancy rate of R_NGrowFrac */ 1204 R_size_t next_nsize = (R_size_t) (next_in_use / R_NGrowFrac); 1205 if (next_nsize > R_NSize + change) 1206 change = next_nsize - R_NSize; 1207 } 1208 1209 if (R_MaxNSize >= R_NSize + change) 1210 R_NSize += change; 1211 } The double variable is R_NGrowFrac with a default value of 0.7, which can even be set from an environment variable. A probably related effect was that the method call system of R did not find a certain method anymore.
