This HPE software update accidentally wiped 77TB of data

A flawed update sent out by Hewlett Packard Enterprise (HPE) resulted in the loss of 77TB of critical research data at Kyoto University, the company has admitted.

HPE recently issued a software update that broke a program deleting old log files, and instead of just deleting those (which would still have a backup copy stored in a high-capacity storage system), it deleted pretty much everything, including files in the backup system, Tom’s Hardware reported.

As a result, some 34 million files, generated by 14 different research groups, from December 14 to December 16, were permanently lost.

HPE explanation

In a press release, issued in Japanese, HPE took full responsibility for the disastrous mishap:

"From 17:32 on December 14, 2021 to 12:43 on December 16, 2021, due to a defect in the program that backs up the storage of the supercomputer system (manufactured by Japan Hewlett Packard), the supercomputer system [malfunctioned]," a rough (Google) translation of the press release says.

"As a result, an accident occurred in which some data of the high-capacity storage (/LARGE0) was deleted unintentionally. […] The backup log of the past that was originally unnecessary due to a problem in the careless modification of the program and its application procedure in the function repair of the backup program by Japan Hewlett Packard, the supplier of the super computer system. The process of deleting files malfunctioned as the process of deleting files under the /LARGE0 directory."

The backup process on the supercomputer has since been suspended, until the software problem is fixed and contingencies put in place to ensure something like this never happens again. Once that happens, the University plans on resuming the backup.

It was also said that the researchers initially thought that some 100TB of data was lost.