News

KNMI archive data transferred to SURF

The KNMI will from now on store a large part of its data with SURF. The data obtained from measuring stations, radars, satellites and models is essential for the institute's day-to-day operations. Over the course of 6 months, 6 petabytes of data were restructured and moved.
Tape opslag

The Data Archive stores data on tapes for the long term

The Royal Netherlands Meteorological Institute (KNMI) in De Bilt has been storing its data, generated by weather and climate models and other research, on tapes since the early 1990s.  The capacity, management and accessibility options of this so-called mass storage system (MOS) have been upgraded several times in recent decades.

"Due to the ever-growing volume of data and the need to store this data in a future-proof manner, KNMI has now outsourced storage of the data to SURF. The SURF Data Archive is the ideal place to house the KNMI's archive data."
Gijsbert Kruithof, data architect at KNMI

Observation, measurement and research data that is no longer actively used is stored safely and permanently in the Data Archive. For other data, such as data that needs to be quickly accessible and easily shared for individual use, KNMI uses the SURFdrive and Research Drive storage services.

Special network connection

Over a six-month period, a total of almost 150 million files with a size of 6 petabytes (6000 terabytes) were migrated to SURF's data centre in Amsterdam. Some were reorganised to increase storage efficiency and sent via a special network connection (SURFlichtpad). Another part was sent on a tape copy by courier. 

"The move was a big job. Not everything could be transferred to the new destination just like that, partly because of the minimum file size required by SURF. By merging files and transferring personal files to SURFdrive, we worked with KNMI to reduce the number of files to almost a tenth of their original size and clean up 6% of the data."
Hans van Piggelen, data consultant at SURF

Since November 1, 2021, the KNMI has been working almost entirely with the SURF Data Archive for long-term storage of research data. It will take a few more months to migrate all incoming tapes to the data archive. Once that has been completed, the MOS can be permanently switched off.

Schotel op dak KNMI