Grid: for processing and storing large data sets

Do you want to process and store large amounts of data? The Grid may very well be suitable for your project. The grid infrastructure consists of a large number of clusters for computing and data storage, which are interconnected via a fast network.

Onderzoeker achter scherm

What kind of research do you use the Grid for?

You can use the Grid systems (or part of them) for parallel processing of large-scale datasets in a very short amount of time. You can also build a production platform specifically for your own research community that provides automated data analysis and distribution for researchers. A number of demanding research projects have set up their own data processing facility on our Grid infrastructure, such as:

  • CERN's particle accelerator: Large Hadron Collider
  • Gravity wave detectors: LIGO-Virgo by Nobel Prize winners Rainer Weiss and Kip Thorne
  • large-scale DNA analyses: BBMRI, Project MinE
  • dark matter experiments: Xenon1T
  • instrument for the analysis of the earth's atmosphere from space: Tropomi
Nobel Prize winners have used the Grid to measure gravitational waves.

What is the Grid?

The Grid consists of a large number of clusters for computing and data storage, which are interconnected via a fast network. These clusters are located in data centres distributed all over the world.

Are you working on a Dutch project with a large amount of data? Then the Dutch part of the Grid is very suitable.

For the largest scale international projects, you can contact EGI, the European Grid Initiative, of which we are a member. EGI has a central helpdesk and monitors the homogeneity of the participating providers. This makes the Grid suitable for use as one distributed system.

Custom support

We have extensive experience in supporting projects that involve the processing of large amounts of data. Apart from aspects specifically related to the use of large-scale infrastructure, the complexity of such projects arises mainly from project-specific scenarios and the scope of the problem. Based on our broad experience with such scenarios, we offer customised assistance in building dedicated production platforms for data analysis and exchange for your own user community.

Complete support with our additional services

If you use this service, you may also be interested in the following services:

Long-term storage of research data

The Grid, the HPC Cloud and Data Ingest are all connected to the central archive of SURFsara. This archive offers you extensive options for storing your research data. In addition, you can also use the PID (Persistent Identifiers) service on data that is stored on SURFsara storage services, such as Data Archive. Do you want to store your data securely over long periods? Then make use of our Data Archive service.

Visualisation: immediately clarity of results

Do you work with calculations that produce large amounts of data? Then you should use our visualisation techniques and support. Visualisation helps you to better interpret the results of your calculations.

Send data quickly with SURFlightpaths

Do you want a fast and reliable connection from your own network to our systems? With your own lightpath, you can very quickly send data to and from the Grid or the HPC Cloud. A light path is a direct connection that is shielded from the Internet. It is extra secure, reliable and suitable, for example, for privacy-sensitive information. The biggest challenge with these light paths is to connect them to the systems on both sides. We will help you by bridging the final metres between the end point of a light path and your data sources.

Consultancy: independent advice

Our consultants support you from the first analysis of the problem to the final implementation. They provide independent advice on, among other things:

  • Accessing the Grid
  • submitting jobs
  • how to ensure that Cartesius or the Lisa Computing Cluster deliver an even better performance
  • methods for approaching your data
  • design and optimisation of your own software
  • the exact design of your data storage system
  • how to organise your data infrastructure
  • how to make optimal use of our calculation and storage facilities
  • integrating your virtual infrastructure into your work processes
  • optimisation of applications
  • running your software in parallel for faster processing

Depending on the size and complexity of your question, you will receive a customised proposal. We offer you many options in the field of Big Data Services. This includes education and training, but also advice about architecture and the use of technology. For more information, please contact our consultancy service.

Technical specifications of the Grid

These specifications refer to the Dutch grid cluster at SURFsara in January 2019. Although this cluster is only one of many clusters in the international the grid service, it does give an idea of the quality as well as the capacity of our grid facilities.

Grid computing facility
Operating system Linux CentOS 7.x 64bit
Total number of cores

10.000 in NL (latest Intel CPUs), more than 650.000 cores accessible worldwide

Total memory (RAM) Up to 8GB per core, 320GB per node
Total temporary memory space (scratch) Tot 200GB per core, 12TB per node
Network backbone 1200 Gbit/s network connection with the Grid storage facility
Grid storage facility
Disk storage capacity total of 10.000 TB
Storage on tape 41.000 TB of stored data (2018)

Do you have any questions about the technical specifications? Please contacthelpdesk@surfsara.nl. More (technical) information about the use of the Grid service can be found under the Grid user information.

Want to find out more about the possibilities?

Please contact us:

Mail info@surfsara.nl

This is an optional SURF service.