Distributed computer system for fast processing of big data sets
Large-scale data processing
Projects can utilize these systems (or a selected subset) simultaneously in order to perform large-scale processing of high-volume data sets, and build tailored production platforms for data analysis and data distribution for their own user community. For example, experiments such as those involving the Large Hadron Collider (LHC), gravity wave detectors (LIGO-Virgo), large-scale DNA analyses (BBMRI, Project MinE), dark matter experiments (Xenon1T), space-based earth-observation instruments (Tropomi), and other demanding projects have their own data processing facilities built on top of our Grid infrastructure. SURFsara offers the Dutch National Grid infrastructure in cooperation with Nikhef and RUG-CIT.
National and European
The Dutch part of the Grid is connected to national grid initiatives of other participating countries, tied together under the organisational umbrella of EGI, the European Grid Initiative. This allows Grid users in the Netherlands and abroad easy access to one another's resources. EGI offers centralized service management utilities and ensures the homogeneity of participating clusters.
Access to data storage and visualisation
The Grid maintains excellent network connections with the rest of the SURFsara data and compute infrastructure: with the facilities for storing data on disk and tape, and visualisation services, for example.
Full range of services
The Grid service in the Netherlands enables scientific projects to share resources related to data processing power, data storage, and services. This is especially relevant for research communities: the Grid facilitates user-friendly collaboration between researchers in a particular field, both domestically and internationally. Grid users have access to the following services:
- Standard grid services such as interactive hosts for job operations, resource brokers and virtual organisation management services;
- Virtual file system services (CVMFS) that allow for centralised distribution of user software to all relevant machines;
- Facilities to integrate orchestration and fault-tolerance into project workflows;
- Tailored platforms for the operation and automation of continuous production run.
We build upon extended experience in supporting projects that have a large data-processing component. Apart from general aspects of using our large-scale infrastructure, most complexity in such projects usually derives from project-specific scenario’s and the sheer scale of the problem. For that reason we offer tailored consultancy to those projects, including assistance in building dedicated production platforms for data analysis and data distribution for their own user community.
Our helpdesk is available by telephone and email, or you can schedule a personal appointment. If you have any questions or want to report a problem, please send an email to firstname.lastname@example.org or phone +31-208001400. The helpdesk is available during office hours (9:00-17:00).
The power of the Grid is most evident in the processing of extremely large data sets. In the following examples, you can see how this works:
- SURFsara supports SRON analysing Tropomi satellite data
- The Large Hadron Collider experiments: detecting the Higgs boson and other particles
- LOFAR: serving the long-term archive of the LOFAR observatory