Hackathon high performance computing with Python and RS-DAT

Making efficient use of high-performance systems can be challenging. That is why the eScience Center and SURF have developed the Research Stack for a Distributed Analysis environmenT (RS-DAT). RS-DAT is a framework, based on the Python-Jupyter-PyData-Dask ecosystem, that makes it easier for non-experts to apply distributed data processing techniques and run their analyses on the supercomputing and storage systems at SURF, such as Snellius, Spider and dCache.

The goal of this hackathon

During the hackathon, you can bring in your research problems, and we will help you apply the RS-DAT framework to them. Some examples of things we can help you with are:

• speeding up your Python code with the help of parallelisation;
• analysing large data sets that do not fit on one machine;
• making efficient use of storage systems in your workflows.

In addition, this hackathon serves as a follow-up session to the RS-DAT training, where you can ask more in-depth questions. 

This is an open hackathon, so you can join one or several days or even part of a day. If you have questions or would like to discuss your use case beforehand, feel free to reach out to tim.kok@surf.nl.  

For whom

Researchers and research software engineers that would like to scale their data analyses in Python to high-performance computing systems.

Requirements

• Personal computer and terminal with SSH.
• Experience with Python.
• Basic knowledge of Unix shell is useful. 

Note: on March 19, we will organise a training on RS-DAT that will provide a solid basis for participating in this hackathon. Future trainings and hackathons will be listed here and on the TDCC website.