How to make internet-of-things data safely and efficiently available?
The Internet of Things (IoT) allows you to collect interesting data. For research purposes, for example, or to gain insight into how busy it is on the campus. How can educational and research institutions collect this type of data efficiently and securely and make it available?
SURF is developing various services in collaboration with these institutions. Lolke Boonstra (TU Delft) and Jeroen van Ingen Schenau (University of Twente) are already using them. "These kinds of IoT services are valuable for the whole of education and research."
Monitoring traffic flows on the campus
At TU Delft, they have noticed in recent years that more and more researchers are interested in wifi data. For example, which IP addresses are connected to which access point in the network, and when. This is very interesting data, because it tells you a lot about the occupancy and use of buildings, but also about the traffic flows of people across the campus.
"Since Covid-19 and the accompanying measures, we have received more and more requests: with data from the wifi network, you can easily monitor whether the maximum building occupancy is exceeded," says Lolke Boonstra. He is an IT research expert at TU Delft and deals with IT support for researchers. "We had already been working for a few years on the question of how to make wifi data available to researchers. Researchers can submit requests to IT management each time, but this costs both the researcher and the IT manager a lot of time and effort. And a lot can go wrong, for example when it comes to guaranteeing privacy."
Data streaming platform
Lolke and his colleagues started thinking about how researchers could access wifi data in a standardised way. This required a data streaming platform. At its core, a data streaming platform consists of a server or a number of servers that receive real-time data from source systems (e.g. sensors). This is done via APIs, among other things. Researchers can log on to the platform and, depending on their rights, download and use data from the system. Other services can also be connected to the platform, for example for creating 3D visualisations or overviews (dashboards).
“If you process data, you will soon have to deal with the GDPR.”
Together with SURF, TU Delft has developed the TU Delft Data Platform for Researchers. Lolke: "We started by hiring consultancy to program this platform, initially with the sensor data from the Green Village field lab on campus. SURF set up a Kubernetes cluster on which the Kafka software runs, the software for streaming data. We then expanded and developed the platform together. The platform now runs in the public cloud, by the way, and that is also how SURF will be offering the service in the future. One of the advantages of the cloud is that it makes it easier for us to expand the services we purchase from SURF with extra functionality. This extra functionality will also be used for the data of the Green Village."
Personal data
There are not only technical aspects to building a data streaming platform. "If you process data, you soon have to deal with the GDPR," says Lolke. "In the case of wifi, for example, you are processing IP addresses and that is personal data. It is not easy for a researcher to comply with the GDPR. To make his task easier, we partly regulate the privacy guarantee in the data streaming platform: for example, we pseudonymise the MAC and IP addresses of devices that use the wifi network.
In their data management plan, researchers still have to indicate which data they want to use for which purpose, and how long they want to store the data. But a large part of the accountability for the GDPR takes place in the data platform, and also in the systems that supply the data to the platform. This also makes it easier to justify the use of the data to the legal department, which is a major advantage of this way of working."
TU Delft campus
Good collaboration
The University of Twente also makes use of SURF's IoT services. They already have a data streaming platform running there, but use SURF's LoRa network server, which is based on open source software from Chirpstack. Jeroen van Ingen Schenau is a network administrator at the UT and is involved in IoT projects. "We have a number of LoRaWAN gateways on campus. These are receivers that use a special wireless protocol to send data from sensors all over the campus. For example, presence detectors in rooms, temperature sensors, parking sensors and so on. These gateways send the received data to the network server. From there, the data goes to the data platform for processing."
"We also have some influence on the development of new features."
The UT already uses a network server based on The Things Network. This network has a wide coverage and is mainly used by researchers. But for the business data (the data originating from the aforementioned sensors on the campus), the UT was looking for a highly reliable network server. The LoRa network server turned out to be exactly what they were looking for. Jeroen: "This server is hosted and managed by SURF itself. And because we have direct contact with SURF, we also have some influence on the development of new features. The collaboration is going well and it creates confidence that SURF is back in direct contact with the Chirpstack developers."
A lot of interest
The data platform is now up and running in Delft. Lolke: "There is a lot of interest. We're getting requests from master's students but also from major research projects. Another good example is the research into machine learning by my colleague Susan Branchett. The data platform is not yet officially in production, but that is just a matter of time. At the moment, we are still busy with data wrangling: preparing raw data for reading in the data platform and ensuring that data is delivered in a standardised manner. That costs us time now, but it will save researchers a lot of time later on."
More plans with the data platform
Delft University of Technology and the UT already use SURF's IoT services, but they are not yet official services. They are working on that. Lolke: "Together with SURF we have developed this wonderful data platform, and we see that it can also be of great benefit to other institutions. A standard solution will save other universities a lot of research and development work. And we ourselves have many plans for this data platform. TU Delft's policy is that the campus itself should be an object of research, and we want to be able to measure and investigate as much as possible. Think, for example, of energy consumption data. This is very interesting for researchers in connection with the upcoming energy transition."
Jeroen is also pleased that SURF will soon be offering IoT services on a structural basis. "We're now using the SURF LoRa network server to process data from a number of business-critical processes. So it's nice to know we'll soon be able to use the service for the long term at SURF. "
Text: Jan Michielsen