When reviewing options for water-data management software some months back, it transpired that the main challenges—in many places—are not so much related to the software, but to keeping the IT systems running.
But let me take a step back.
Modern hydro-meteorological monitoring networks focus on securing data flow. They aim for continuous monitoring and (near) real-time data communication, processing, analysis, and dissemination. Data should enter the system only once—preferably at the monitoring site—after which it is processed to the final product in an uninterrupted chain of processing, quality control, analysis, and distribution. In principle, no or minimal downtime is allowed.
The time-series Data Management System (DMS) is the core application that organizes and controls data flow. Several private companies are offering excellent off-the-shelf DMS applications with a proven track-record.
Challenges in Securing Data Flow
We listed the main challenges in securing data flow. In our experience, they include:
- vandalism and theft (mostly in the field)
- unreliable power (in the office and field)
- data transmission failure (occasional lack of GPRS airtime and corrupted internet protocols)
- server failure (hardware and software)
- corruption of data and software by computer viruses
- inadequate or no data backup facilities
- difficulties (technical and financial) to upgrade and/or maintain DMS software
- difficulties to maintain the database software
- difficulties to maintain the LAN for the DMS server-client setup
- difficulties to maintain the air-conditioner units for the server room
- sub-offices have no access to the database at the main office
IT issues feature prominently on this list. Water offices in many countries do not have the in-house capacity for maintaining complex hardware and software systems, or for advanced database and network administration. And even if they do, specialized IT staff are hard to maintain due to salary disparities between government and the private sector.
The Arrival of the Cloud
The arrival of cloud computing in combination with much improved internet could subject water-data management to fundamental changes.
It is promising. But is it (already) the right thing to do?
We will compare two alternative IT solutions to secure data flow: a) cloud-based solution, and b) in-house client-server setup.
Cloud Solution
All software—database server, DMS, anti-virus, backup software—is maintained in the Cloud. Likewise, all data—raw data, validated data, and processed data—are maintained in the Cloud together with associated data backups.
The Cloud database is password protected (a dual password setup is possible). In addition, the database administrator—i.e. the water agency—can lock specific records and stations to prevent external access. Data export to external applications is achieved through a ftp process (virtual server).
The setup in the above figure is scalable. It also allows for easy sub-office integration and field access to the DMS—provided there is internet.
(Note: in this example, data transmission is conducted through satellite; stations, however, can be equipped with both GPRS and satellite modems)
Client-Server Solution
A central database server is connected through a LAN to one or more Windows workstations on which full versions of the DMS software are installed. Data manipulation—visualization, QC/QA, editing, rating curve development, analysis, and model execution—takes place on the workstations, with data going back and forth to the database server.
The database server is the critical element in the setup. Server failure will stop all data activities while also preventing access to the historic database. The security of the data server, therefore, is of paramount importance in terms of software and hardware maintenance, anti-virus, backup-power, theft, and others. Note that power failure that leads to uncontrolled shutting down of the server quite frequently corrupts server software and the database.
Internet downtime will not affect the functioning of the data unit apart from data export and import of new data from the field stations or other data sources.
Comparative Analysis
Closing Comments
The IT setup for a Water Agency represents a strategic choice that may affect the functioning of the organization in the near and mid-term future. Hence it needs to be carefully considered. Nevertheless, a Cloud Solution is promising, and it could represent an important innovation that may leapfrog the Client-Server setup. Whether this will happen—and how fast—will depend on:
- the quality of Internet connectivity in terms of speed, uptime, and costs; note that Internet access is improving rapidly in many places
- the security of data stored in the Cloud; this cannot be compromised and needs to be ‘water tight’
- the costs of a Cloud solution; expressed in monetary unit (read: US$) per station
- whether the Water Agency can mobilize these recurrent costs—year after year—for maintaining a Cloud setup for the water Data Management System; in order for the solution to be sustainable, the annual service costs should be comfortably maintained with the available resources; donor funds should preferably not be included in this assessment; introducing an excellent system that stops functioning because of budget constraints would represent a bad investment