Using OCaml and Irmin to Help Climate Scientists

What if we could use OCaml to help climate scientists around the globe? A project we worked on with a climate researcher has turned this lofty goal into reality! A website and workflow built using Dream, OCurrent, and Irmin is helping scientists all over the world share and manage data to better understand climate change.

Helping Climate Scientists

Christopher Carcaillet is a professor researching climate change and its impact on different ecosystems. After seeing Anil Madhavapeddy's presentation at 'Functional Programming for the Planet', it struck him that there were more ways that computer science could aid climate scientists, and he contacted Tarides to help him solve a problem he and his colleagues were having.

Researchers use carbon fourteen (14C) to indicate the age of radiocarbon deposits found in the ground, helping them study the charcoal production caused by forest fires and the impact of the ground and climatic factors on historical CO2 retention over time. With many scientists operating around the globe, having a way to store and share their data easily would benefit the entire field. Scientists could use each other's data much more efficiently and contribute their own to the body of research. After Christopher contacted us, our intern at the time, Leonardo Mendes Santos, started a project to create a user-friendly workflow for scientists to record, share, and visualise radiocarbon data.

Creating a User-Friendly Workflow

There were a few prerequisites to the solution Leonardo designed. The database needed to be open source to ensure transparency and to encourage collaboration, and hosting it on GitHub was desired as a widely supported and easily maintainable solution. However, few scientists, besides computer scientists, are confident using GitHub. To make the platform more widely accessible, Leonardo created a website that functions as a straightforward interface for the database.

The website gives researchers access to sophisticated tools for analysing and processing their data without adopting new, unfamiliar workflows. Its main features are that it lets researchers share data and collaboratively build 14C data collections; allows them to automatically convert 14C dates into dates that correspond to the human calendar (a necessary but otherwise lengthy and complicated calculation); and means that researchers wishing to build upon previous results no longer need to copy 14C dates individually from each source paper.

Expanding on the second point, the automated process consumes and outputs csv format files. Since most scientists use Excel to manage their datasheets, Leonardo integrated an xlsx to csv converter to the website. Reducing friction for users encourages more scientists to share their data, contributing to the body of knowledge and making the database more and more useful over time.

This is what the site looks like:

The Tech Stack

Irmin

Irmin is an OCaml library that builds robust, secure, and distributed data stores with audit trails. Since the database was going to be hosted on GitHub, using Irmin was a good choice as it is git compatible. For the radiocarbon website, Irmin provides both the user database—managing the user log-in, account creation, and administration—and a local irmin-git replica of the database.

Using Irmin allowed Leonardo to respond quickly to requests to add or amend features. For example, the scientists wanted to determine the exact R script and parameters that had produced an automated analysis output. Leonardo implemented this functionality by asking Irmin to time travel to the creation of the analysis output and consult the input files (CSV, config, script, etc.) as they were at that time.

From a security standpoint, not all researchers can or want to contribute their original work directly to the public database, as publishing a peer-reviewed paper remains the first step. Leonardo provided these users with a private branch of the Irmin database state, where R scripts also run automatically, but changes and results are not visible to others. Once ready, the researchers can propose their changes for integration into the public database (similar to a pull request).

OCurrent

Using the OCurrent workflow, Leonardo could automate radiocarbon calibrations after users make changes to the database.

OCurrent is an OCaml pipeline language that lets users specify a workflow for keeping their repositories, databases, or other projects up-to-date. With the Git-based database and updates that needed to be made between it and the website, OCurrent was a great option for managing all the different layers of the backend.

Leonardo's pipeline implementation detects changes to input 14C files, configuration files, and script files. It provides continuous integration for any R script analysis running on changes. This means that the output is automatically pushed to the database's repository, and since the database is stored with git, the process provides reproducibility and traceability. As researchers can contribute more R scripts alongside new data, the system is intended to grow and facilitate more complex analysis for everyone. This is essentially a CI (Continuous Integration) flow for data science.

OCurrent is an EDSL, an Embedded Domain-Specific Language, which can specify inputs and outputs as a graph. This enables OCurrent to keep track of dependencies and only recompute the parts of the graph that have changed. The results are cached, ensuring that OCurrent doesn't rerun something it shouldn't.

OCurrent integrates with Docker and Nix to run builds, giving the user a backend website 'for free' with a graph (pictured below) to help monitor and debug any issue in production. OCurrent also integrates with Github Webhooks to keep users notified of changes.

Dream

Dream is a web framework for OCaml that is easy to use and set up, with many quality-of-life features for developers. These include templates for interleaving OCaml and HTML, a composable router, and unified error handling. Plus, since it's boilerplate-free with a short and simple API, Dream lets users get up and running quickly and is compatible with a host of different popular tooling. Dream supports TLS, WebSockets, and GraphQL and requires no setup beyond installing one package, Dream.

Picture This

Here's an architecture diagram of the tech stack for the radiocarbon project, showing you how Dream, OCurrent, and Irmin work together to provide an online service.

Until Next Time

Are you curious about using Irmin in your projects? Contact us to discuss how to use the database to build mergeable, branchable, and distributed data stores. Leonardo's accomplishments highlight the powerful combination of Irmin, OCurrent, and Dream to automate and deploy an interactive database on a website.

Connect with Tarides online on Bluesky, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects.