3.4. Separation of concerns¶
The phrase “separation of concerns” was coined by Edsger W. Dijkstra and highlighted in his essay “The role of scientific thought” [Dijkstra, 1982]. More on some details below. The idea behind this approach is that with any inherently complex project, we need to focus all our attention on smaller parts if we actually want to get something done. In some sense, part of the Unix philosophy – do one thing, and do it right – reflects Dijkstra’s approach.
3.4.1. The role of scientific thought¶
In his seminal essay “The role of scientific thought” Edsger W. Dijkstra reflects on the impact scientists with their thinking and problem-solving approach had on the early development of informatics and software engineering. Having received his PhD in Theoretical Physics in the late 1940s, he became the first “programmer” or “software engineer” in the Netherlands. Due to his strong background in (Theoretical) Physics, he has a (probably) unique view on software and programming that particularly in context of scientific software should resonate with their developers that are mostly scientists themselves.
3.4.2. A strategy to cope with the inherent complexity¶
In his essay, Dijkstra argues in favour of applying applying the scientific method to programming and software development. This means in particular that the only sensible way to approach an inherently complex problem is to dissect it so long as we end up with a problem we might know a solution for or at least how to approach it. Silently unmentioned goes that in order to be able to follow this approach, we need to know our tools and our discipline. So what has gone missing since then? Why do students get educated for five years on average to become a scientist, but think that they can acquire the necessary skills to develop software that meets scientific (or other quality) criteria on the go? And why do we forget far too often that only focussing our limited attention to a small part of the problem at a time is the only viable way to get something done?
3.4.3. Some applications to scientific software development¶
The application of Dijkstra’s “separation of concerns” on the code level is immediately obvious, regardless what kind of software we’re developing: structure your code, use functions, classes, modules, packages, libraries. Much of what has been written about the length and focus of functions (e.g., [Martin, 2008]) is a direct reflection on Dijkstra’s advice.
On a rather high level and specific for scientific software: data processing and analysis is different from data acquisition and viewing. You can incorporate all these aspects in one program, and this may even be sensible, depending on the actual situation. However, make sure to cleanly separate those different aspects in your code. Sometimes, clearly separate programs may be a better choice. A viewer program that allows you to display data as they are recorded, and even apply some basic processing and analysis, can be entirely separate from the actual data processing and analysis. How is that? Simply because viewing data at your experimental setup while or immediately after recording usually requires a convenient user interface such as a GUI, but does not need to concern itself with reproducibility or at least traceability. The latter is a crucial prerequisite for any data processing and analysis considered scientifically sound.
Naturally, separation of concerns leads to modular code, and here again, another part of the “Unix philosophy” comes in: use universal interfaces. Whether this is necessarily always text files as in the Unix world does not concern us here. The idea of the universal interface is important. As an example for software for scientific data processing and analysis: If every function operating on data reads and writes a data structure (“dataset”) containing both data and relevant metadata including a history of performed tasks, you can write a highly modular and at the same time arbitrarily complex framework and even ensure reproducibility or at least traceability.