Why the governor of New Jersey is calling for COBOL programmers
A software environmentalism story
A few days ago, the governor of New Jersey called explicitly for COBOL programmers to help with the development of the unemployment system during the COVID-19 crisis:
This is how the software environmentalism crisis looks like.
In the software development industry, we focus almost exclusively on building systems. The conversations are dominated by new languages, new frameworks, new deployment options. And the body of software grows exponentially. At the same time, we are unable to remove old systems.
From this perspective, our software industry behaves not unlike the plastic industry: we focus on building without any regards to recyclability. That is not sustainable. And, given the breadth of software’s reach, it is not responsible either.
The call for COBOL programmers by the governor of New Jersey takes people by surprise, including the governor himself who said that “There’ll be lots of postmortems and one of them on our list will be how did we get here where we literally needed COBOL programmers.”
And the apparent sudden growth of the #COBOL tag popularity is reason for surprise as well:
After all, we associate a popular tag to be trendy, and COBOL is certainly not considered cool. But, COBOL is merely a symptom. The underlying problem is systemic. COBOL is not a problem per se. The problem is that we associate COBOL with extinction, while it is still one of the most used programming languages in the world.
We might think that changing a system is just a matter of will. Interestingly, in the case of the New Jersey unemployment system, there were many signs. For example, take a look at this one:
I would be surprised if the people that manage this essential system did not know that depending on a technology considered outdated poses a great risk. I’d also be surprised if they didn’t try to address it, and fail for one reason or another.
In fact, I am seeing this with every company we interact with. Everyone seems to have at least one large skeleton in a closet right at the core of the business that they simply cannot get rid of.
When I ask people at various conferences if their company relies on at least one mainframe system, the large majority does. This is not a local problem. It’s a systemic one that requires a systematic rethinking of our work.
We need to change the narrative. Recyclability must be embedded into how we build our systems.
The term “software” was born to contrast “hardware”. At that time, hardware was as large as a building and software was easy to replace. We now live in a different world: hardware fits in my pocket, while we are drowning in software we cannot move.
When developing systems, people often focus on current or immediate functionality. That is shortsighted and dangerous. Functionality is indeed a business asset, but the ability to adapt to changes in the surrounding environment depends on the non-functional parts. The New Jersey unemployment system did its job just fine until recently. Its functionality did not deteriorate. It was the environment that changed and this made the system is no longer adequate. It now needs to be recycled to match the new reality.
To recycle a system implies taking it apart and refurbishing it for other purposes. However, before we can take it apart, we first have to understand the parts.
Interestingly, the activity of understanding a system happens to be the single largest expense we have, with developers alone spending 50% or more of their time reading code. Yet, nobody talks about how this time is spent. That’s the first thing we should change.
We need to start talking about how we read code. Better yet, how to not read code.
Developers read code because they want to understand enough to know what to do next. This is called decision making. Given that we spend most of our energy doing that, it would be more interesting to define our work as being primarily a decision making one rather than construction one.
As the goal is decision making, reading is just the strategy through which we extract information out of our system. It also happens to be the most manual and error prone strategy.
When I ask technical people whether they have pictures about their systems, everyone says yes. But, when I ask whether they believe those pictures are 100% accurate, most people say no. That’s because the pictures were created manually.
A picture or report about a system exhibits either a wish (when the picture depicts a plan) or a belief (when the picture depicts what we think the system is). Both are essential for humans, but neither should be used as basis for decision making.
Decision making is hard enough as it is. Often uncertainties in the environment require probabilistic thinking. However, the reality of our own system should not be a probability. The reality of our system must become a certainty.
The sheer size of our systems far exceeds the ability of humans to grasp without summarization. A picture does tell a thousand words, but those words must be accurate.
Now, accurate pictures are still not enough. For a picture to be effective, it must be specific to the question we want to answer.
Software systems are likely the most complicated structures we have ever created as humans. To this end, it is impossible to predict specific questions. We can predict classes of questions but not specific ones.
That is why pictures have to be created after the question was made explicit. To make this happen the ability to craft a picture must reside inside the team. Given its impact, this ability is not merely a technical one. It has high level business strategic importance.
A good picture or narrative can make humans grasp a data problem orders of magnitude faster than without one. A decade ago this was less obvious, but by now we have ample evidence from all the efforts around data science and the impact it has on business decisions.
As everything about our systems is data, including code, the same principle applies to software development, too. We just have to think of our job through this new lens. And the good news is that this is less difficult than expected.
Just think of this: most systems take data, model it, manipulate it and present it to users that make decisions fast without ever seeing the raw data. So, we, software people, possess this magic skill of transforming data into information to speed up decision making. The skill is already present inside our teams. We just have to apply it to our own work!
Every time developers read, they imply that the only way to answer the question is manually going through raw data. However, through our work and that of others, we showed that most software assessment is rather decomposable as a search or visualization problem.
That might sound academic, but it’s really not. Consider this: when a developer address a data problem in the database, she first writes a query. When the same developer addresses a problem in code, she starts scrolling. Both are data problems. Only the mental models are different.
How do we change this?
We should listen to what Marshall McLuhan and John Culkin told us many decades ago about the nature of our tools. We should observe that there is a deep correlation between the interface of our tools, and the mental models they induce. A database tool has a big query box at the top. People query. A typical development environment has a giant text editor at the center. People scroll and read.
We shape our tools, and thereafter our tools shape us. We should carefully choose the characteristics of our tools because they dictate how we think. Our tools, and not only our languages, are essential. The stakes are too high to disregard this.
How would this look like in practice? Consider a case study of reasoning about evaluating @Deprecated classes in a Java system. In our experiments, we ask developers to estimate the time it would take to go over a given system to find which deprecated classes they can remove and what to do with the ones that cannot be removed right now. For that specific system, comprising of 2500 classes of which 25 are deprecated, the average estimation is 4h, the minimum being 2h.
We picked this problem because this is a rather basic evolutionary problem about which developers often learn in introductory courses. This should be a highly optimized problem that we could not optimize meaningfully any further.
We show how the same problem can be addressed better and more accurately in a matter of minutes. Minutes. Literally.
Here is a description of how that works:
This is an order of magnitude difference when dealing with a simple scenario. The difference is correlated with the nature of the tools. We built a new kind of an environment precisely so that we can think differently. And this difference only grows as the problems get larger.
But, the tool is not the interesting part. The specific tool used is relevant to show what is possible. The most interesting part is that people can understand what the tool conceptually does in minutes, even when they’ve never done any software analysis before! There is no conceptual learning curve because developers have already done the same thing many times, only on other data. This ability is essential and it is reusable.
Tools are essential in software development. The key characteristic is that they should be customizable inexpensively while working on the problem. They should be moldable.
Every time you replace manual work with automation, the real value is extracted when you change the business processes. The same applies here. We have seen historically that when tests became code, we shifted towards various forms of test-driven development. When infrastructure became code, we reshaped companies around DevOps.
Tools must become code, too. And this leads to moldable development, a way of guiding development through the lenses of custom tools.
This article is already way too long. Does it cover the whole space? No. Do we have all the answers? Certainly not. But, we do have the beginning of the conversation. And this is a conversation we must have.
Our kids will only know a world based on software. We already messed up the real world for them, and that is certainly something we must address. Still, while addressing the real world, we should not create a larger problem.
The New Jersey unemployment system is but a case that happened to gain media attention. But, the problem is systemic. We are drowning in systems we cannot move.
The software world is in a crisis of environmental proportions, too. But, that is addressable. And like with the real world problem, we must see it for what it is and make it an explicit subject of conversation.
This article is based on a Twitter thread I wrote not long ago:
