Out of the Complexity Crisis
Hello, In 1972 David Parnas introduced the phrase "information hiding". In his paper on software modularisation he contrasts two approaches. The approach that was common at the time was to split a program based on its flow chart. Each processing stage could be handed to a separate team. Different modules worked in succession over the same memory layout. Today we would say that they are tightly coupled. A design decision to change the in-memory representation affects all modules. The alternative he proposes is to use modules to encapsulate design decisions. He proposes a Line Storage module, with subroutines for storing and retrieving data. We now recognise this as an Abstract Data Type (ADT), a term formalised by Barbara Liskov that same year. The rise of Package ManagersAs compute power and storage capacity grew in the ensuing decades, modularisation and information hiding would allow building and maintaining ever larger and more complex programs and systems. Module reuse was on top of everyone’s mind during the heady Object Orientation days of the 1990’s. The internet is going mainstream, and it’s rapidly changing how free and open source software is developed. In 1995 the Comprehensive Perl Archive Network (CPAN) launches, the first language specific package archive and manager. (Hat tip to Adam Nesbit’s excellent Package Manager Timeline) Package managers would quickly become the defining feature of modern software development. Maven, PyPi, rubygems, npm. A Cambrian explosion of libraries followed, in every language ecosystem. Individuals and corporations alike could publish their creations, and vie for influence. And with all these building blocks freely available, the cost of developing complex applications tumbled. When the browser, originally built to navigate hypertext documents, became a powerful application delivery platform, the SaaS boom was born. We’ve kept pushing that information hiding envelope, to make ever more complex systems tractable. We routinely depend on hundreds of transitive dependencies to build a basic web application. Our own code is dwarfed by hidden code. We call it, we run it, but we don’t see it. And gradually these packages that gave us so much became a liability. With information hiding, you only care what a package does, not how it does it. But sometimes, often, the how does matter. The result is software that is slow, janky, error prone. What a line in a dependency file also hides, is the who. Package managers allowed rich language ecosystems to develop, and with them a complex sociology. More than ever programmers came to identify with the language they programmed. High profile authors became household names, and having their name on a package acted as a mark of quality and trust. A thriving conference scene helped to solidify informal networks of people, who pulled together to maintain and develop these ecosystems, and the supporting infrastructure, which allowed an untold number of startups to thrive. But a blessing can easily turn into a curse. We’ve managed to sleepwalk into a software crisis of epic proportions. A paradox opened up, while a vast package ecosystem made it ever easier to rapidly build out new applications, it also made it ever more challenging to engineer elegant, reliable systems. Every module hides a design decision, and these details do matter. The how matters. And in the age of "supply chain attacks" (a horrible phrase which casts a shared commons into the mold of vendor-buyer relationships) turns out the who matters just as much. Not only can people become compromised, and thus become an attack vector, you are also dependent on them for bug fixes and security updates. Now there’s an additional risk, that they let AI lose on their projects, leading to large changes that are hard to vet. A routine update is more likely than ever to add large amounts of code to your project which you did not ask for and which are impossible to scrutinise. Project maintainers on the AI bandwagon are also more likely to alienate contributors. The sociology around open source is rapidly eroding. The Complexity CrisisOf course, smart people have been saying for a long time that every dependency is a liability. That you should think twice, thrice, before adding a new one to your project. That you should vet them carefully. Not just look at the code, but at the release history, the contributors, the project governance. Unfortunately performance pressure has been climbing steadily upward. Engineers are told to skip anything that isn’t "ship features". Software no longer needs to work, it just needs to make money. (paraphrasing Ben Fry). Add several years of layoffs and AI mandates, and it’s easy to see how this crisis keeps escalating. We’ve made sure that both inside companies, and in the supporting ecosystem, people are disincentivised to truly grok the workings of complex systems. The bandwidth they might have had to attempt to do so has been curtailed. The sociology that underpins the commons that all growth in tech is based on has been decimated. Meanwhile the complexity of these systems, and the parts they are made of, is spiraling out of control. And so everything is crashing down around us. One big outage after another. Several remote code execution vulnerabilities in widely used software every week. Engineers running around like headless chickens, trying to keep services patched and in the air, while the ground keeps shifting underneath them. To get through this period, where everything is crashing down around us, where the bulk of software has been made incomprehensible through LLMs, we are at some point going to have to start rebuilding. To move slowly and fix things. I don’t know what that will look like though. In my twenty years of doing this I have never witnessed a contraction of that kind. A moment of reflection leading to more sustainable methods. I don’t know if the industry is capable of such a shift. It will only happen when it becomes too costly not to. Meanwhile I will do what I do best, be the wayward who still tries to deliver delightful digital products. Take what I consider my professional responsibility. We are going to have to do more with less, vetting each library, both its code and its sociology. Keep our systems lean and easy to reason about. I think that’s a good thing. What am I up toThe last few years have been a little non-stop for me, both professionally and in my personal life. I put a lot of work into Gaiwan, and all the projects we had the privilege to work on. Organised two conferences, moved countries, bought a house, and helped my partner move from another continent. Now that the curtain has fallen on Gaiwan, and things are starting to feel a bit more settled personally, it was time to take a break. So I packed my trusty touring bike into a cardboard box, and flew off to Taiwan for a three week cycling holiday. I have a long history with Taiwan. My first visit was over twenty years ago, to visit a family friend. A few years later I went back for an intensive nine month Mandarin language program. It’s become a bit of a second home for me, and I’ve been back a handful of times since then, often staying for a month or more, which is the main way I’ve kept up my Mandarin over the years. But then pandemic and life events starting getting in the way. After an eight year hiatus I figured it was finally time to go see beautiful Formosa once again. And I have to say, it did me good. Biking through the foothills and valleys, making small talk with strangers, enjoying the nature, the amazing variety of food, and drinking liters and liters of tea. It was the reset I needed. Now I’m back and slowly getting my head back into the game. Establishing a writing practice. Figuring out what’s going to come next. It’s still a weird and difficult time, in tech and in the world at large, but I can’t help but be excited. Ain’t no time like the present to shape the future. Previous PollLast time I asked you all about your relationship with AI coding tools. Do you use them, do you have to use them, and how do you feel about that? We got 34 respondents. Because I tried to cram a few different dimensions into a single poll it’s not very scientific, so I don’t want to overstate any results. These polls are meant to be informal vibe checks about my audience. It’s also a small sample size, and probably says more about my particular audience than the state of the industry. Still a few things stand out for me. A third of respondents does not (or no longer) use AI tools for coding. AI tools have become ubiquitous quickly, but there is still a significant contingent that doesn’t want to use them. If you are one of these people, know that you are not alone, even if it may feel like that sometimes. Zero votes for "I HAVE to use them at my JOB, and I LIKE IT". People are not enjoying their AI mandates! Still, for every person that has to use it at their job, two people are choosing freely to use it. There was one write-in answer, "I CHOOSE to use them for my OWN PROJECTS, and I have MIXED FEELINGS.", and even though it was only added later it still got a good few votes. We also have several people saying they choose to use these tools, either for their job or for personal projects, but still not liking it, which is perhaps another way of saying they have mixed feelings about the whole thing. https://pollariser.eu/poll/04fa283a-fed9-4b41-a3fe-4e6a3056e6ac New PollThe end of the year is in sight, which tends to put people in the mood for reflection. So a simple question this time: "How was your 2026?" It’s a complicated time, in the world, and in tech. But that’s also something that isn’t experienced by everyone in the same way. I’m curious against that backdrop how you, dear reader, experienced life this year. HeartConfI sent out a HeartConf newsletter before my trip: HeartConf News: Venue, Tickets & Sponsors. tl;dr: Early Bird Tickets are on sale, and the search for Sponsors is open! It’s been relatively quiet on the organising front as I just took a long break. I did manage to send out a few batches of HeartConf postcards to companies and individuals.
BookmarksResearch published in the Harvard Business Review
Solidarity forever. I thought this was fascinating. It chronicles the astonishing rise and fall of "short drama" as a format in China. These are 1-2 minute clips, often shot in vertical (portrait) mode, watched in rapid succession on dedicated mobile apps. The market saw explosive growth in ‘23/‘24/‘25. Then, AI generated series flooded the market. A short case study of how AI is going at Google. So far, it seems to be removing rather than creating business value. PictureA picture of Chiayi county, western Taiwan.
SupportDid you enjoy this email? Share it with a friend! This is a reader supported publication. Please consider becoming a paid supporter. ![]() |