Written on June 15, 2020.

Tags: systems, engineering, readings, complex, organization

Table of contents

• Complexity and engineering
• Do we always want less complexity?
• Avoiding more ‘complexity’
• Conclusion - at this point

Before jumping on the options we have at hand, let’s try to summarize previous post with a table.

Complexity and engineering

If we adopt a measure of complexity that looks something like the entropy and measure the number of possible configurations of its components, these systems seem to be sorted in an increasing level of complexity. The further you go down in the table, the more details you would need to know to be able to replicate one of these systems.

Do we always want less complexity?

This might seem like a crazy question and it might seem obvious that we might want to ‘keep things simple’.

This is definitively something we hear often. The notion of ‘emergence’ is about complex system showing behaviors than none of its part have and thus the whole being more than the sum of its parts. Nothing says that the value one derives from these behaviors must have a positive value or not.

Without much evidence to back it, I would argue that there seems to be a positive correlation between the level of complexity of a system and its ability to get stronger as it faces more challenges. I cannot say about the causality though.

Robustness and resilience are pretty commonly desired properties. Antifragility is definitively even more desirable.

I have not yet come to read about a small system that can be used to simulate the emergence of an antifragile behavior - let me know if you know one. For robustness, if we make the leap to see the existence of long-range connectivity as a mechanism to get to robustness - ability to withstand local disruption by leveraging resource from another part of the network for example, behavior as demonstrated by graph percolation seems pretty interesting.

In the absence of evidence that more complexity is the solution to get to antifragility, one might be careful to add parts, interactions, … deliberately and be happy with the level of complexity it already reached.

Avoiding more ‘complexity’

In the previous section we have discussed the point that it is not clear that all complexity is bad. Before going further, let’s remind our self the paper No Silver Bullet - Essence and Accident in Software Engineering by Brooks, Jr and the distinction he makes between ‘essential’ and ‘accidental’ complexity: it must has to be ‘hard’ to some point - to solve your problem, but there is no need to make it harder than necessary and that’s what tools are for.

Let’s get more concrete now, but still refer to another paper Out of the Tar Pit by Moseley and Marks. This paper comes as a response to Brooks’ and argues that complexity in software engineering can be tamed by keeping states aside from the logic - as functional code and leverage referential transparency and composability. In these recommendations one should recognize patterns that are used again and again in software system design: layering, segmentation, decoupling, composition, modularization as ways to assemble parts in a controlled manner.

MSA (Micro Service) typically leverages this model by creating a bounded context: focus on one area domain, separating states from the logic, communicating by APIs (like REST) to facilitate the composition. This structuration is not only for the software components.

Modern software development organization of work (as adopted in SaaS and Internet companies) leverages these principles by creating bounded context at across all the layers: value proposition, domain expertise, ownership and eventually code and operation with STOSA (Single Team Owned Software Architecture) for example. These external boundaries of these teams create some segmentation between the inside of the team and the external world that limit the perturbations and helps create a somewhat less complex environment for the team members. But make no mistake, the entire system this is contributing to create is definitively one that falls in the ‘system of systems’ category and it has its properties with changing context, multiple changing and conflicting goals, need to design for continuous quality as things are always ‘ON’, full scale prototypes don’t exist and you have no idea what a change would cause until you have experimented it. Assuming that an average human can only keep so many things in mind and so many relationships, this way of composing organization by connecting nucleus that starts at ‘human scale’ with a limited number of team members (2 pizza teams), a delimited perimeter, limited number of services with themselves bounded scope, seems to be fairly reasonable and yet build constructions that can achieve more than their parts and demonstrate collective behaviors that none of their parts have.

Conclusion - at this point

Can we really remove complexity? Maybe that’s not even desirable but we might want to limit its proliferation when possible.

So many other questions come to mind: Can we shift complexity around? Does that even make sense? Can complexity be a local characteristic that could be distributed unevenly? entropy in isolated systems can, but they need to be isolated… What are the levers to control complex system? Can we engineer complex systems?

If you come to anything interesting around that, please send it my way.