My Software Engineering Principles

This post originally appeared as an internal team communication at BetterUp.

Here are some of my personal principles regarding individual contribution to delivery.

horses, not camels

“Tomorrow, sell your camel and buy a horse. Camels are traitorous: they walk thousands of paces and never seem to tire. Then suddenly, they kneel and die. But horses tire bit by bit. You always know how much you can ask of them, and when it is that they are about to die.”

―Paulo Coelho, , The Alchemist

When building systems, observability needs to be a first class principle. Observable systems are transparent. You can peer into them and begin to understand their inner workings. We must understand our systems to anticipate issues and prevent catastrophe. Observability enables deep introspection and accountability. When we build a complex system it takes on its own behaviors. The goal is then to close that gap between how we think the system works and how it really works. We can then understand how much we can ask of the system and what it will do under certain inputs. Building a horse means we can peer into the system, identify when it getting tired, and take preventative measures to avoid exhaustion. Things you can do to improve the observability of a system is:

  • logging
  • event sourcing
  • alerting
  • storing intermediate results
  • readable code
  • tracing
  • automated profiling
  • visualizations (see Netflix’s Flux )
  • interpretability (in the case of machine learning)

say what you mean. ask for what you want

“Well,” the talkative man said, “perhaps I should ask a different question. Be more specific, that’s what my mother always said. Say what you mean and ask for what you want. What’s the story of you getting that first brand of yours?”

- Brandon Sanderson, The Way of Kings

As an individual contributor it is your responsibility to get to the truth. The underlying reality you are faced with. Not just the symptoms of an issue, but the thorny, ugly, hairy problem causing the issue. Treating symptoms without diagnoses allows that infectious bacteria to grow stronger. It will rear its ugly head again and with more dire consequences. Getting to the truth requires radical inquiry and reflection. It requires asking hard questions and asking for help. Different perspectives from different angles and expertise are necessary to understand reality. Back and forth questioning clarifies the problem domain. Being direct to share your own perspective and empathetic to understand others perspectives is what it takes to get to the truth.

problems over solutions

"Because I love you, I must kill you."

-Ernest Hemingway, The Old Man and the Sea

We like solutions. We jump to them. We take comfort in them. Solutions are temporary. They are the trophy fish eaten by sharks on the journey to land. But the struggle against that fish, and how we performed, will stay with us forever. Problems are forever. There is no such thing as a solved problem. Systems don’t solve problems. We build systems to deal with problems, but they are always there. But we can do our best to build a solution that allows us to focus on other problems and to minimize their relevance. But, solutions will come and go. Problems are opportunities. Opportunities to challenge yourself and grow. To fight that trophy fish for days on end with a nearly empty bottle of water and fishing line cutting into your back and hands for days. Problems are there for you to prove to yourself you can do it. Problems, as with Santiago and his prize fish, you must love and respect. But, ultimately, you must kill them even with uncertain outcomes and with only your resolve and ingenuity.

don't talk about it. be about it.

“It’s not easy to find the Philosopher’s Stone,” said the Englishman. “The alchemists spent years in their laboratories, observing the fire that purified the metals. They spent so much time close to the fire that gradually they gave up the vanities of the world. They discovered that the purification of the metals had led to a purification of themselves." ...“It’s only those who are persistent, and willing to study things deeply, who achieve the Master Work. That’s why I’m here in the middle of the desert. I’m seeking a true alchemist who will help me to decipher the codes.”

- Paulo Cuehlo, The Alchemist

Talking is important to align, clarify, and plan. But if you aren’t pairing that with action then you are avoiding the problems. Talk can only take you so far and action is required to make meaningful progress. Over-collaborating is possible. When you call repeated meetings or meetings without first doing your own homework, you are potentially using other people to do your work for you. Collaboration works best when experts come together on a problem and tackle it from different angles. But the key is to be prepared. That means digging into the problem and wrestling it. It requires leaving your comfort zone and joining a caravan into the desert to seek mastery. Asking others for input too early, without appropriate research or understanding, can devolve into group think. Group think can lead to premature solutions since everyone is trying to solve a problem before the meeting ends so it doesn’t feel like wasted time. Meetings are best used as ways to review and discuss findings from actions. Those findings then inform next steps and responsibilities for the group. I find people are worried about building the wrong things. That is a valid concern but it assumes that whatever you build is meant to be used. The purpose of preliminary action is to try ideas to identify further understand the problem at a deeper level. Once you understand the problem, throw away the prototype and build a solution. It wasn’t a waste of time. It was the best use of your time.

imagine forward. reason back.

Put imagination, reason, and action together and it looks like this:

  • You imagined your future.
  • You reasoned backward.
  • Then you acted.
Of the three steps, imagination and actions are usually easiest. Imagination and action are what most people do everyday. Imagining and action go forward. The way people are used to going. But reasoning goes backwards. ... Reasoning ties together imagination and action.

- John Braddock, A Spy's Guide to Strategy

Whether you are building a system, working on a prototype, or ideating, a helpful strategy is to imagine forward, reason back. Put yourself into the future and look around. Notice the details of what you see and start probing.

How did I get here? What do I like about this? What don’t I like about this? Imagine you are trying to start a colony on Mars. Put yourself there. You are standing on the Red planet. What do you see? What is around you? How did you get there? Maybe you notice a rocket in the distance. Okay, how did the rocket get there? Well it had to land from space. There must have been a journey from earth. That means there must have been supplies, engineers onboard, communications equipment to talk to earth, some sort of navigation, etc. Does the ship go back to earth? If so, how do we refuel it? What kind of propellant does it need? Can it be made on Mars? And so on and so on until you find yourself back on earth in present day. The answer to those questions will lead you to Mars.

I know this analogy is probably not relevant entirely to you but the technique is. If you are developing an service, tool, library, user interface, put yourself into the intended audiences shoe.

Imagine using that API. What kind if questions would you want to answer from it? What would you like to do with it? Are the endpoints consistent? Are they easy to use? Is it easy to extend and adopt? Does it look like other APIs? How fast are the endpoints? Do I need to make multiple requests to answer the questions I want? What kind of models are needed to support this API? Does it need a database? What language is it written in? Is there a client library? Where is the service deployed? How do I view the logs? Will it alert me when something goes wrong? Once you start down that path of questioning, you will be led to clarity. It is easy to get stuck thinking about potential solutions or problems. Fear, uncertainty, and doubt (commonly referred to as FUD) stifle motion. Moving past fear, uncertainty, and doubt puts you in motion. And once you start moving, it is hard to stop.

multiplicity

Whatever the starting point, the matter in hand spreads out and out, encompassing ever vaster horizons, and if it were permitted to go on further and further in every direction, it would end by embracing the entire universe.

-Italo Calvino, Six Memos for the Next Millineum

Multiplicity is the idea that everything is part of a system of systems. Dependencies and connections are everywhere. But only through time will BetterUp internal confidential materialthose details emerge. And you have to listen to them and absorb them. After time and reflection you will be able to see the connections and reach a deeper understanding.

In engineering, we tend to think of dependencies as complexity or as issues. That may be true but we feel the complicate things only because we like tidy issues. Ones we can fit neatly into our head and solve. Everything, however, is interdependent. Your actions, you system, your designs will influence some other part. This is why the idea that pops into your head is normally insufficient. You haven’t sat with the problem long enough to allow those details to surface and to digest them.

You may hear this concept pop up as second-order thinking. Moving past cause-effect to visualize downstream consequences of an action. Moving past the immediate concerns can help you make better decisions. Something that may cause immediate discomfort can have tremendous reward. While something that looks immediate rewarding can cause tremendous disaster later on. Actions viewed through the lens of immediate gain will be doomed to suboptimal return. Playing the long game, by paying excruciating detail to the present, is how truly innovative things are built.

consistency

When Lieh Tzu brought the medium once again, he emerged saying, “Your master is inconsistent. I can’t read anything from his face. Get him to straighten up, and I’ll give him a reading.

-Chuang Tzu, Answers for Emporers and Kings (from The Essential Chuang Tzu by Sam Hamill)

Consistency for programming really comes down to does this part of the code feel like other parts. On a system level, it is when one subsystem feels familiar to another. Sounds simplistic but ends up being really important when you have a really large project. It lowers the cognitive burden of diving into other parts of the code and system. It means you can scale the system without necessarily scaling the team. Now there’s two aspects of consistency for programming: (1) code written according to norms of the language, which really only comes from experience with a language, and (2) is consistency within a codebase, which comes from experience working on same system for an extended period of time.

There are practices and tools that can help accelerate that consistency. As a practice, code reviews where more developers are reviewing the code of other developers such that inconsistencies can be remediated. For tools, auto-formatting your code such that, stylistically, the code looks similar across different areas. This latter point may not seem like a huge deal but it removes a lot of difficult and opinionated conversations around code style and code reviews become more focused on the content and not so much the style.

Think about user interfaces. You notice when something feels out-of-place or not quite right. It jars your thinking and it becomes the focus of your thoughts. It doesn’t feel great producing an inconsistent interface. Developers should feel the same. Inconsistencies are glaring and can distract you from understanding code. And even going to layer higher, inconsistent APIs are inconvenient to use and, potentially, difficult to understand. When we expect an API to behave a certain way and it doesn’t we break the principle of least surprise. We can say that it’s inconsistent with our expectations. Maybe it’s unfair to have that expectation to begin with but this is where naming it’s hard because they mean something sets up that expectation essentially for what’s going to be the behavior.

holistic optimization

If you divide a problem up the way things are down nowadays, everybody gets their little specialization, you have the person that’s going to work to make this and the person that’s going to make the best that. And they’re going to be coupled together some way with an interface. But in many cases there are built-in inefficiencies from that.

―John Carmack, , Tech Talk with UMKC-SCE

Before making any part of a system faster, leaner, or bigger, you must understand how that part fits in the whole. This becomes especially apparent in optimization. You can profile a slow area and spend energy optimizing it only to have little to no impact on the end experience. To dig deeper you have to put yourself into the end experience. Similar to the imagine forward, reason back process, start asking your self “what happens next?” As you step through part of the experience, you will make your way through the various services and code bases. You can then begin to see and understand the system flow. This process takes time but is extremely rewarding in terms of the knowledge you gain about the inner workings of a system.

Instrumentation can really help here. If you have automated profiling for all your services, then it makes it easy to leap between services and understand how it decomposes into code and the cost of any procedure. Where I’ve seen this fall apart is when there are many services and dissimilar tools for instrumentation. Having a single tool or service that can be used to investigate the performance of all of your services reduces that time to find bottlenecks significantly. Using a single tool reduces the cognitive cost to go answer performance questions rather than them asking others to investigate on their behalf.

review your work. prioritize reviews.

Design reviews, code reviews, any review. It’s a blocker. If someone is waiting on you for a review, then you are blocking them. This mindset should be the default but it requires some mindset shifts before putting code into review.

Before putting code into review, ask yourself if you if this is the best you could do. Did you impress yourself? To force yourself into this mindset, try opening a review or pull request without reviewers. Go through the code and leave comments for yourself. Fix the issues you found. Then add reviewers.

Another technique to help reviewers is to create smaller reviews. Smaller reviews are approachable and invite a deeper level of participation from the reviewer. Large reviews take preparation and discussion prior to review. Be kind and create smaller reviews. Be prepared for more comments. Setting expectations prior to a review can help expedite the process. If the code is meant as a prototype, be upfront about that. It is okay to defer work to a future iteration, but if a reviewers expectations differ from yours then the process will draw out. When there is deferred work or known deficiencies, add TODOs in comments. That lets reviewers and readers know that this could be better or there is an unhandled situation. If you are using a work tracking system, like Jira, I suggest opening a ticket and leaving the ticket slug in the comment. That increases the visibility to management and coworkers about known issues.

find the holes

In safety critical systems, there is usually a long development cycle leading up to a big event. For rockets, that’s a launch. During that development cycle there is continuous testing to find holes in the system. Failure to imagine scenarios and to answer unknowns can lead to mission failures. Ultimately, potentially a loss of life. Many of the decisions are moral in nature. Because of that, there is increased pressure to do right and weed out the wrong.

For highly available, reliable services, the same level of imagining failure states and testing them is required. As an engineer, you must imagine the different ways a failure could happen. If you are making a API call, imagine what happens if it doesn’t work. Or what happens if it returns back unexpected responses. Handle those cases. At scale, no matter how improbable, you will hit that situation. This can be one of the hardest mindsets to develop and build discipline around. We can go look into the source code of another service and see how it works.

Once we know how it works our mind is polluted and can prevent us from implementing defensive programming procedures. Which seems contradictory to some of my earlier statements regarding observability. But observability only helps you investigate the current implementation of the system. We can’t know how a service or system will change or behave in the future. Defensive programming is our guard against an evolving system and to enable system evolution. Anticipate those failures and handle them. They aren’t edge cases, that is how the system behaves.

grow systems, don't build them

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

―John Gall, , The Systems Bible

Systems aren’t built. They are grown. This may be the hardest realization to work into your day-to-day activities as an engineer. We want to plan things out, document the particulars, and build a complex system in components that work together. Part of this is because the nature of work in an organization. Sufficient planning and coordination must be done prior to implementation (to some degree) such that different teams can work on different parts and then synthesize the result together.

What tends to happen in reality is that the system is overly complex. Pieces don’t come together quite right and they are made to fit through patchwork changes. When a system is built, there is a specification. When it grows, there is an evolution. An ever-present change. The system can accomodate changes because that gardeners knew that the system was going to evolve. With specifications, there is a right and there is a wrong. Changes become difficult to swallow. Engineers grumble that the specification is poor.

Growing a system means to start small. Very small. A seed. The seed sprouts just as our understanding of a problem space sprouts. As we spend time with a problem, the system grows. It handles the new information and weeds are pulled. Eventually, the once-seed forms a beautiful, fruit-bearing plant that nourishes rather than requires nourishment.