How Things Should Be Done

The following contains concepts and principles which I believe to be fundamental requirements for designing and implementing “good” complex systems. It is by no means an exhaustive list. I still have a good bit of editing left to do. But it offers some insight into my perspective on what matters in software development.

Exceptions to every rule

These can be read as “rules”, and doing so may produce good results in most cases. However, no rule works as intended everywhere. IMHO there are no perfect rules|principles|methodologies|programming languages|etc for developing complex systems. Those who insist otherwise just haven’t hit the limits of their sacred cows yet. Those who know better retain an open mind, and stay willing to consider new ideas, adapt accordingly, and keep learning.

Original Tidbits

This section is a loose collection of maxims and aphorisms I’ve developed over the years…

Performance / Optimization

Zeroth law of optimization: It doesn’t matter how fast you can get the wrong answer

You can’t improve what you can’t measure

Reliability in Complex Systems

Top three rules for maintainable complex systems
     Don’t repeat yourself
     Don’t repeat yourself
     Don’t repeat yourself

A foolish consistency is the hobgoblin of little minds, but not all consistency is foolish

Provenance matters

Testing

Everything you build will get tested - the only question is when and by whom.
Clients and customers are quite adept at finding whatever you missed.

If you want to reach the moon, test with your best.

If you have to fix something twice, you’re doing it wrong.

Friends don’t let friends ship regressions

Adopted Precepts

Concepts I generally endorse, although without the witty phrasing

Balance and Harmony

All these are beliefs I hold strongly. I don’t insist that others do - I’ve often worked with talented people I sincerely respect who think quite differently on a few of these. There’s nothing on this list I consider immutable - other than perhaps “exceptions to every rule”. Maybe.

OTOH if you refuse to discuss or consider dissenting opinions you’re dealing with dogma, not science or engineering. That said, questioning is good and should be encouraged (although there are some reasonable caveats).

Complexity and Compromise

ALL complex systems must deal with uncertainty (even the awful ones - typically by crashing) and typically juggle a formidable list of requirements. Consider the “Good, Fast, Cheap” cliche - simple (at least on the surface) but virtually impossible to pull off in practice. The requirements for complex projects will inexorably contain a few desirable deliverables which are mutually exclusive or simply out of reach. They must build upon a collection of decisions related to tradeoffs concerning these requirements (representable by a DAG  if you’re really lucky).

But invariably, whether you blame Murphy, the uncertainty principle , TNSTAAFL , or the whims of fate, the chances of finding a decision set that completely satisfies all the requirements for a complex system may deal in probabilities on the scale of rolling dice until the heat death of the universe.

For example, I never encourage violating DRY, but there are legitimate cases where I have (and would again). Sometimes disparate toolchains prevent simple sharing, and development of custom DevOps integration to keep things DRY reliably wouldn’t be cost effective. Some forms of code sharing may breach regulatory or security requirements regarding isolation between development groups responsible for different areas.

Perhaps you disagree with me - in fact, I suspect I might agree with some disagreements. For example, we may have significantly different definitions of what constitutes a “complex system” (even without contradicting any of the above). Or you may object that nothing unattainable should be in the requirements in the first place.¹ This could lead to a plausible scenario in which one might achieve a “perfect decision set” without winning the cosmic lottery.

And yet, doing so reinforces the primary premise, which is that there are no perfect rules. We simply lack the means to comprehensively and unambiguously define rules and requirements with absolute clarity and no chance of misunderstanding. Chaos and uncertainty are intrinsic, transcendent, and unavoidable. There will always be exceptions.

So embrace the chaos. Learn to love tradeoffs - they can be a positive source of differentiation if you can find a better decision set than the competition. Accepting compromise doesn’t always mean you failed, and it can mean you’re on the right track towards success the inflexible can never reach.

DRY

I’m rather fanatical about the value of the DRY principle (Don’t Repeat Yourself). I consider anything that violates it to be at best a source of technical debt. It almost invariably increases risks within the system². It also often becomes a major constraint on the ability to scale the complexity of larger systems.

Short is Sweet

Brevity is the soul of wit

I have only made this letter longer, because I have not had the time to make it shorter³

All else being equal, shorter is usually better. There are exceptions, particularly with naming. But expressing all that you need to express (whether in prose or code) in its most concise form takes effort, and usually some time to edit and reflect.

If you’re reading this document (and many of the others on this site) in the early summer of 2025, there’s still a lot of overly verbose content. It will take some time to edit things down to a reasonable size, but the existing content will get smaller.

I’ve observed a similar dynamic with source code. Some of the best code actually gets smaller over time, while increasing in flexibility, reliability, and performance. However there is an additional dynamic in play, beyond the extra work required for concise prose. With code, there’s often a lot you don’t know ( interactions and use cases in practice, performance, etc…) until you’ve gotten the first cut up and running. Intentional observation and analysis over time can reveal hidden correlations and inspire better ideas.
After which judicious refactoring can result in quantifiable improvements in capability and performance and a reduction in size and complexity (especially in interfaces).

Documentation matters

This is important enough to have its own page

Testing matters

Also worthy of its own page

Interface trumps Implementation

Yet another worthy of its own page, which will be coming soon.
A few highlights:

• quick & dirty subsystem implementation behind a well designed interface can dramatically improve development velocity without adding excessive risk.
• A perfect implementation with a poorly designed interface is a recipe for disaster.

or…

Write Readable Code

Amateur and junior developers (and even a few brilliant seasoned ones) may see writing code as a way to show how clever they are. Or be obsessed with code formatting (making it as compact as possible or mind-numbingly dispersed), or employing the latest flavor-of-the-month syntactic sugar every chance they get.

Older, wiser, experienced types have had to read far more code than they’ve written, which instills an appreciation for the value of readable code. In many cases, readability of the code is arguably more important to the success of a project than more commonly sought “priorities” such as performance optimization (especially outside the fast path - you did identify your fast path first, right?).

There are many ways to improve the readability of your code, some of which are language specific. Someday I’ll write more on that. For now, I’ll offer a few simple and near universally applicable tips.

• Names matter.
  • void DontFearDescriptiveNames( int onTheOtherHandYouCanTakeThisToFar );
• Coding style matters
  • which coding style you use - how you manage indenting code blocks, whether you use camelCase or under_scores, etc. - is ultimately not that significant
  • what is significant is picking a style and using it consistently

“Simple things should be simple, Complex things should be possible”

• Attributed to Alan Key, see Alan Kay’s Approach to Accessible Complexity 
• Back when C++ was young and Rogue Wave was a major player, there was an oft repeated mantra which went something like “Make simple things easy, and hard things possible”. While researching its origin to see if that might have been an original saying (we had some very bright bulbs in those days) I found the Alan Keys quote, from which it was almost certainly paraphrased.

Successfully employing a methodology is more than following checklists

I’ve more than once butted heads with well-meaning Agile fanboys who insisted on dogmatic application of commonly Agile-associated buzzwords ⁴ like YAGNI⁵ and MVP⁶. Somehow they simultaneously missed the significant role of communicating customers and understanding their needs in the Agile methodology⁷

Obviously, implementing features you don’t need yet is usually a bad thing. Best case, it’s an opportunity cost -and that’s only if you actually get it right and it proves useful over time.

However, designing interfaces/APIs which accommodate (but not implement) known, rational future requirements can be good. Rejecting such from knee-jerk adherence to YAGNI/MVP dogma is… not.

And citing YAGNI/MVP as a justification to avoid the upfront work necessary to provide a DRY interface wrapper around high risk business critical features - for example, making sure all financial transactions against the ledger holding the account of record is centralized, secured, and thoroughly tested - is a waste of time because copy&paste reuse of SQL code which modifies the ledger “is quicker and easier” and “we should be Agile” and “we’re stuck with junior developers” and “you’re over-engineering things” - in an application for PCI compliant payment processing⁸… Ok, deep breath, dismounting soapbox, putting that pet peeve back on the shelf.

This demonstrates some of the challenges of maintaining balance. Names were omitted and some details changed to protect the innocent (I never proved them guilty) but needless to say these were inspired by actual experience, in which I strongly disagreed with some other architects. And yet their motivation wasn’t wrong. YAGNI⁵ and MVP⁶ are both reasonable and valuable principles to employ in the design/development cycle.

Ultimately, ongoing disagreement between leads and high level team members can cause more damage than simple choosing either option. Sometimes you just gotta agree to disagree, and if you can still achieve consensus and present a unified front⁹ then you’ve found balance. If you can’t, maybe it’s time to move on.

Notable Mentions

Domain Driven Design 

I somehow missed out on this - guess that can happen if you spend more time actually building stuff instead of playing buzzword bingo and figuring out what new hotness you can leverage for FUD to get more consulting clients by convincing them they need to rebuild everything from scratch (looking at you, Y2K and Y2K era Java “Enterprise” apps…). I only recently encountered it when researching the origin of Simple things should be… - which is a shame, because what little I’ve seen so far has quite a bit in common with some of the design decisions in my current projects like the LCPF

Upcoming additions

Other significant concepts I’ll be expanding on :

• Using Units
• Domain Language