The Pitfalls of Organisational Paralysis

We do love a good 'right' vs 'wrong' argument in the data world. Python vs R, Kimball vs Inmon. In the same day, when trying to come up with some "best of both worlds" approaches, I came across two separate pieces of advice; one encouraging me to centralise, and the other (a principles post from dbt Labs themselves) advocating for the implementation of a data mesh!

The reality of these approaches is that neither is correct, yet neither is wrong. Organisations simply need a defined strategy, a series of milestones to achieve to enact that strategy, and then a series of steps needed to meet those milestones. The technological paradigms broadly don't matter. Not a popular opinion, but definitely a hill I'll die on.

I've seen it over and over again, but organisations tend to fail because of a lack of clear strategic approach - the 'JFDI guys'. I saw the phrase "programmatic discipline" the other week and I think it captures perfectly what organisations are poor at. We can't push on with a programme of work to achieve a strategic goal, because we need to see things landing to feel like it's moving forward.

Why is this?

Often, it's driven by the "fool me once" outcomes where organisations, or even just individuals, have seen black holes before and are committed to avoiding another at all costs. Pouring effort and capital into a lost cause is terrifying for any leader - the idea of having to go back to the board and admit to sunken costs, is not a welcome one. Sometimes it's because the urgent is simply always prioritised over the important. I'm working on something like this right now, where a new platform to drive significant cost reductions, effort reductions and drive downstream value is constantly de-prioritised to stand up legacy reports that are, in many ways, totally wrong anyway.

The way out is simply commitment. Data leaders need to commit to showing outputs, even if they are light in content, at regular intervals - this will ensure that leaders are much more comfortable knowing that work is underway. Then leaders need to ensure that they're setting the strategic direction and defending progressive work from the onslaught of demands on older systems - thereby ensuring strategic delivery.

At all costs, organisations need to absolutely ensure that they do invoke paralysis on themselves by not developing for future at all. The tools don't matter, having a strategy and ensuring delivery does.