Why a Sucky Tech Stack isn’t the Problem

It doesn’t matter where you look in the data sphere, which tech stack you’re using, or which architecture you have in place – somebody thinks it sucks.

Everything always sucks

As a user who has slid from place to place across the technology landscape, I have to say I find a lot of joy in the various idiosyncrasies of each platform and its invariable plethora of fanatical followers; always ready to explain why everything else isn’t as good. The problem is that I simply don’t agree. I honestly can’t say I have any recollection of ever agreeing en masse with any hardline advocacy for one set of tooling.

Now don’t get me wrong, I’m not getting at people who advocate for stability. We’ve all been there: organisation X is dying under its own weight because it has 225 dashboards being held up by an ex-employee’s Entra ID running on a laptop that’s now kept in the server room to avoid people unplugging it. In many cases there’s a reason to advocate heavily for a good tech stack, and often it’s critical to the long-term survival of the data function, but the bit I struggle with is the ideological view that 95% of the needs of most organisations aren’t met by five or ten different tech stacks.

“Databricks is on the way out!”

“Snowflake is expensive!”

“What even is Synapse?!”

In too many cases, people are quick to point to the perceived failures of other systems, rather than learn the ins-and-outs of that system and where it might add value. I’ve said for years – I think Synapse is a sneaky great candidate for cost-efficiency for small data functions. Finance, or HR, or Project Management teams just looking to get the numbers straightened out. Almost always under 10m records in the fact tables – it’s a dream. You run a spark pool batch job for 30 minutes overnight and then you live on the serverless pool, which is as cheap as it gets for importing data. It plays perfectly with Azure. It works almost natively with the wider Microsoft ecosystem (like Business Applications).

Yet, the excitement that takes over the masses when they explode into action over a perceived “lesser” tool is palpable. Hundreds of online figures of influence offering their positions on Power BI vs Tableau, Python vs R (an old classic), Databricks vs Snowflake – the battles are endless. And it doesn’t end there!

When we’re done arguing about tooling, we’ll get started on architectures. ELT vs ETL – wait a minute both of those are dumb, medallion architecture is king. Wait a minute, medallion architecture is exactly the same thing!

And still, this all comes in spite of the fact that you’ll be able to achieve 95% of organisational use cases, more or less straight away, with just about any of the above.

I blame FAANG

The constant obsession that we have the same problems as the chosen few is just nonsense. Some people may be reading this and thinking, uh, well, yeah, we have much, much more data than they do – and sure, I believe you – but I do not think that is the norm. In most cases, people are working with datasets that are simple and fundamental to every organisation – finance, HR, sales, etc – and they’re struggling with it, too! Data professionals are fed a steady dose of “Ex-Google” employees offering keynote speeches at data events (many of which are excellent). While these are interesting, and a lot of the content offers food for thought in the abstract, it often deals with technological questions that are so far different from the issues of most data people:

1. Does the data model meet the reporting need?
2. Is the pipeline productionised?
3. Can we get this report shared with the business?
4. How do we know the numbers are right?

Very little in the operational delivery of positive answers to those questions has anything to do with the tech stack. In fact, it’s almost always the people involved getting things over the line. Great project management, great people management, great stakeholder management. These are the things that deliver great products, great projects and great relationships with organisations.

So, the ask is simple...

Next time you want to roast somebody’s idea of a good tech stack – stop. Think about whether it matters and get back to the task in hand. Data has to deliver value, or it ceases to be something worth investing in – and I’d bet if you’re not doing that, it’s probably less about the technical nuances, and more about the delivery.