"Just Plug It In!" - AI Contraptions from Hell

I've already broadly articulated my thoughts on developing with AI. That being said - I just can't help but see the value in it, I think everybody does. It feels like most organisations are running around in a panic trying to figure out how not to fall behind, while trying to avoid the crushing pain of rushing ahead and into a trap.

The thing that I think it's fairly consistently missed is how to integrate AI tooling into an organisational estate. Everybody argues over the best tool:

Is it Claude?

Is it ChatGPT?

Is it Gemini?

Is it Copilot?

...but those discussions are, honestly, important at a more microscopic level - when you need a model to do a job. Not when you're architecting enterprise AI.

When the notion switches to building AI systems that are forward-thinking and trustworthy in an organisation, then a much wider range of practices have to be considered. Yes, AI is a generational leap in our ability to interact with systems, but that doesn't mean it doesn't need great systems design and architecture. In fact, since AI is, for the first time, a foray into non-deterministic systems where the system needs to be incredibly highly optimised, it's arguably more important that we ensure it can get all the information it needs cleanly and with minimal effort. So, where to start?

Centralised Data with Logical Federation

Understanding that physically centralising and logically federating data matters for AI systems. Often we see designs like this one:

This approach is Copilot simply being plugged-in to anything and everything - and it's often the outcome of relentless PoC'ing. This creates an environment where cross-functional questions need to hit three or four different places to find answers - before expecting an LLM to perform reasoning on the outcome, which it can't. This leads to hallucinations and, eventually, chronic trust gaps in the system for end users.

The need is to provide semantic and logical information (ie, some text-based information and a glossary of terms that the business agrees to). Tools like Fabric and Databricks offer the ability for all data and supporting materials to be kept in the same place, which reduces the need for multi-hop architectures. Instead of a plugging the expensive AI engine into various data in various locations - you bring the data to your own estate. In doing so, you can add policy, regulatory, and legal documentation, along with data glossaries, to ensure the model understands the 'language' of the business, so that it can be highly optimised for the actual task of retrieving information based on the prompt from an end user - who is also likely talking in the language of the business. It should come as no shock when "Large LANGUAGE Models" are good at this sort of activity - assuming all the definitions and documentation is available for it to refer to in each prompt.

Is the Model a Fish or a Monkey?

What's that quote? "If you judge a fish by its ability to climb a tree..."?

All these models can be a bit 'horses for courses'. I've seen organisations plug Copilot into back-end estates simply using the native connectors. "Native Connectors" is a great little bit of jargon that makes it feel seamless - but it's anything but! Copilot is not a good text-to-SQL engine, in fact it's not really particularly well-optimised for anything other than typical office productivity tools - and even then, that's the premium version. Instead, the architecture should look to use the right model for the right job. Data agents in Fabric offer performant text-to-SQL models that can be optimised over semantic models (from Power BI). In doing so, it means that you can ask for information and there be a really strong chance that the response actually be what was requested - not a misunderstood 'near miss' hallucination.

Then there's the wider development of these varying models for really explicit use cases. Microsoft Foundry gives the ability to pick a model and develop quite a nuanced agent that can be used for a specific purpose. Need something to read standardised documentation? Foundry can do that. Just pick the appropriate model, point it to your documents in OneLake (the storage account underpinning Fabric) and let it do the work it needs to do. It pays to build things for the task at hand - not just "plug stuff in".

Put a Pretty Bow On It

If you have a physically centralised architecture, with logical federation, a semantic layer with a range of supporting documentation, and fit-for-purpose LLMs that actually suit the task they're being asked to do - you're almost totally there. This next step might be something that has to be built (and maybe an AI code tool could do that!)

In Microsoft's estate, these various smaller models can all be published and made available in a very convenient way - paving the way for model management centrally within the organisation. In true, strategically ideal fashion, Microsoft then allow those models to be surfaced with Copilot - which is a fantastic way to tie it all up.

Need data from your pre-existing data models? Just ask in Copilot, it'll get it from the right place.

Need to know what some government regulations are? Just ask in Copilot, it'll farm the job out to a high-context reasoning model.

Need to understand what market trends are for competitor products? Just ask in Copilot, it'll use a small language model (SLM) to efficiently collect and parse that information.

It'll look something like this:

It really is a super slick mechanism - and best of all, you can modularly replace these tasks with improved models as time passes. You shouldn't be locked down, because your architecture always allows you to ensure you're working with the best models available for the job.

In summary, don't be tempted by the "just plug it in" crowd, or your token count will go crazy. Start with a good architecture, keep building, and reap the benefits of a highly trustworthy, enterprise AI solution.

Or bin it all off and go live in the woods - up to you!