Admit It, Your Azure DevOps Estate is a Disaster, Isn't It?

I think most organisations really have to admit it, Azure DevOps probably does an ok job of managing your deployments, it's likely got a fairly messy, but perhaps functional git repo, but the boards - good God - the boards are a graveyard of optimistic estimations and long-forgotten tickets. If you're being honest, and be honest, how many times have you stumbled across a ticket and said, "oh, we can just remove that, it's not needed now."

Agile boards are treated as a massive burden. They're often spooled up with the best of intentions, tasks dutifully created and records filled out, but loads of data just ignored in the end. Crucially, we almost never use the underlying data for what it should be used for: predicting what we can do next time.

In your data practice, one of your core manifesto principles should be something along the lines of: 'trust is our primary metric'. If the team feels that their work-tracking tool is just a stick to beat them with, they will feed it garbage data. We used to get that CONSTANTLY and it is an absolute JOKE of a discussion. "Did that yesterday. Took eight hours. Thanks, bye."

But that's where we have to change our mindset. Treating the DevOps board as a core system in its own right, designed to protect the team’s time, we can turn it into an incredible asset for estimation, and in turn, forecasting - with forecasts based on previously completed work and tickets and the information surrounding those.

Welcome to the world of Reference Class Forecasting.

I'm going to stick explicitly to DevOps, the board, and the process here - we’ll get into the fun stuff later, like how to take your historical DevOps data and build an auto-estimation engine using machine learning. But before we do that, we actually need reasonable information.

Here is how you capture the right data, without turning everything into an admin exercise.

We don't have to overcook the data entry

To forecast accurately, you need to abandon the "inside view" (a developer guessing how long something will take based on today's optimism, or not guessing at all because he's absolutely obsessed with not being caught out, as I often find) and embrace the "outside view" (how long similar tasks actually took historically, both chronologically, and in terms of 'touch time').

To define a "similar task", known in statistical modelling as the 'reference class', we need strict, governed outputs. Forget the free-text descriptions. You need these core fields:

• Activity Type: (Design, Documentation, Development, etc).
• Tags: This must be a strictly controlled vocabulary (e.g., API-Integration, Data-Ingestion). If you let developers free-type tags, your data model will collapse under the weight of 40 different spellings of 'miscellaneous'.
• Area Path: The domain or team owning the core deliverable.
• Original Estimate: The team’s initial "inside view" guess.
• Remaining Work: The burn-down trajectory.
• Completed Work: The total, completed time against the ticket, as per the developer's best guess.
• State & State Changed Log: Essential. We need timestamp math to calculate dwell time versus active working time.
• Total Open Time: The chronological time between 'Started Work' and 'Completed Work'.

Notice what’s missing?

The Assignee.

If you want this to work, you must remove "Assignee" from your forecasting model. If you tie an estimate to an individual, you aren’t forecasting the work; you are profiling the worker. The moment developers think an ML model is tracking their personal speed, Goodhart's Law kicks in. They will pad their estimates, fabricate their hours, and the data is permanently poisoned. We want to know how long a specific class of work takes the practice to deliver, not how fast Dave is on a Tuesday. Dave reserves Tuesday for updating his Friends Reunited* page, he has no interest in velocity.

*Sidebar: if nothing else comes of this article, I want the kudos for the Friends Reunited reference.

Support tasks are the best thing you don't do

Ever had a sprint completely derailed because you spent three days explaining your architecture to another team? Of course you have. It's an absolutely nailed on productivity killer. Always has been. To build an accurate historical backlog, we have to quantify this "invisible work." That's what we use Support Tasks for.

This is a tiny, low-information ticket designed to capture effort against another team's Area Path. No acceptance criteria, no lengthy descriptions. Just Title, Assignee, Area Path, and Completed Work. If I’m a Data Engineer and the Source Solution Architect spends 45 minutes explaining a gnarly XML output to me, they raise a 0.75-hour Support Task against their own team’s Area Path. That makes sure they're credited with their work, clearly dissociates it from planned effort, and gives us a lot of clarity around tasks that have to go through a spaghetti-like web of shit in order to get completed. I once had six distinct support tasks listed against one of my own tasks as part of a migration piece - it's an invisible death for everybody's productivity if it's not measured.

Suddenly, with this done, we can see the friction. By tracking Support Task Count and Support Task Dynamics (ie, how widely spread the help has been), we shift the conversation to points of unique complexity in projects - which is VITAL for programme or project managers estimating similar projects for another client. When stakeholders ask why a feature took two weeks instead of one, we don't give them anecdotal excuses. We show them the network graph proving the team spent 20 hours unblocking dependencies. It weaponises the DevOps board when it comes to those challenging HiPPO (highest paid person's opinion) conversations and shows exactly where difficulties existed.

The process is broken

You can have the best data collection process in the world, but if your engineers don't fill it in, it’s useless. The temptation here is to mandate that everyone logs 40 hours a week to ensure "good data hygiene." Do not do this. Hang on, let's go again in bold. Do not do this. If you demand 40 hours of logged time, you will get 40 hours of logged time, regardless of what reality looks like. It has to be submitted by developers and for developers. It's not a behavioural or performance submission - it's a work forecasting submission.

Instead of targeting timesheet quotas, target the habit.

• The Board MUST BE Reality: During your daily stand-up, share the screen. Do not give updates from memory; walk the board. If a developer spent 4 hours yesterday fighting a fire, but it isn’t on the board, it gets added live during the meeting. If it isn't in DevOps, it didn't happen.
• The Heartbeat Metric: We don't care if you logged 7.5 hours today. We care that every active ticket has a heartbeat. A ticket in 'In Progress' must be touched daily - a tweak to 'Remaining Work', a state change, a new comment. It's absolutely critical that we're brushing across every ticket, every time. Even if the message is "deprioritised today due to the WeWork office padel championship". I'm reliably informed that that's the sort of nonsense that goes on in those hipster offices.
• Best-Effort is Gospel: Explicitly tell your team that time logging is a "best-effort" estimation, not a rigid audit. Paradoxically, allowing for approximate data yields a much cleaner dataset for machine learning than demanding exact, fear-driven precision.

If you implement this schema and enforce this daily heartbeat, within a few months, you won't just have a backlog. You'll have a high-fidelity, dual-lens dataset showing exactly how hard your tasks are (Touch Time) and how much systemic friction is slowing you down (Cycle Time).

And that is exactly what we need to feed the machine.