What an alert actually costs you before anyone looks at it

Short read

This post covers the hidden cost of an alert: the time it waits in a queue before anyone looks at it, which is usually far longer than the investigation itself. It shows you how to measure time to first look in your own team, and why tuning, hiring and better prioritisation don't remove the wait.

We also explain how investigating every alert the moment it arrives, as urgentic does inside Microsoft Sentinel, removes the queue altogether.

Most teams measure how long an investigation takes. Almost nobody measures how long an alert sits there first.

That gap is where most of the cost of an alert hides. It doesn't show up in any report, because nothing is happening during it. The alert has fired, it's in the queue, and it's waiting for someone to be free.

The queue is the cost

Take an alert that needs four minutes of real investigation. If it waits forty minutes to reach someone, it's a forty-four-minute alert. The four minutes was never the problem.

This is an illustration, not a benchmark. The point holds whatever your own numbers are: the time an alert spends waiting is usually far longer than the time spent working on it. Yet the waiting is the part most teams don't track.

It happens for ordinary reasons, not because anyone is careless:

• Triage starts when a person is free, not when the alert arrives. A queue fills at the speed of your environment and empties at the speed of your team.
• Overnight alerts wait for the morning. Unless someone is on shift, whatever fired at 3am is first looked at over coffee.
• A busy queue pushes the quiet-looking alerts down. Analysts sensibly start with whatever looks most serious. Everything else waits longer.

The alerts that get pushed down are the ones nobody has looked at yet. That's the same set as the ones that might matter.

Why the waiting matters more than the working

An investigation answers one question: is this real? Until someone asks it, you don't know whether you're looking at a misfiring rule or the first sign of someone inside your network.

While the alert waits, three things are true at once:

1. If it's harmless, nothing is lost, except that it's still taking up a place in the queue ahead of the next one.
2. If it's real, whoever caused it has the same forty minutes you do, and they're using theirs.
3. Either way, you can't tell which, because the only way to know is the investigation that hasn't started yet.

That third point is why "we'll prioritise the important ones" only goes so far. Priority comes from the alert's label: its severity, its rule, its source. Importance comes from what's actually happening, and you only find that out by looking. A low-severity alert can be the start of something serious, and a high-severity one can be noise.

How to measure it in your own team

You don't need new tools to see this. For a week, record two times for every alert:

• Time to first look: from the moment the alert fired to the moment a person opened it.
• Time to decision: from first look to "this is nothing" or "this needs action".

Most teams that do this find the first number is the bigger one, often by a long way, and that it swings with the time of day, the day of the week, and who's on leave. The second number is fairly steady. That tells you where the cost really is.

It's also worth noting how many alerts were never opened at all before being closed in bulk, or simply aged out. Those weren't investigated quickly; they weren't investigated.

What doesn't fix it

The usual responses each help a little, but none of them removes the wait:

• Tuning rules to fire less reduces the volume, but the alerts that remain still queue, and every rule you silence is something you've decided not to see.
• Hiring more analysts shortens the queue during working hours. But a person can only look at one alert at a time, and round-the-clock cover is a large, ongoing cost.
• Better prioritisation decides which alerts wait, not whether they wait.

All three try to make the queue more manageable. The queue itself stays.

What changes when investigations run on their own

The alternative is to stop treating investigation as something that waits for a person.

When every alert is investigated the moment it arrives, the time to first look drops to almost nothing, for every alert, at any hour, however busy the queue is. Each one has been looked at, its evidence gathered, and a finding written up before a person needs to be involved. Your team then starts from an answer rather than a question, and spends its time on the alerts that turned out to need judgement.

That is what urgentic does. It works inside Microsoft Sentinel, investigating incidents as they come in, rather than adding another dashboard to watch.

It also changes the economics of security operations. A human team costs much the same whether it's watching a small estate or a large one. That's what puts full security operations out of reach for many smaller businesses, and it's why larger teams still carry a backlog. Investigations that run on their own have neither that cost floor nor that volume ceiling. So the same thing that makes round-the-clock investigation realistic for a smaller business is what lets a large security team, or a managed security provider, keep pace with everything that fires. It's better value for money not because it does less, but because nothing waits.

The short version

• The expensive part of an alert usually isn't the investigation. It's the time before the investigation starts.
• That time is invisible unless you measure it, so measure it: time to first look, for every alert, for a week.
• Tuning, hiring and prioritising make the queue easier to live with. Investigating every alert as it arrives removes the queue.

It's not the four minutes that change. It's the forty.