Security Watch when no alert arrives

AI agent security, part 3 of 4. A critical finding produced no message while the scan completed normally. Five tests show what Security Watch records about routing, model failures and delivery.

We want fewer security alerts because somebody has to read them and decide what to do. But once we start filtering, we also need to explain what was left out. A channel with no messages does not tell us whether nothing applied to our systems, a model dismissed a story, or the monitor could not finish its work.

Security Watch is our open-source bot for following advisories and security news against a declared system inventory. A vulnerable dependency can remain deployed while its advisory gets lost in a channel. The bot keeps findings and evidence locally and can route notifications to Slack or email, with a record of why a finding was raised or a message was held. We can then check what the bot decided and what still needs investigation. Its inventory can include agent libraries, runtimes and tool servers, alongside the other services the team maintains.

For this article, we ran five scripted cases against the published 0.1.0-alpha.0 package. We supplied a fictional advisory and local replacements for the source, model and email services. No real provider requests or messages were sent. We kept the results so readers can check what the bot recorded in each case.

A critical finding with nowhere to go

Our first case had enough information for a supported package match. The inventory listed example-parser version 1.2.0. The fictional advisory marked versions from 1.0.0 up to, but excluding, 1.3.0 as affected and gave the vulnerability critical severity. The inventory was fresh.

Security Watch recorded an affected finding with critical source severity and critical priority. The scan completed without degraded status. It queued zero notifications, because we had selected no channel for library findings.

We need to inspect routing before relying on the bot. Its default setup leaves notifications disabled until an operator selects and configures them. Even an urgent override has nowhere to send a message when the category has no channels. The doctor command reports the active routes, and an explicit test-notification command can check a configured destination. Setup and notification routes

The finding itself holds several different answers. Source severity describes what the advisory reported. Applicability records what the bot could establish about the declared package and version. Contextual priority helps decide how to handle that finding. A critical advisory can concern a package version we have already replaced. A missing version leaves us with an investigation, even when the article sounds very certain.

In part two, on Security Watch and Agent Guard, we tested an old staging inventory that listed a fixed version. Its result remained unknown. We still need somebody to verify what is deployed and whether the vulnerable code is reachable. A recently edited inventory file does not do that work for us.

What the model can change

Structured package evidence can be checked without AI. News is less tidy. One article may discuss a new vulnerability, mention an older CVE and describe several products that share a vendor. Matching a product name gets us a candidate worth reading. It does not establish that our deployment is affected.

The optional model analysis helps classify that news and assess the impact claimed by the article. The bot checks the output structure, whether the cited evidence exists and whether the supporting quotations actually appear in it. A second model can review the assessment, with disagreements kept visible. Those checks give us something to inspect, but a real quotation can still be interpreted incorrectly. News analysis and evidence requirements

We tested the boundary using the critical package finding. Our simulated model returned a structurally valid opinion calling it irrelevant and proposing low priority. The bot accepted the response as complete, while the finding stayed affected and critical. It queued one email notification under the configured library policy.

Then we made the model transport throw an error. The package finding again stayed affected and critical, with an email job pending. This time the model status was unavailable, the scan was degraded, and a separate high-priority monitoring-health finding recorded the problem. We had left health channels empty in the test, so that health finding was stored locally. Neither case sent a message during the scan.

Both model cases left the structured package finding and queued email intact. News filtering has more room for judgment. In this release, evidence-backed general-news classifications and AI-reported exclusions can suppress noncritical news notifications. Source-critical severity and enabled exploitation safeguards still take precedence. A team using AI to reduce its news queue needs to review the conclusions it is relying on, including the ones that produced no alert. Published decision rules

In an earlier release evaluation using real provider requests, 19 news-and-system pairs produced nine completed assessments, seven conflicting classifications and three rejections for unsupported claims. All 36 requests returned HTTP 200. The inventory listed product names without installed versions. Successful requests could not fill in that missing deployment information. This was a small operational sample, separate from the five local cases here. Published release evaluation

Article text and model output are untrusted inputs. The model has no shell, browser or notification tools. It cannot change recipients or authoritative version ranges. This limits what a hostile article can cause through the bot, although it can still try to mislead the analysis. Calling the model’s output evidence does not make it authority to change a system. Agent Guard covers a separate part of that workflow, checking whether a task may perform an operation. Security Watch trust boundaries · Agent Guard article

A queued alert still has to arrive

Security Watch commits finding changes and their notification jobs together in SQLite. Delivery then has its own state. An eligible finding and a pending email job tell us that the bot has work to send. They say nothing yet about what the destination accepted.

For the delivery control, our local email adapter reported that every intended recipient had been accepted. The job became accepted after one call. That status describes destination acceptance. A person reading or acting on the message is a further step.

In the last case, the adapter threw after being invoked, leaving no usable receipt. The job became unknown. We closed and reopened the same database, ran the recovery step and tried processing pending deliveries again. The job remained unknown, and the adapter had still been called only once. The bot had kept the uncertainty instead of automatically repeating the send. Five runnable cases · Recorded results

A real timeout can happen after a destination accepts a message. Before retrying an unknown job, the operator needs to check receipt. Known failures before sending and explicit transient rejections have different retry handling. Our test exercised the uncertain path and a graceful database reopen. It did not test a real SMTP server or a process crash. Delivery and recovery rules

There are also ordinary reasons to hold a notification. Lower-priority uncertainty can wait for a review digest. An operator can acknowledge a finding revision or suppress it with a reason and expiry. Material changes create a new revision, while cosmetic article or model wording does not create a fresh alert. We can reduce repeated messages while retaining the history needed to explain the current decision. Notification policy

Where we would use it

Teams using Dependabot or Dependency-Track already have useful ways to follow vulnerable components. Dependabot works from the repository dependency graph and supports triage and remediation workflows. Dependency-Track monitors an application portfolio using SBOMs, with policies and notifications of its own. Those are established capabilities we should account for when deciding whether another bot earns its place. Dependabot documentation · Dependency-Track documentation

Security Watch keeps the declared deployment context, source evidence and handling history together. An operator can inspect why a finding is still uncertain, why a notification was held, or why delivery needs review. The bot can run without a model, and its open-source implementation lets us reproduce those decisions with local evidence. That is the workflow we would add where the team needs to follow both advisories and security news against its own inventory.

A vulnerable dependency can stay unnoticed after deployment, and a product advisory can get buried in the news a team already follows. Repeated messages add another problem when people start ignoring the channel. The bot gives us a way to collect and route that information, and to see when its own collection or analysis is incomplete. Remediation still requires a verified finding and a decision from the people responsible for the system.

What we still have to verify

The published alpha accepts library inventories across twelve ecosystems, but that does not mean equal version-analysis support. Only npm has supported range evaluation and exclusions. Other ecosystems require positive exact OSV results or explicitly listed affected versions. Unknown versions and unsupported ranges stay uncertain. Generic products and SaaS services get candidate matching and optional news analysis, without a general adapter for vendor backports or configuration-dependent exposure. Coverage limits

The bot follows configured sources and their available history. It cannot establish that a system has no vulnerabilities from an empty search. This release has npm lockfile import, but no general SBOM ingestion, automatic patching or distributed workers. It is an Apache-2.0 experimental alpha intended for small installations, without an independent security audit. Published release

Running it also means looking after a monitor. Source failures need review. The health command reports whether monitoring is working, and a healthy monitor can still be tracking critical vulnerabilities. An optional external heartbeat can stop sending success pings when monitoring becomes unhealthy. The receiving service has to notice the missing pings, because a failed host cannot report its own absence through that same host. Monitoring health and heartbeats

Enabled advisory queries can disclose package names and versions to their services. A configured model provider receives selected inventory context and evidence. The local database keeps findings, source material and analysis, and is protected by filesystem permissions rather than application encryption. Configuration, database access and backups need to be treated as operational security data. Data handling

To reproduce the five cases, save the linked script in an empty directory and run it with Node.js 24 or newer. We used Node.js 26.12.0.

npm init -y
npm install --ignore-scripts --save-exact @shieldaai/security-watch@0.1.0-alpha.0
node watch-silence.mjs

The successful-looking statuses in these cases answered different questions. The scan was not degraded even though a critical finding had no route. The model response was complete even though it proposed an irrelevant, low-priority classification for that finding. An accepted delivery recorded the destination’s response, with no evidence that a person had reviewed it.

If another agent picks up security work from the alert channel, it needs those distinctions before deciding that work is finished. It needs to know what happened to the finding and whether someone reviewed it or made the required change. A completed scan, a valid model response and an accepted message cannot supply that on their own.

Continue to part four: Agent Guard and the permission to try again.