AIOps Ingest

Catalogue

What this deployment can ingest, and what the pluggable tiers cost you.

Processor profiles— these are not equivalent (ADR-0002)

ProfileEnriches CICarries topologyOperated byNote
data-prepperdeferred to engineyesexternal Data Prepper deploymentCI enrichment is deferred to ARGUS-engine, so correlated insight lands later and raw search results are not CI-tagged.
otel-onlydeferred to enginenoexternal OTel CollectorSmallest footprint. Cannot carry topology signals, and CI enrichment is deferred to ARGUS-engine.
python-normalizerinlineyesthis platformDefault. Vendor-neutral and full-featured; CI resolution happens inline.

Sources

SourceSignalCollectionPhaseStateDescription
cmdb.servicenowtopologypull1builtCI inventory and dependency graph via the ServiceNow Table API (cmdb_ci, cmdb_rel_ci). Authoritative CI source per ADR-0001.
cmdb.postgrestopologypull1builtCI inventory from the ARGUS aiops.cis / ci_identities / application_cis tables. Development and demo provider.
generic.unclassifiedlogpush2builtMessages the syslog classifier could not fingerprint. A real source with a real topic, not a bin: unclassified traffic is stored and counted so misconfigured senders are visible rather than silently filed under the wrong source.
host.linuxlogpush2builtsyslog/journald from Linux hosts via agent or syslog gateway.
host.windowslogpush2builtWindows Event Log channels via agent or Windows Event Forwarding.
host.aixlogpush2builtAIX and other Unix syslog. Common in PH bank core estates.
net.sysloglogpush2builtRouter/switch/firewall syslog. All vendors share this source; the vendor is an attribute (observer.vendor) resolved by the classifier, not a separate source.
net.snmp_trapeventpush2plannedSNMP v2c/v3 traps from network and facility equipment.
k8s.eventseventpush2builtKubernetes event stream.
db.oraclelogpush2built
db.mssqllogpush2built
db.postgreslogpush2built
mw.webspherelogpush2built
mw.tomcatlogpush2built
generic.httpeventpull2builtConfigurable HTTP poller for tool APIs that expose a JSON list endpoint. Covers the common shape while vendor-specific connectors are built.
mon.solarwindseventpull2plannedAlerts, config baselines and drift from SolarWinds. Pull connector — same shape as aisoc's API collectors.
mon.dnacentereventpull2plannedNetwork state, topology and compliance from Catalyst Center.
mon.thousandeyeseventpull2planned
itsm.servicenoweventpull2plannedIncident, change and problem records.
vuln.qualyseventpull2planned
backup.veeameventpull2planned
host.metricsmetricscrape3builtCPU/memory/disk/network from node exporters or OTel hostmetrics, via Prometheus remote-write.
k8s.metricsmetricscrape3built
net.netflowmetricpush3planned
apm.metricsmetricscrape3builtRED metrics: scraped, or derived in-pipeline from logs and events now and from spans once Phase 4 lands.
apm.otlptracepush4builtOpenTelemetry spans over OTLP, both protobuf and JSON. Consolidates the existing ARGUS demo trace path into the shared pipeline.
k8s.logslogpush4builtPod/container logs from an in-cluster OpenTelemetry Collector's filelog receiver, exported over OTLP/HTTP (protobuf or JSON) to the same receiver that serves apm.otlp. Correlates to a span via trace.id/span.id when the workload is instrumented.
aws.cloudwatch_logslogpush7plannedLog groups via subscription filter -> Firehose -> HTTP ingest. Subscription is the scalable path; polling GetLogEvents does not keep up at estate scale.
aws.cloudwatch_metricsmetricpull7plannedGetMetricData polling, or a CloudWatch metric stream via Firehose — cheaper and lower-latency at volume.
aws.ekseventpush7plannedControl-plane logs via CloudWatch. Cluster events and pod metrics come from an in-cluster Collector, not the AWS API.
aws.ecseventpush7plannedTask state changes via EventBridge; Container Insights for metrics.
aws.rdslogpull7plannedSlow-query and error logs via DownloadDBLogFilePortion; Enhanced Monitoring for metrics. Fits the connector framework.
aws.elblogpull7plannedAccess logs land in S3, so this needs an S3-notification collector — a different collection shape to everything built so far.
aws.healtheventpull7plannedAWS Health events. Small volume, high value for correlation.
azure.monitor_logslogpush7plannedDiagnostic settings -> Event Hub -> ingest. Event Hub is Kafka-wire-compatible, so this may be a Kafka consumer needing no new transport and reusing the processor tier unchanged.
azure.monitor_metricsmetricpull7plannedAzure Monitor metrics API, or diagnostic settings to Event Hub.
azure.akseventpush7plannedControl-plane diagnostics; cluster events via in-cluster Collector.
azure.acalogpush7plannedContainer Apps logs. aisoc already deploys to ACA, so the log shape is known.
azure.vmlogpush7plannedAzure Monitor Agent to Log Analytics, or direct to Event Hub.
azure.sqllogpull7plannedSQL Database diagnostics.
azure.activityeventpush7plannedSubscription activity log — the Azure analogue of CloudTrail.

Destinations

Sinks
  • elasticbuilt
  • memorybuilt
  • opensearchbuilt
  • sentinelplanned
  • splunkplanned
  • stdoutbuilt
Metric stores
  • memorybuilt
  • remote_writebuilt
  • stdoutbuilt
CMDB providers
  • postgresbuilt
  • servicenowbuilt
  • staticbuilt