Provenance-based Graph Neural Networks using Heterogeneous Graph Transformers - Part 2

In this post we directly continue from Part 1 (https://blog.pr0kythera.com/posts/new-post-here/), we ended up with: a single minimal test HGT (hetrogeneous graph transformer) which worked but could tell us nothing besides the pipeline was a success, and three hopes and dreams: more scenarios, a temporal feature, and an A/B test of the three-scope idea against a single GNN (the control) to determine if my hypothesis that 3 differently scoped approaches out performs a single GNN that looks over the entire graph.

I somewhat apologise that the following is very long. I’ve also decided to commit to hopefully a shorter part 3 with the aim to make the project a bit more in the realm of real world application rather than solely one focused on research. Once I have something a bit more generalisable i’m really keen to release the whole project. But currently everything is built against Evidence Forge output and also only the specific scenarios at that.

For a tldr headline. We successfully proved hypothesis! what I am calling “PARALLAX” a multi-scope Heterogeneous Graph Transformer ensemble out performed a control HGTConv encoder trained over the whole graph. What follows is how we got to that point, the problems encountered along the way and the iterative improvements that drastically improved out output over 2 rounds of modifications.

To be complete transparent Claude Opus 5.8 and Sonnet 5.0 were heavily utilised in this project. The design, direction and this writeup is all my own. But as you can imagine authoring everything by hand would be months or years worth of work for a team. A GNN detection project like this has been rattling around in my head for a long time but the advent of frontier models over the past 12 months make something with this scope actually achievable in my afternoons and weekends. With that in mind i’m trying to be very cogent not to be just producing slop. My personal take is as long as there is heavy human involvement in the input, process and output LLM’s are a fantastic enabler. The problems lie with overreliance and a lack of understanding and engagement Rubbish in Rubbish out isn’t by any means a new concept and it’s being amplified by LLM’s.

A recap (for readers who somehow ended up here without bothering to read Part 1)

My fundamental thesis is as follows: attack signals in enterprise telemetry lives in causal CHAINS across log sources, for example: a login that created a new process, that process reads LSASS, and then wrote the output to a file, the process went on and crossed a network boundary, on the new host it then installed a service.

Part 1 framed this as a provenance-graph learning problem, we chose a Heterogeneous Graph Transformer (HGT) as the model family because it keeps per-node-type (a service is treated differently to a process) and per-edge-type semantics distinct instead of flattening everything into one generic type. Most importantly I proposed that three GNNs looking at the same graph through three different windows (a process-subtree scope, a cross-domain-boundary scope, a whole-graph scope), fused by a fourth model. It also introduced EvidenceForge (Cisco Talos), the synthetic multi-source log generator this entire project depends on.

Data pipeline, scenario generation, and validation

From one scenario to twenty

Quickly to recap the data pipeline this image does a good job of desrcibing what the process:

(The data pipeline)

Part 1 ended on a single scenario. Building a study meant going from one to twenty: thirteen attack scenarios built in EvidenceForce (A1-A13), two “failed/contained” scenarios (F1, a firewall-blocked lateral move; F2, a “contained intrusion”), one dual-independent-attacker scenario (D1), and four benign scenarios (B1-B4) 20 total, all built through a repeatable procedure so the process was identical for each scenario. The reason I wanted to include additional scenarios where a SOC identifies and contains a malicious attack chain part way though as well as a scenario featuring dual attackers is so the model does not only get clean training data to learn from. Similarly the benign examples are important so we don’t end up with a model that labels everything as malicious because everything it trained on was malicious.

The procedure was as follows:

1. Author the YAML from a scenario spec described above, using only documented storyline event types (service_installed,scheduled_task_created), because EvidenceForge has no psexec or schtasks event type.
2. eforge validate <scenario>.yaml - validate and fix issues as soon as they rear their ugly heads.
3. eforge generate <scenario>.yaml -o <out_dir> -target default Here we produces the deterministic log set. Also run eforge eval against four hard quality gates (spec conformance, value plausibility, causal ordering, event presence) any issues and it gets sent back to step 1 to iterate.
4. Inspect the RAW output against the spec, before any pipeline code touches it. Read GROUND_TRUTH.json as the oracle, then confirm each storyline step actually produced the expected records in the expected shape. This is where divergences between “what the spec asked for” and “what the engine actually emitted” get caught and analysed and potentially re-authored.
5. Run and adapt the pipeline (parse -> correlate -> label) each step validated against the scenario output.
6. Verify the built graph against ground truth entity-by-entity ensure timestamps are present and ordered.
7. Record and close the loop Document scenario created successfully with any required changes.

(Scenario Build Loop)

EvidenceForge’s scenario-authoring layer is LLM-assisted and therefore non-deterministic, but generation from a finished, committed YAML is fully deterministic and seeded from scenario.name.

The matched-pair method

Most of the 20 scenarios aren’t just independent data points. They’re actually six matched pairs, each isolating one axis of variation while holding everything else constant.

Why? With only 20 scenarios total, an unpaired comparison can’t tell you whether a result reflects the thing you actually changed (transport, dwell time, persistence mechanism, OS) or just incidental differences between two unrelated scenarios. By sharing EvidenceForge’s RNG seed and changing exactly one axis, a matched pair isolates that one variable against an identical background. So when scope 2 catches both A1 (SMB) and A2 (WMI) equally, there we have the evidence that the model generalizes to “crossed a trust boundary”! Rather than just overfitting to one lateral-movement technique’s specific fingerprint.

• A1/A2 (lateral movement): SMB vs WMI transport
• A3/A4 (credential access): LSASS dump vs DCSync
• A5/A6 (persistence): service creation vs registry Run-key
• A7/A8 (dwell time): ~12 minutes vs ~7 hours
• A9/A10 (background noise): low-noise vs ~12x background volume
• A11/A12 (operating system): Windows lateral vs Linux pivot

The trick that makes this work is EvidenceForge seeding its run RNG from the scenario’s own name: field. Two YAML files that share the same name and differ in exactly one field will generate identical background activity and identical malicious object identities, diverging only where the deliberately changed field takes effect. A2’s YAML is byte-for-byte identical to A1’s except for one line:

# scenario_a1.yaml
name: a1-smb-lateral
...
storyline:
  - step: remote_logon
    event: logon
    # network_transport defaults to "auto" == stock SMB behaviour

# scenario_a2.yaml - SAME name, ONE field different
name: a1-smb-lateral   # <- shared seed, not "a2-..."
...
storyline:
  - step: remote_logon
    event: logon
    network_transport: wmi   # the only diff: DCOM/RPC 135 instead of SMB 445

Due to some issues which I won’t get into not every pair is this closely aligned. But they’re close enough to meet our goal to be able to compare them.

Where EvidenceForge itself had to be extended

Two of the six pairs needed a capability stock EvidenceForge 1.9.0 didn’t have. Rather than give up on them we created an additive local fork patch.

WMI transport (A1/A2). Stock EvidenceForge’s remote admin storyline bundle is SMB-only. There’s no way to author a WMI/DCOM lateral move at all. The patch adds one field, network_transport: "auto"|"smb"|"wmi", to the storyline logon event; wmi routes a type-3 logon’s establishing cross host connection to DCOM/RPC (port 135, dce_rpc) instead of SMB (445). The edits spanned the scenario model, the auth-session action, the activity generator, and the storyline engine and auto (the default) reproduces stock behavior exactly.

Registry persistence (A5/A6). EvidenceForge has no Reg run key persistence event the stock path is unfortunately a 50% probability, EDR-pool-random registry side effect, which means an authored Run key persistence step was never reliably emitted. The patch hooks the process generator: when a storyline reg.exe process runs reg add <Run-key> [/v <value>] [/d <data>], it now deterministically emits one registry_modify event to that exact key, stamped with the process’s own actor ID - and skips the stock random side effect for that process only. Our existing correlator picks this up through its existing generic actor-to-object path; no correlator change was needed! Only the labeler needed a new rule (rule 3b, scoped specifically to CurrentVersion\Run targets, so an incidental malicious registry write event elsewhere such as A3’s Office key touch correctly stays unlabeled).

F1: when the correlator has to synthesize a node from nothing

Scenario F1 (firewall-blocked lateral movement) forced a different kind of extension. A connection blocked by a firewall produces no eCAR FLOW record at all on the source host. Only the Zeek sensor (a REJ connection state, on the source segment) and a new ASA syslog parser (ported from EvidenceForge’s own MIT-licensed reference implementation) see it at all. That meant the correlator had to learn to synthesize a socket node when the graph would otherwise have no node to hang a label on. Normally we’d expect to see something in a table like Microsoft’s DeviceNetworkEvents:

def build_asa_deny_nodes(self, zeek_events) -> None:
    """For each retained ASA 106023 deny, ensure a socket node exists
    at its 5-tuple identity - even when eCAR is silent. Resolution
    order: (1) an eCAR-built socket at this 5-tuple, if one exists;
    (2) else a Zeek conn record at the same 5-tuple, synthesize from
    Zeek fields; (3) else synthesize a BARE socket from the ASA
    record's own 5-tuple alone."""

That node deliberately carries no actor edge. Nothing in the graph links any process or session to it, because the only candidate (a Windows 5156 filtering-platform event) is attributed to the generic “System” PID 4 shared by all network activity on the host, a useless anchor. Building machinery to wire that up was considered and explicitly declined as disproportionate in this project.

Validation

Every one of the 20 scenarios has its own validator script that re-reads the assembled feature artifact from disk and checks for the following:

• Malicious node/edge counts match ground truth exactly, entity-by-entity
• Every malicious node has at least one edge or is a recorded exception
• Timestamps are present and correctly ordered
• Both baseline reconciliation flags in the manifest are True

Feature engineering

Why this is harder than it sounds

A graph node like “the process pd.exe opened LSASS at 14:32:07” means nothing to a neural network until it becomes a vector of numbers.Turning “what happened on a compromised machine” into numbers a model can compare across twenty different attacks, on different hosts, with different personas and different background noise is the actual work of this stage. It’s also a bit more complicated than simply turning a scenario’s own logs into numbers.

(Feature Engineering)

The script feature_config.py created one fixed global schema for all 9 node types (process, socket, session, file, service, registry, module, account, scheduled_task) and all 16 edge/relation types, shared by every one of the 20 scenarios. Each node type gets a fixed-width feature block: one-hot encodings over static, hand crafted vocabularies (hostnames, origins, Zeek service names), multi-hot encodings for things like “which merge stages enriched this node,” and scalar structural features (in-degree, out-degree, total degree, and per-edge-type degree breakdowns). Every node and every edge in every scenario’s graph gets a stable index which is globally shared, This allows the three differently scoped GNNs agree on which embedding belongs to which entity.

One schema decision worth calling out for the non-ML reader: usernames and persona names are deliberately NOT features. Only structural and derived values go into the tensors. This matters because personas rotate across scenarios specifically so that no single name is consistently “the attacker”. If a name leaked into the feature vector, the model could trivially learn:

“this specific username = malicious,” which would be a shortcut that says nothing about the general provenance-graph thesis.

The “abused node” concept (9C)

One label-design decision to call out is. A node can be benign (node label 0) while still being the target of a malicious edge. A3’s LSASS process is a real, legitimate Windows process. It did nothing wrong by existing, but the edge representing an attacker’s process opening it with 0x1FFFFF access is malicious. The project tracks this explicitly as an abused metadata flag: a benign node with at least one malicious incident edge. It’s the concept that makes A3’s LSASS and A4’s krbtgt account symmetric rather than looking like two unrelated one off decisions.

Round 3’s feature-layer additions: os and log1p

Two later, smaller feature-layer changes are worth mentioning here:

An os=windows|linux|unknown node feature, added per-hostname by checking which sensor artifact files actually exist for that host.

row.update(_onehot(hostname_os.get(node.hostname), fc.OS_VOCAB, "os_",
                    other_slot="unknown"))

This mattered because A12 (the one Linux pivot scenario) is otherwise indistinguishable from a Windows host at the structural level except through is_cross_host and the flow bridge an explicit OS signal was found to be a gap.

The three-scope architecture

Part 1’s thesis was that attack signal lives in causal chains spanning log sources rather than in any single event taken in isolation. A single whole-graph GNN sees that whole chain in one pass, which sounds like an advantage, but it also means every node’s representation gets diluted by attention over the entire graph, including tens of thousands of irrelevant benign nodes on a large scenario like A10 or D1. The bet: giving the model three deliberately different fields of view over the same graph one tight and local, one narrow and targeted, one as wide as M1’s own control. Then finally combining their verdicts, recovers signal that gets averaged away under a single unrestricted view.

Scope 1 - process-subtree

Scope one is seeded on every process node, restricted to process lineage relation types only (parent/child creation, and the file/registry/socket edges hanging directly off that process’s own subtree), with a shallow hop budget. Structurally, this is the same as a SOC analyst pulling up one process’s ancestry tree in an EDR console.

This is also where R-CAID’s framing is directly relevant (root-cause localization within a provenance graph, rather than a single whole-graph verdict). Scope 1 is this project’s own analogue of that same idea, built as one input among three rather than the whole system.

Scope 2 - cross-domain-boundary

Scope two is seeded specifically on entities and edges where is_cross_host=1. A structural feature already in the schema, not anything scenario-specific. Lateral movement lights this feature up regardless of which technique caused it (SMB, WMI, RDP all set the same flag), which is exactly why the PAIR-LATERAL matched pair (A1/SMB vs. A2/WMI) exists: if scope 2 only worked for one transport, it would be overfit to a technique rather than to the boundary-crossing structure itself. This would be analogous to the SOC checking for connections from a compromise host during an investigation.

Scope 3 - maximal-context

Scope three has the widest supported “fanout”, no relation-type restriction structurally the same encoder family as M1’s control, but now one of three inputs rather than the only one. The is the hardest aspect of manual investigations casting a wide net on activity can easily lead to rabbit holes and in some cases threat actors generating events purposely to distract analysts.

(Three Scoped GNNs)

Why fuse three verdicts?

Different scopes are expected to win on different scenario shapes. A fixed choice of scope would mean picking a single trade off point and living with it across every scenario shape in the study, from a single host LSASS dump to a 70,000-node noisy background to a two-attacker D1. A per instance combination of scopes can in principle do better than any one fixed view.

Fusion is a small stacker (logistic regression, with an MLP tried as an ablation arm) over {scope1_logit, scope2_logit, scope3_logit, regime features}, fit with explicit anti-leakage discipline: an inner held out subset of each fold’s training scenarios generates the out of sample logits the stacker is actually fit on, so it can never simply learn to trust whichever scope happened to overfit hardest on data it had already seen.

One finding worth flagging:

A directly comparable paper in this same problem space, PROVFUSION (arXiv:2604.14685), reports that simple voting-based fusion of multiple provenance-graph views beats a learned stacker across nine benchmark datasets. A result this project’s own Round 2 work tested against its own data, and which came back with rather mixed results.. More on that later!

The model doing the seeing: HGTConv

A Heterogeneous Graph Transformer (HGTConv in PyTorch Geometric) is a transformer-style attention mechanism that is “type-aware” - a login node attends differently to a neighboring process node than to a neighboring socket node, using separate learned weights for each (source-type, relation-type, destination-type) combination, instead of just treating every edge in the graph as interchangeable the way a plain GNN would. It was chosen specifically because it preserves per-node-type and per-edge-type semantics through attention rather than collapsing everything to one generic type, which matters here because the schema’s 9 node types and 16 edge relations (dc_replication, scheduled_task_create, session_installs_service, and so on).

Model build, test, and validation

The 4-fold split

A 4-fold scenario-level rotation was chosen, one benign scenario anchoring each fold:

Fold 1 test: B1, A10, A2, A4, A6
Fold 2 test: B2, A1, A3, A5, A9
Fold 3 test: B3, D1, A8, A12, F2
Fold 4 test: B4, A7, A11, F1, A13

We Verified all 20 scenarios appear in exactly one fold’s test set; all six matched pairs land in different folds so whenever either arm is held out, its partner is in that fold’s training set. Meaning that the axis contrast itself is always evaluated out-of-sample. Every fold’s test set contains exactly one benign anchor and at least one attack scenario; the three largest graphs (A10, D1, B2) land in three different folds rather than stacking them together.

(Matched Pair Scenarios)

The M0-M4 build sequence

• M0 - sampling-equivalence gate. Before trusting any sampled training run, prove that sampled and full batch forward passes on a frozen model produce matching logits. Run first on F2 (smallest scenario with real malicious labels. The smaller B1 has zero positives, making its AUPRC degenerate and useless for this check).
• M1 - single-GNN control. One HGTConv encoder over the whole graph, shared weights across all 20 scenarios, trained/evaluated on the 4-fold rotation. M1 acts as baseline the whole three scope thesis has to beat.
• M2 - three-scope treatment + fusion stacker. The architecture Part 1 proposed, now fully specified: scope 1 (process-subtree, shallow fanout, process-lineage relations only), scope 2 (cross-domain-boundary, seeded on boundary-spanning entities where is_cross_host=1), scope 3 (maximal-context, widest supported fanout). All three loaders run against the full node/edge set, so every scope produces a prediction for every node and edge, aligned via the same global index. Fusion: a small stacker (logistic regression, upgraded to an MLP as an ablation arm) over {scope1_logit, scope2_logit, scope3_logit, regime features}, fit with anti-leakage discipline - an inner held-out subset of each fold’s training scenarios generates the out-of-sample logits the stacker is actually fit on, so it can’t just learn to trust whichever scope overfit hardest.
• M2.5 - a UNICORN-style whole-graph A/B. A lightweight Weisfeiler-Lehman structural sketch (not a full re-implementation of the NDSS 2020 paper just an appropriate analogue) plus a supervised classifier, reduced to the same graph-level grain as M1/M2 via top-k-mean of edge logits. M2.5 was built to answer a separate question: does node/edge-level localization actually buy anything over a whole-graph verdict?
• M3 - temporal message passing. A TGN-style persistent memory scope, fused as a fourth input to the same stacker, gated specifically on the scenarios most likely to need real time-ordering: the dwell pair (A7/A8) and D1 (two independent attackers interleaved in time on one shared identity).

Round 1 results

What AUPRC actually measures?

AUPRC or Area Under the Precision-Recall Curve to it’s friends summarizes, across every possible alert threshold at once, the tradeoff between precision (of the things you flagged, what fraction were actually malicious) and recall (of the things that were actually malicious, what fraction did you flag). It’s the best metric here specifically because of the class imbalance this project keeps running into: in these graphs, malicious nodes and edges are routinely under 0.3% of the total, so a metric like plain accuracy would report 99.7% by predicting “benign” for everything and be worthless.

The results

Comparing M1 (single whole-graph GNN) against M2 (three-scope treatment + fusion stacker):

To suprise no one as I spoiled it at the start The central thesis held, cleanly, in every fold with roughly a 5.7x margin on edge level AUPRC. This ended up being the single strongest, result in the whole study to date. In plain terms: given the same graph, the same features, and the same training budget, three GNNs looking at the same data through different windows and combined by a simple average substantially outperformed one GNN looking at everything everywhere all at once.

One potential suprise however was that simple Averaging beat learned fusion, for edges. The ablation table (control vs. naive average-of-3 vs. logistic stacker vs. MLP stacker) found that for edge level prediction, plain averaging (0.5755) beat both the logistic stacker (0.5487) and the MLP stacker (0.4791).

This flew in my naive assumption that a learned combination should always beat a fixed one, reported as such rather than routed around. (For node-level prediction the opposite held: the MLP stacker was the best arm. The two tasks disagreed on which fusion strategy won, which turned out to be an early sign of the per-fold volatility which the later Round 2 changes attempted to reduce.)

The three-scope approach fixes structural complexity. A per scenario breakdown answered the question from M1" “why are D1 and A10 the weakest scenarios” with a sharper distinction than “M2 helps on hard scenarios.” D1 improved dramatically under M2 edge_auprc 0.020 (M1) to 0.455 (average-of-3), ensemble synergy that beat every individual scope. A10 (same attack as A9, but with ~12x the background noise - a scale problem, rather than a structure problem) did not improve: average of 3 (0.049) was actually worse than M1’s own control (0.088) there, because the maximal context scope specifically drowns in the extra noise volume. The three-scope architecture, in other words, is a fix for a specific kind of difficulty, but by no means can be viewed as a simple unilateral resolution.

Does node edge localization beat a cheap whole-graph detector?

M2.5 compared M1 and M2 (reduced to one graph-level score per scenario) against a cheap Weisfeiler-Lehman structural sketch baseline, on graph-level AUROC. Round 1’s result was mixed/negative: the cheap sketch (AUROC 0.7969) beat both of the project’s own systems (M1: 0.5625, M2: 0.7344). Digging into this found it was almost entirely one scenario’s fault: B2, deliberately constructed “hard negative” (busy, suspicious-looking, but entirely benign activity). Excluding B2 alone, M2’s AUROC jumped to 0.9792. This appears it was likely due to the fragility of a crude top-k-mean graph level reduction against one adversarially hard benign graph.

Round 2 - Further literature reviews

After Round 1 closed, a second round of literature reviews checked this project’s own empirically found bugs and open threads against published work in the same problem domain.

Interestingly something unexpected popped up during this second lit review. Which shook me to my very core.

To back up slightly this project was first conceived around late march early April. Thats when the initial research and discovery phase brought me to the idea of the three stacked models. Mid April I was very fortunate to take a lovely holiday away which meant I made very little progress besides a bit of studying around GNN’s and the art of the possible. During this time a fantastic paper “Beyond Nodes vs. Edges: A Multi-View Fusion Framework for Provenance-Based Intrusion Detection” apparently came to a similar hypothesis:

“PROVFUSION: a multi- view anomaly fusion and voting-based detection frame- work. To avoid conflating signals, we treat attributes and structure as separate views (details in §4.2). Specifically, PROVFUSION characterizes each system entity from three distinct views: (i) an attribute view that captures deviations in an entity’s intrinsic features, (ii) a structural view that identifies abnormalities in the entity’s role and structure patterns, and (iii) a causal view that evaluates the plau- sibility of interactions (i.e., edges) the entity participates in. We quantify each view independently, fuse them across seven anomaly dimensions, and finalize decisions via a voting-based mechanism, yielding stable performance across heterogeneous score scales.”

The core idea follows the same principal. A single monolithic view of a provenance graph misses signal that a combination of narrower, complementary views recovers. PROVFUSION frames this as “beyond nodes vs. edges” where as this project frames it as “three windows beat one.”

To be clear however, they are quite different in a few ways. This project’s three scopes are the same model looking at three different amounts of the graph. PROVFUSION’s three views are three genuinely different models, each looking at (roughly) the same neighborhood but scoring a different property of it. Whilst we’re both suggesting that looking at three aspects of the graph is better than just a single view, the method is different. I am however very glad of diving back into literature. We’re in such a fast paced world with publications these days that something incredibly relevant can be reviewed and cited within a projects lifecycle.

As an aside as this isn’t an academic research project just a curiosity learning project I do feel very validated that the idea is both novel but is under active research by academic institutions.

Reflecting on Round 1 results and from our exciting finding of the PROVFUSION paper I prioritized six-item list for Round 2 updates.

• Dropout + per-layer residual connections and degree-aware re-injection (both models)
• Pre-norm placement (M2 only, since M1 had no diagnosed logit-magnitude gap to target)
• A voting/rank fusion arm (testing PROVFUSION’s finding directly)
• An M2.5 re-run plus a more robust median-of-top-k graph-level reduction;
• A larger stacker-fit subset
• A cheap, targeted M3 probe varying the temporal batch size specifically on D1, to test the batch-staleness hypothesis without paying for a full retrain.

Round 3 - pipeline-level changes

Round 3 moved one tier up in cost: changes that require regenerating the stored feature artifacts for all 20 scenarios. Four items: an os=linux|windows node feature; log1p-scaling the stored degree columns (specifically aimed at closing M2’s still-unresolved logit-magnitude gap); a graph-relative normalized timestamp added at the data-loader level (abs_ts_norm, alongside the existing raw timestamp, non-destructively); and replacing the old fixed, fold-wide class-imbalance correction (a pos_weight scalar in the 878x-3119x range) with a much gentler, per-scenario, per-epoch 20:1 negative sub-sampling scheme.

Round 2 + Round 3 results: compared to Round 1

Round 2: a clean architectural win.

Good news everyone? The results show every fold improved on both metrics for M1, no exceptions. For M2, every fusion arm improved on both axes except logistic-stacker edge AUPRC, which was flat. The logit-saturation guard (max edge logit under a ~36.0 float64 sigmoid-saturation threshold) improved substantially for M2 - from failing on all 4 folds under Round 1 to failing on 2 of 4 - though it did not fully close.

(Round based Results)

The central thesis held cleanly, edge-level, on all 4 folds under Round 2’s shared architecture generation. But the margin did narrowed substantially from ~5.7x under Round 1 to ~1.9x under Round 2. This narrowing turned out not to have a simple explanation. M1’s absolute gain (+0.3118 edge_auprc) actually exceeded M2’s (+0.2103), which rules out the convenient story (“M2 just had less headroom left before a ceiling of 1.0” - M2 had 0.42 of headroom remaining. This would be more than enough to match M1’s gain if the same mechanism applied equally to both). Two hypotheses were left open with the available data:

M1, as a single unrestricted whole-graph encoder, may lean more heavily on coarse global degree signal than M2’s narrower, relation-type-restricted scopes do, so the degree-reinjection change was a bigger relative unlock for M1 specifically;

M2 received an extra change (pre-norm placement) that M1 did not, there is the potential that change’s own known stability-vs-peak-performance tradeoff cost M2 some ceiling.

Node-level, the picture had one characterized exception: M2 beat M1 on 3 of 4 folds, with fold 2 (which includes B2, the study’s own deliberately hard negative) going the other way, however this has not been confirmed as the culprit.

A follow-up pass on the same round’s already-saved logits tested two more ideas from the literature review and both came back negative-to-mixed, reported in full rather than smoothed over. Voting/rank fusion (testing PROVFUSION’s finding directly): a naive vote-count arm was the clear worst of five arms tried, because with only 3 scopes it produces just 4 distinct values, which is far too coarse a ranking signal for this domain’s extreme class imbalance; a rank-average arm was respectable but still didn’t beat plain averaging.

A larger stacker-fit subset (8 scenarios instead of 4). It helped the logistic stacker broadly, but hurt the MLP stacker, including one absolutely catastrophic single-fold collapse. Model complexity interacts with fit set size non-uniformly; more data is not a universal fix for stacker instability. Average-of-3 remained the recommended, most stable fusion arm on both axes throughout.

One clean reversal did land: M2.5’s Round 1 finding (“the cheap sketch beats both our systems on graph-level AUROC”) flipped entirely under Round 2. Both M1-reduced and M2-reduced now beat the sketch on both AUPRC and AUROC (M2: AUROC 0.9688 vs. the sketch’s 0.7969).

The deliberate fix built specifically to address the B2-driven fragility behind the Round 1 result (a median-of-top-k reduction, meant to be more robust to one outlier high-confidence false-positive edge) turned out to be unnecessary and was itself rejected the results being uniformly worse than the simple top-k-mean it was meant to replace, most plausibly because Round 2’s other changes had already fixed B2’s specific fragility as a side effect.

The targeted M3 batch-size probe (varying only BATCH_EVENTS, re-training only the temporal scope on D1’s own saved static logits) produced a striking but inconclusive result. At the finest granularity tested (BATCH_EVENTS=20), the temporal scope reversed and actually helped D1 for the first time in the study (delta +0.0404). However the intermediate value (BATCH_EVENTS=50) was the worst of the three values tested (delta -0.0796), which does not fit a simple “smaller batches monotonically reduce staleness” narrative.

Single-seed, single-run at each value. This was found that it cannot separate a real effect from training-run noise, and is flagged as the most promising unexplained thread in the whole probe rather than either adopted or dismissed.

Round 3: M1 improves cleanly; M2’s picture gets more complicated

M1 improved cleanly across the board, every fold, both metrics. The most plausible driver being the new negative-subsampling scheme. A ~20:1 ratio replacing the old 878x-3119x fixed pos_weight, a roughly two-orders-of- magnitude gentler imbalance correction.

M2’s picture is… mixed:

• Edge-level, the central thesis still holds on all 4 folds M2 beats M1 in every fold. But the margin narrowed again, past Round 2’s own narrowing: from ~5.7x (Round 1) to ~1.4x-2.5x (Round 2) to ~1.02x-1.29x (Round 3). Fold 4’s margin, 1.02x, is barely above parity, the smallest edge-level margin recorded anywhere in this study.
• Node-level, the win rate regressed: M2 beat M1 on only 2 of 4 folds (down from Round 2’s 3 of 4). Round 2 had one characterized exception in fold 2. Round 3 adds a second exception (fold 4) with no established cause yet.
• The logit-saturation guard, the exact thing the log1p feature-scaling change was aimed at fixing. This turns out WORSE, not better. Round 2 had 2 of 4 folds exceed the 36.0 threshold; Round 3 has all 4 (36.00-52.41), the worst logit-magnitude result recorded for M2 in the entire study. The negative subsampling change, is the most plausible driver, since a far gentler imbalance correction plausibly lets the optimizer reach sharper, higher-magnitude decision boundaries without the old extreme pos_weight’s implicit damping effect. While there’s no obvious mechanism by which log1p-scaling raw features would increase logit magnitude.

TLDR

The summary for a reader who wants the state of the central thesis in one sentence: three-scope beats single-scope, edge-level, in every fold, across all three rounds so far. That being said the margin has shrunk each round, and the node-level half of the same claim is the weakest and least settled part of the whole study right now.

Translating AUPRC

A single AUPRC number, doesn’t answer the question a SOC analyst actually has: “if I deployed this, what would my alert queue look like?” Precision at fixed recall, computed directly from the real held-out predictions (pooled across all 4 folds - all 20 scenarios, each scored out-of-sample exactly once), answers that more concretely:

Edge-level (241,553 pooled edges, 242 positives, base rate 0.10%):

This is a somewhat counterintuitive finding: M1 has better precision than M2 at 90-95% recall, despite M2’s much higher overall AUPRC. AUPRC integrates over the whole curve and rewards M2’s substantially stronger middle band even though M1 wins the tail. In practical terms: if the deployment goal is “keep the analyst’s review queue clean, catch most things,” M2 is the clear win. If the goal is “catch nearly everything, and accept the noise that comes with it,” M1 is not obviously the worse choice, despite its much lower headline number a nuance the single AUPRC figure hides completely.

A separate, more reassuring check: for every attack scenario, how deep would a SOC reviewing one combined, pooled queue have to go before that incident’s first real alert appears? Both models surface nearly every incident (worst non-outlier case: M1 rank 55, M2 rank 26, out of ~241,000 pooled edges). F2 - the deliberately subtle, contained-before-lateral-movement scenario - is a hard outlier for both models (ranks 740 and 207 respectively), which is arguably the expected, not the alarming, result: F2 was built specifically to be the hardest case in the study. The practical reading: the three-scope treatment’s real value-add isn’t “catches incidents the single-GNN control misses entirely” - both catch nearly everything. The real advantage is the cleaner mid-recall queue shown in the precision table above.

Detect, then reconstruct?

A recurring pattern in published provenance-graph intrusion detection (KAIROS, ORTHRUS, and similar systems) is “detect, then reconstruct”: don’t try to score every node and edge with equal confidence - catch one suspicious anomaly, then use graph structure to walk outward from it and recover the rest of the causal chain.

(Detect Then Reconstruct)

If we think about it this maps naturally onto how a SOC actually works: an analyst doesn’t need the model to flag all thirteen malicious edges in an incident, they need one alert to trigger an investigation, after which normal incident-response process (following identifiers, pivoting on hosts and sessions) does the rest.

This project tested that pattern empirically on its own real held-out predictions.

Step 1: does at least one alert fire?

Yes, almost always, for both models, covered above in the first-alert-rank table. Whilst I am repeating myself here it’s important to note it’s the necessary precondition for everything that follows: if the first alert never fires, there’s nothing to expand from.

Step 2: does one alert’s neighborhood recover the entire malicious subgraph?

Essentially no, when seeding from each scenario’s own single highest-scoring true-positive edge, and doing a pure structural breadth-first search outward over the observed graph only the scenario F2 reaches 100% coverage from a single seed. Every other scenario plateaus well short of full coverage.

A4 makes a good case study for why. Direct inspection of its malicious- adjacent edges shows its 7 malicious nodes split into 5 structurally disjoint clusters – the DCSync process and its immediate neighbors form one cluster; a session/process/socket/service hub forms a second; a second, entirely separate session forms a third; and two individual nodes are isolated, their only edges leading to non-malicious neighbors. The root cause: EvidenceForge mints a fresh session or process identity per anchoring event, rather than reusing one identity across a whole authored narrative, so what a human analyst would naturally call “one attacker session” can show up as two or three entirely separate graph nodes with no edge connecting them, even within a single scenario’s own ground truth. This is the identical mechanism already documented for D1’s two independent attackers sharing one compromised identity shown to occur within one attack’s own footprint.

Step 3: Seeding from the model’s own top-K alerts

The single-seed BFS is a useful diagnostic, but it’s not how a SOC actually works: a real alert queue mixes true and false positives, and an analyst expands from however many alerts the top of the queue actually contains, not from a hand-picked perfect seed. Re-running the same expansion from the top-K ranked edges (true or false positive, K increasing from 1) and finding the smallest K at which BFS recovers the full malicious node set produces a bimodal result:

• 8 of 16 attack scenarios converge at K=1-3, at 100% precision – every seed alert used is a real true positive, zero false-positive budget spent. These are the scenarios where the malicious footprint is already one tight structural cluster.
• 6 of 16 converge only at K=41-69, with precision collapsed to 24-64% – dozens of false-positive alerts have to be absorbed before one happens to land near the missing structural island.
• 2 of 16 (A12, F1) never converge even at K=80.

The critical finding here: whether seed-and-expand works at all is predictable from graph structure, not from model confidence. The scenarios that fail don’t fail because the model scored them poorly – many of the “Regime B” scenarios’ true positives are ranked well inside the top 20. They fail because the ground-truth malicious footprint itself is structurally fragmented, so no amount of additional high-confidence alerts helps once every already-connected cluster’s own alerts are exhausted – only a coincidental false positive landing near the missing island bridges the gap.

Next Steps

I still have a laundry list of things I want to do with this project. The three major ones are:

• Build a lot more varied scenarios,
• More deeply explore the possibilities and capabilities behind the idea of detect then reconstruct
• Work out how the data pipeline could be migrated to ingest real world datasets rather than being tied to EvidenceForge output

My plan was a two part project but largely because i’ve been enjoying ths process of learning and exploring so much I want to continue to improve and iterate and expand the capabilities. I’m delighted that my core hypothesis was confirmed although either result would have been educational and I am also happy a second round of literature reviews show academic research as recently as April (after this project started) is also exploring a very similar 3 scope PIDS idea.

References

• Hu, Z., Dong, Y., Wang, K., Sun, Y. “Heterogeneous Graph Transformer.” WWW 2020. arXiv:2003.01332 (the model family this project builds on directly via PyTorch Geometric’s HGTConv)
• Han, X., Pasquier, T., Bates, A., Mickens, J., Seltzer, M. “UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threats.” NDSS 2020. https://arxiv.org/abs/2001.01525
• Goyal, A., Wang, G., Bates, A. “R-CAID: Embedding Root Cause Analysis within Provenance-based Intrusion Detection.” IEEE S&P 2024. https://gangw.cs.illinois.edu/rcaid-sp24.pdf
• Cheng, Z. et al. “KAIROS: Practical Intrusion Detection and Investigation using Whole-system Provenance.” arXiv:2308.05034
• “ORTHRUS: Achieving High Quality of Attribution in Provenance-based Intrusion Detection Systems.” USENIX Security 2025. tfjmp.org/publications/2025-usenixsec.pdf (also the source for the VELOX comparison, per that same body of work)
• “MAGIC: Detecting Advanced Persistent Threats via Masked Graph Representation Learning.” USENIX Security 2024. arXiv:2310.09831
• “Beyond Nodes vs. Edges: A Multi-View Fusion Framework for Provenance-Based Intrusion Detection” (PROVFUSION). arXiv:2604.14685
• “PIDSMaker: Building and Evaluating Provenance-based Intrusion Detection Systems.” arXiv:2601.22983
• “Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks.” arXiv:2112.14936
• “GRANOLA: Adaptive Normalization for Graph Neural Networks.” arXiv:2404.13344
• EvidenceForge: https://github.com/Cisco-Talos/EvidenceForge