One thousand crashes a night: how AGELAIOS tests failure containment

On a tactical command node, a single software failure can have consequences far beyond the process that caused it. A mapping module may exhaust memory. A third-party service may enter a restart loop. A sensor adapter may stop without releasing its network interfaces. If the platform cannot contain these events, one defective component can degrade every other service and eventually compromise the Common Operational Picture.

AGELAIOS treats that risk as an engineering requirement, not as a promise. A module failure must remain inside the module’s assigned boundary. Supervision must detect the failure, recover the resources and apply the configured restart or revocation policy. Meanwhile, unrelated modules and the Common Operational Picture must remain available.

That is the anti-cascade requirement. Every night, the reference continuous integration environment challenges it with controlled failures. The headline number is 1,000 crash and restart cycles, but the number alone is not the evidence. The evidence comes from what the campaign measures, how failures are reproduced and whether the test can detect a deliberately broken configuration.

Start with decisions that can be replayed

Recovery begins with supervision. This component decides when a module starts, when it may restart, when delays escalate and when repeated failure should lead to revocation.

At the core of AGELAIOS, those decisions are represented as a pure state machine. It receives a current state and an event, then returns a new state and a list of actions. It does not consult the system clock directly, open sockets or modify the filesystem. Even time enters the model as an explicit event.

This design produces two practical benefits.

The first is determinism. Feed the same events into the same initial state and the supervisor must make the same decisions. A sequence involving rapid crashes, overlapping timeouts and revocation during recovery can therefore be exercised in memory, quickly and repeatedly.

The second is reproducibility. When a test campaign reveals an unexpected decision, the relevant events and timings can be captured and replayed as a regression test. A difficult sequence is no longer described only in a log or left to chance. It becomes an executable case that the project can keep.

This level of testing is necessary, but it is not sufficient. A state machine can validate decisions without revealing a leaked mount point, an orphaned cgroup or a virtual interface left behind by a process that died abruptly. Those failures only appear when the software meets an operating environment. That is where the chaos campaigns begin.

Chaos v0: make small leaks impossible to ignore

The first campaign focuses on the basic module lifecycle. On emulated x86-64 and ARM64 nodes, the harness launches a real module process and terminates it without warning. The module receives no graceful shutdown period and no opportunity to clean up after itself.

Supervision must then complete a precise recovery sequence:

1. detect the termination;
2. classify the module state;
3. reclaim the resources associated with that instance;
4. apply the module’s restart policy;
5. return the node to a defined steady state.

The continuous integration campaign repeats this sequence 1,000 times. It verifies two invariants throughout the run.

Unrelated services remain available. The harness continues to probe the other node services and the Common Operational Picture. A recovered module is not enough if the recovery operation disrupts the rest of the platform.

The resource inventory returns to its baseline. Processes, mount points, virtual network interfaces and cgroups are inventoried before and after the campaign. The final inventory must match the initial one.

This explains why the campaign runs 1,000 cycles. The purpose is not to suggest that 1,000 failures represent every possible operational condition. Repetition amplifies cumulative defects. A leak of one object per restart can look harmless in an isolated test and become unmistakable after a long sequence.

Chaos v1: recovery under concurrent pressure

Real incidents rarely arrive in a convenient order. Power instability, a shared dependency or a defective update can affect several modules within the same time window. Recovery activity can then overlap with new failures.

Chaos v1 introduces that pressure. The node hosts 16 modules while the harness terminates random subsets in simultaneous bursts. Some failures occur while earlier restarts are still in progress. This combination exercises event queues, lock boundaries, resource accounting and restart throttling under concurrency.

Success is defined as convergence, not merely survival. Once fault injection stops, every module must reach a state allowed by its policy, either active or deliberately stopped, within a bounded period. The node must also satisfy the same resource inventory invariant used in chaos v0.

Each run records its random seed. If one particular storm exposes a race or an unexpected transition, the team can recreate the same sequence instead of waiting for probability to reproduce it.

A green test needs a credible way to turn red

Automated tests can pass while testing nothing useful. A fault injector may target the wrong process. A health check may query the wrong endpoint. Two empty inventories may compare as equal even though collection failed.

For that reason, each chaos campaign includes a negative control. The harness runs a deliberately weakened configuration in which an isolation protection is removed and the resulting failure is expected to escape its normal boundary. This run must fail.

If the weakened configuration passes, the campaign is invalidated. The result shows that the harness cannot detect the defect it claims to cover. This control does not make the test infallible, but it gives a successful campaign substantially more evidential value than a sequence that only knows how to report green.

Containment is a system of boundaries

No single mechanism provides the whole anti-cascade property. AGELAIOS combines several controls, with the execution profile selected according to the module and its risk:

• Linux namespaces give a module a restricted view of processes, mount points, network resources and identifiers [2]. They reduce what the module can observe and affect.
• Cgroups place explicit limits on CPU, memory, process count and input/output [3]. A runaway workload is constrained by its own budget instead of consuming the node without bound.
• Seccomp limits the system calls available to a module [4]. Reducing the permitted kernel interface also reduces the actions available to faulty or compromised code.
• MicroVMs provide a hardware virtualization boundary for profiles that require stronger separation, following the lightweight virtual machine model demonstrated by projects such as Firecracker [5].
• WASI applies an explicit capability model to WebAssembly modules [6]. Access to files, networks and clocks is granted rather than assumed.

These mechanisms are not presented as five identical layers applied to every workload. Namespaces, cgroups and seccomp form a Linux isolation profile. MicroVM and WASI execution address different module types and assurance needs. What matters operationally is that the selected boundary is explicit, enforced and included in the relevant test campaign.

The negative control connects this architecture to observable behavior. Removing a protection must change the outcome of the test. Otherwise, the protection exists only in the diagram.

What an evaluator can conclude, and what remains outside the test

The campaigns provide repeatable evidence for a defined scope. They show how the reference implementation responds to the injected failures, on the tested emulated architectures, with the stated module configurations and invariants. They do not prove that every possible failure is contained, nor do they replace testing on target hardware or assessment of a deployment-specific policy.

That boundary is important for a serious acquisition or integration decision. It turns a broad claim such as “the platform is resilient” into questions that can be examined:

• Which failures are injected?
• Which services must remain available?
• Which resources are inventoried?
• How quickly must the node converge?
• Can a failed run be reproduced?
• Does the harness reject a known-bad configuration?

At publication, chaos v0 and v1 run alongside the other suites in the reference repository, which reports approximately 930 automated tests across 9 continuous integration pipelines. The harnesses, invariants and logs are intended to be inspected and replayed. An industrial evaluator can run the same scenarios with its own modules and policies, then retain the results as part of its assurance record.

The emulator documentation provides the accessible starting point for a single injected failure. The full campaigns extend that method to endurance and concurrent fault storms.

For AGELAIOS, anti-cascade behavior is therefore more than a feature statement. It is a requirement with observable acceptance criteria, a repeatable campaign and a control designed to challenge the campaign itself. That is the level of evidence on which a resilient tactical platform should be evaluated.

Sources

1. Principles of Chaos Engineering, “Principles of Chaos Engineering”, 2019.
2. man7.org, “namespaces(7), Linux manual page”.
3. man7.org, “cgroups(7), Linux manual page”.
4. The Linux Kernel documentation, “Seccomp BPF”.
5. Firecracker, “Secure and fast microVMs for serverless computing”.
6. WASI, “WebAssembly System Interface”.
7. A. Basiri et al., “Chaos Engineering”, IEEE Software, vol. 33, no. 3, 2016.