A scientific workload, a website, and the engineering between a heartbeat and a safe restart
Dr. Julio C. Spinelli · Technical application note · 20 September 2026 · Reviewed article 0.3
v0.2.0 ZIP. A source review found defects that prevent the portable package from completing physical commissioning and leave important recovery transitions unimplemented. Do not arm or enable unattended recovery with the unchanged v0.2.0 package. Use it for source review and offline exercises; live commissioning requires a separately versioned, corrected and tested release. The corrections below explain the blockers. The published ZIPs and their checksums are unchanged.Abstract
An unattended computer can lose its Ethernet interface while its processor continues a long scientific calculation. Conversely, a website can fail while the operating system and network remain healthy. A room-wide power interruption can remove the computer, power controller, router, modem and switch together. These failures require different responses.
This note describes a layered recovery system built around an iBoot-G2+, an independent USB heartbeat, Ubuntu services, verified scientific checkpoints and persistent recovery records. It begins with the questions sent to Dataprobe support, follows the measurements and implementation choices, and explains the September 2026 outage that exposed a software integration defect. The central engineering requirement is to restore service without treating an unverified scientific state as a valid checkpoint, or an unverified recovery attempt as success.
No UPS was installed. A watchdog stores no electrical energy and cannot save work that was only in memory when utility power disappeared. The portable software supplies a reference implementation with live actions disabled at installation, but its passing offline suite did not establish a complete operational lifecycle. The review in this edition identifies the missing behavior and the evidence required before a corrected release can be used.
- Understand the system: §§1–5 explain the hardware, responsibilities and heartbeat.
- Audit the work: start with the new source-review findings, then the review ZIP walkthrough and evidence classifications.
- Prepare an implementation: read Appendix A and the blocked physical-test stages in Appendix B. They specify what a corrected release must demonstrate.
- Integrate a real calculation: use §7 and Appendix C; the bundled integer-square sample is not a scientific application adapter.
- Ask an agent to implement it: copy the complete GPT-6 Astra Max prompt in Appendix E.
Key terms in plain language
| Term | Meaning |
|---|---|
| ACK | The exact iBootACK response. It proves a recent exchange, not scientific progress or outlet configuration. |
| HOLD / quiescent / READY | HOLD asks the workload to stop admitting work. Quiescent means its writers have stopped. READY is a separate verified statement that its saved state is valid and safe for the planned action. |
| Source binding | A recorded hash ties an accepted configuration or receipt to the exact code and service definitions it authorizes. |
| Durable intent | An action record written to persistent storage before dispatch. An uncertain result keeps that record; restarting the manager does not permit another attempt. |
| Reset allowance / epoch | One permission to attempt a controller reboot. An epoch identifies one generation of that permission; the old spent record remains. |
| Fence / INHIBIT | Persistent conditions that block further actions while an incomplete or unsafe transition is investigated. |
| Guardian | A role in the historical design. In the portable ZIP, a controller mutex exists, but the claimed scientific drain on heartbeat loss is missing. |
| Commissioning | Proving the actual target’s behavior with fresh configuration, workload, timing and physical observations before unattended use. |
Source review: corrections and release blockers
These findings apply to the published portable v0.2.0 source. They do not invalidate the separately recorded historical RecipeGram observations. The timer failure was reproduced offline using the real Context.unit() with an inert command runner; no host service or device was contacted. The other findings follow the corresponding implementation paths. Fixing this article does not patch the downloaded software.
| Finding | Observed behavior and consequence | Required correction |
|---|---|---|
| Physical-test gate rejects a timer | core.Context.unit() accepts only .service names. device._manager_must_be_inactive() also requests iboot-reference-manager.timer, so power-test, accept-test and abort-test fail with INVALID_UNIT. No manager timer is generated by this package. | Correct unit-type and absent-unit handling; exercise the real Context and actual generated unit set. Do not arm while the test and abort paths are unavailable. |
| Heartbeat-loss drain is absent | heartbeat.py does not call workload preparation or create a science HOLD. The manager blocks with HEARTBEAT_UNHEALTHY_GUARDIAN_OWNS_QUIESCENCE, but no separate drain implementation supplies that promise. | Design and test timely workload containment for persistent heartbeat loss while the host still runs. A kernel lockup or abrupt AC loss still cannot guarantee a graceful drain. |
| Postboot continuation and health gate are incomplete | Pretest preparation leaves a durable HOLD. The worker respects it after boot. resume returns STARTED_GATE_PENDING with healthy:false; CLI status is not a science health check, and ordinary CLI health requires the manager to be active. | Explicitly verify READY, resume once, then prove fresh progress or validated completion through a reviewed commissioning-health interface while the manager stays inactive. |
| Recovery proof must precede the test reboot | When arming consumes a controller-reset allowance, proof needs that same live heartbeat invocation and host boot. The existing sequence never explicitly captures it before stopping heartbeat. | Persist successful recovery proof after four regular ACKs and before any planned stop/reboot. This starts evidence/cooldown accounting; it does not grant an early replacement allowance. |
| Recommissioning and maintenance have lifecycle gaps | An upgrade changes the configuration binding, invalidating the old physical receipt. Existing test records block a new test/acceptance. A spent allowance can also prevent rearming from MAINTENANCE because renewal requires ARMED. | Add a versioned reacceptance transaction and a coherent maintenance policy that retain all test and spent-action history. Waiting in maintenance or deleting records is not a repair. |
| Evidence ownership and final activation were misstated | Under the documented sudo invocation, capture validation accepts root-owned files. The original guide also omitted manager and renewal/mail activation. | Use reviewed root-owned private capture copies. Add explicit postacceptance activation gates and exact unit names; see Appendix B §6.1. |
Validation boundary: 210 tests previously passed under four masks (840 executions, UID 0). That result remains historical evidence about those exercised paths. It did not test the real timer lookup or establish the missing whole-system behavior. A corrected release needs targeted regressions, ordinary-user tests and target commissioning before any production claim.
1. The questions that started the project
The initial question was not simply whether the controller could cycle an outlet. It was whether a Linux application could maintain a well-defined heartbeat while Ethernet was unavailable, stop a scientific workload safely when recovery required a reboot, and avoid a repeated power-cycle loop when infrastructure returned late.
A message sent on 9 September 2026 asked Dataprobe about the USB exchange, Linux serial behavior, timing, suspension, autonomous switching, retry limits and power restoration. The 10 September reply clarified several points and recommended hands-on tests; the responder did not claim to have tested this Linux implementation. Its attached March 2021 CLI extract did not contain the raw heartbeat packet format. These are dated support statements, not a certification of this application.
Current document retrieval also located the separate iBoot Utility V130917E manual. Its protocol section supplies the case-sensitive heartbeat and response tokens and serial settings. Thus, “the main manual did not settle our implementation questions” is accurate; “Dataprobe never documented a heartbeat protocol” is not. Neither that utility section nor the support reply specifies a line terminator. The exact byte exchange below comes from the deployed source and its tests. [1, 2, S1]
1.1 Questions, answers and remaining work
| Engineering question | Vendor documentation or dated support answer | What the application must establish |
|---|---|---|
| What is sent and returned? | Support and the utility manual identify iBootHB, iBootACK and iBootNAK. |
Exact framing, bounded reads, stale-buffer handling and acceptance of fragmented responses. |
| Does heartbeat require a login? | Support says no. Configuration commands use the authenticated CLI. | Keep heartbeat and management sessions distinct. Status queries are not heartbeat substitutes. |
| Which serial settings and Linux driver? | Documentation gives 9600 baud, 8 data bits, no parity, one stop bit and no flow control. Support points to FTDI virtual COM support without a Linux-specific guarantee. | Discover the actual USB identity and stable device path; verify permissions and serial-open behavior on the target. |
| When does monitoring start? Is there boot grace? | Support says it starts after controller power-up/reboot and has no additional heartbeat restart delay. | Measure boot-to-first-ACK with the router unavailable. Do not use the AutoPing restart setting as heartbeat grace. |
| Can maintenance suspend enforcement automatically? | action none suspends the heartbeat power action. Support says it has no automatic restoration timer. |
Read back the effective state, record who owns maintenance, and explicitly restore the intended configuration. |
| What do live changes reset or preserve? | The reply does not completely define countdown reset, persistence or NVM behavior. | Verify saved settings and behavior. Avoid unnecessary repeated writes. |
| Can Ubuntu request a delayed, autonomous graceful cycle? | The reply discourages relying on the legacy Graceful Shutdown facility for this Linux/USB use case. | Implement workload preparation in host software. Do not invent a documented atomic delayed-cycle command. |
| How many cycles occur and what rearms them? | Support describes bounded actual cycling and a valid heartbeat allowing response to a later failure. Counter behavior is not identical to physical switching. | Observe the load; do not equate a growing trigger counter with another relay operation. |
| What happens after total AC loss? | Support describes Initial State ON and immediate heartbeat monitoring on restoration. | Verify BIOS AC-restore behavior, retained device settings and infrastructure startup ordering. |
| Does Disable OFF prevent the intended timed cycle? | The support answer was qualified; documentation distinguishes timed cycling from persistent OFF operations. | Test the chosen configuration on the actual firmware and load. |
| What firmware applies? | Support recommended 1.62.117 for G2+/G2S on 10 September. | Record the unit's own version. Model family compatibility does not prove identical behavior on every release. |
The web manual gives a failure-count range of 1–99 while its CLI listing gives 1–999. The support reply does not resolve that difference. The reference value 30 lies within both; do not generalize beyond the verified range.
The full companion evidence matrix separates each question, support answer, document location, implementation evidence and unresolved point. Private mail headers, signatures and the original correspondence are excluded from the public package.
2. Why a single watchdog is insufficient
A heartbeat proves a narrow fact: some code executed an exchange recently enough for the receiving timer. It does not prove that the website returns useful content, that the scientific workers are advancing, or that an emergency recovery service can still execute its pinned source files.
A ping proves reachability to one destination under the conditions of that probe. A PID proves that a process identifier currently exists. Neither proves numerical progress or the absence of an integrity block. Likewise, a single absent ACK can mean transient transport trouble; it is not sufficient evidence that the host has locked up.
The reference installation therefore observes separate layers:
| Layer | Useful evidence | What that evidence cannot establish by itself |
|---|---|---|
| Electrical path | Actual load ON/OFF observation; configured initial state | Filesystem integrity or saved scientific work |
| USB heartbeat | Fresh exact ACKs from the current service invocation | Effective outlet configuration or scientific progress |
| Primary Ethernet | Interface presence, link, address, expected route and appropriate connectivity probes | Website/backend correctness |
| Optional Wi-Fi | Address, source-specific route and configured reachability | External WireGuard or inbound NAT availability |
| Application | A defined safe backend health transaction and public endpoint checks | Scientific continuation |
| Scientific workload | Valid committed results, accepted startup/canary checks, current progress or validated completion | Future progress under every failure |
| Recovery machinery | Source bindings, unit results, budgets, absence of HOLD/open failed incident | Successful future physical switching |
A host lockup can prevent all host-side checks from running; the external timer is valuable precisely because it continues outside that execution environment while AC remains available. A lost router, however, cannot be repaired indefinitely by restarting a healthy server. The software must classify the fault, bound its actions and preserve an unresolved result when it cannot establish recovery.
2.1 Priorities and failure boundaries
For a planned intervention, the preferred progression is observation, bounded repair, verified preparation, and only then the authorized disruptive action. If connectivity returns before the reboot handoff, scientific work can resume without an unnecessary reboot.
For abrupt room-wide power loss there is no opportunity for the host to prepare. Durable result publication and restart validation become the defense. The already committed scientific results must be distinguishable from incomplete attempts, and the calculation must reject an incompatible continuation environment.
Backups remain necessary for failed storage, deletion and corruption outside the checkpoint contract. A UPS would address a different problem—providing energy for a controlled shutdown—but was not part of this installation. All hardware interfaces here are external; the design does not require an internal ATX modification.
3. What the system contains
Figure 1. Original engineering diagram. Main supplies the host. Router, modem and switch are separate loads, though they can share the same room-level AC failure. USB carries the heartbeat; Ethernet carries management. The latter path still requires working management connectivity.
Figure 2. Responsibility boundaries. Evidence records link attempts to their outcomes. An email is a report about those records; it is not the authority that permits another reset.
3.1 Five mechanisms that must not be confused
| Mechanism | What changes | Governing condition |
|---|---|---|
| Hardware heartbeat expiry | iBoot cycles the Main outlet, removing host AC for its configured interval | Missed heartbeat threshold and the device's configured action/cycle limit |
| Ubuntu warm reboot | The host requests an operating-system restart | Governed recovery, valid scientific preparation and current preconditions |
| iBoot-controller reboot | Controller firmware restarts to recover management/heartbeat communication | A bounded, recorded controller-recovery procedure; this is not an outlet-cycle command |
| Hardware rearming | A later heartbeat/failure episode can again cause the configured hardware response | Firmware behavior clarified by support and operator-reported project tests (§9.2); not the R24 ledger |
| R24 allowance renewal | Software makes one future controller-reboot allowance available | Verified successful recovery, a full 24-hour cooldown and current accepted health/evidence |
R24 does not schedule a daily reboot. It does not add 24 hours to the hardware watchdog timeout. It does not replenish a failed or uncertain controller recovery. An unused allowance remains one; it does not accumulate into a queue of resets.
3.2 Division of responsibility
The iBoot owns its outlet timer. The USB process owns the recurring exchange and bounded communication recovery. systemd owns service invocation identity and startup. The workload adapter owns preparation, readiness and validated continuation. The network/application observers identify a fault domain. The disaster manager owns an incident and coordinates bounded recovery. The controller guardian prevents uncontrolled reuse of a controller-reset allowance. The R24 observer evaluates whether a new allowance is justified. The email queue retains reports while connectivity is unavailable.
A private, persistent record is needed before dispatching an action. If the process dies after dispatch but before recording completion, the outcome is uncertain, not a fresh opportunity to repeat the action. A software update must not erase spent records simply to make a new version pass.
4. Compatibility: demonstrated facts and portability objectives
| Component | Demonstrated or inspected reference | Portability boundary |
|---|---|---|
| Controller | iBoot-G2+; firmware 1.62.117 reported in captured device output | Other models/releases require their own tests. A G2S has different network hardware. |
| Ubuntu host | Ubuntu 26.04 LTS in the project; captured kernel 7.0.0-31-generic, systemd 259.5-0ubuntu3.4 |
This is one installation, not certification of all Ubuntu releases. |
| USB | FTDI serial connection; exact path and permissions discovered locally | Do not copy a different unit's serial identifier or assume /dev/ttyUSB0 is stable. |
| Workload | Stage10c with its R2 scientific continuity implementation | Future workloads need an explicit adapter and validation contract. |
| Recovery | HB4 T2, C2 F1, O3 repair, R24 O3 F1 | Historical ZIPs bind exact local paths, users, source hashes and receipts. They are not generic installers. |
| Reference companion | Version 0.2.0; implemented recovery components and 840 offline test executions | Separate portable derivative; target-host and physical acceptance are still required. |
Dataprobe lists 12 A at 105–125 VAC and 10 A at 210–240 VAC, with IEC C14 input and C13 output. Check the actual model label, regional cord and connected load against the applicable rating. Do not assume a power strip of unrelated loads is equivalent to the single intended host. The G2+ product page describes one 10/100 network port; the two-port switch belongs to the G2S. [3]
The current product firmware link names 1.62.117, but the inspected revision-history page ends at 1.61.116 (June 2022). This discrepancy is recorded rather than silently calling the older listed release “latest.” This article does not prescribe a firmware update. [3, 4]
4.1 Genuine hardware and screen evidence
Enclosure
Input · USB · Ethernet
Main output
Figure 4. Genuine project screenshot of an early disarmed heartbeat setup, uploaded 13 September 2026. It displays Source USB, Frequency 10 seconds, Fail Count 3, Action None, Main. The inactive Cycle field shows 1. This is a configuration illustration from the early test phase, not evidence of the later production values or an armed outlet. The original pixels are retained; no interface or measurement has been generated.
5. The byte exchange and communication recovery
Historical implementation. The retry timings and T2 observations in this section describe the accepted RecipeGram heartbeat helper. They are not a statement that the portable ZIP has passed the same live tests.
The deployed heartbeat sends exactly seven ASCII bytes, with no appended carriage return or line feed:
| Direction | Bytes | Hexadecimal |
|---|---|---|
| Host → controller | iBootHB |
69 42 6f 6f 74 48 42 |
| Controller → host, positive token | iBootACK |
69 42 6f 6f 74 41 43 4b |
| Negative token described by vendor | iBootNAK |
69 42 6f 6f 74 4e 41 4b |
These heartbeat bytes are distinct from authenticated CLI commands, whose command-line termination is handled by the management implementation. Do not append \n merely because a terminal-oriented command normally uses it. The deployed parser checks the eight bytes it reads; the paragraph below states its trailing-byte limitation explicitly. [E-HB, 2, S1]
The accepted helper uses pyserial at 9600/8N1 with xonxoff=False, rtscts=False and dsrdtr=False. It requests DTR and RTS low before opening, uses an exclusive serial open and a separate checked flock, waits 0.25 seconds after open, and clears the buffers. This is the requested software state; it does not prove that every USB driver suppresses every opening transient.
The normal interval is 20 seconds. A write has a one-second timeout, and the ACK is accumulated in bounded read slices under a two-second deadline. The implementation reads up to eight bytes and requires that result to equal iBootACK; any excess bytes are handled as stale input on the next exchange. It does not test for a ninth trailing byte in that same read. A potentially unbounded flush/tcdrain is deliberately avoided. The ACK supplies the evidence of transmission.
After a failed first exchange it waits one second and retries on the same port. After a second failure it closes, waits, reopens and tries a third time. Only then may the bounded controller-recovery path run, with the existing allowance reserved durably before dispatch. An uncertain or partial dispatch stays spent. After controller reboot it waits ten seconds, then permits at most six reopen/ACK attempts, five seconds apart on failure. Healthy USB exchanges continue even when no reset allowance remains.
The T2 live report records controller command dispatch at 2026-09-18 14:08:35.754 UTC and a post-reboot ACK at 14:08:46.794 UTC, followed by four regular ACKs. The approximately 11.04-second interval includes deliberate waits; it is not an exact controller boot-time measurement. The recorded result is PASS_HEARTBEAT_RESTORED_NOT_ARMED, so this was communication recovery, not a Main-outlet cycle. [E-HB]
The local counter called “sequence” identifies exchanges in the helper's log; it is not a sequence field transmitted inside iBootHB. A missing log sequence therefore does not establish a periodic device protocol defect.
5.1 An isolated missed acknowledgment
The historical incident is identified in the project as sequence 327, not 308. The operator's correction names “September 14 at 23:30 EST.” That zone label is retained as reported; it has not been silently relabeled EDT or converted into a precise UTC event. Earlier diagnostic recollection places the last successful ACK at sequence 327 before helper failure, while the operator calls it the missed-heartbeat incident. The raw aligned transcript is needed to resolve that boundary exactly.
A later manual probe returned the exact ACK with the device still present. That observation supports bounded retry and serial reopening before escalation. It does not establish a periodic firmware failure, nor justify restarting the entire scientific workload after one missed response. [H1: operator report; raw incident transcript pending]
5.2 Controller management is a separate failure domain
The verified controller command is reboot, issued only after the management session is established and the required state is checked. set main cycle is an outlet command and must never be substituted for it. Controller reboot itself still needs a tested Initial State policy; a claim of “controller only” must be verified against the actual outlet behavior.
The T2 implementation handles fragmented prompts, prompt/echo differences, and a firmware case where the first expected prompt is absent. The operator reported an interactive Telnet login overlapping the September 18 recovery problem. Separate captured results establish failed USB exchanges and refusal of controller recovery when a valid login could not be established. The recovered archives do not independently reconstruct the entire interactive-session timeline. An interactive session must not be left open during automated operation. [E-HB]
Telnet is plaintext. Keep controller management on a protected management network reachable through the intended local or VPN path; do not expose it through Internet port forwarding. Store credentials through the target's protected local procedure and exclude them from logs and article assets. USB heartbeat independence from Ethernet does not make a Telnet recovery command independent of Ethernet or other working management connectivity.
6. How to prepare an Ubuntu target
This section separates reproducible exercises available now from the production acceptance work still required. Do not run the historical RecipeGram installers on another host. Do not replace an active heartbeat service with the companion package.
6.1 Inventory once, parameterize once
Maintain one configuration inventory. Values are selected from the target, not copied from private reference-machine identifiers.
| Inventory item | Required evidence or rule |
|---|---|
| Run user/group and permissions | Numeric identity, serial group membership, private state owner, service privileges |
| Paths | Immutable versioned code directory; separate durable state and evidence directories |
| USB device | A discovered /dev/serial/by-id/ path; verified physical controller match |
| iBoot management | Address on the protected network, model/version, exclusive-session ownership |
| Main outlet | Physical cable tracing and a confirmed harmless test load before host commissioning |
| Primary NIC | Device name and hardware identity; supported recovery method for this driver/platform |
| Optional Wi-Fi/WireGuard | Explicitly configured interfaces, routing rules and independent external access tests |
| Services/endpoints | Unit names and a safe application/backend health transaction |
| Workload adapter | Version, input/environment identity, progress semantics, prepare/ready/resume contract |
| Secure local transport configuration, recipient, acceptance evidence and inbox confirmation | |
| Timing/limits | Heartbeat period, hardware threshold, legitimate boot bound, preparation policy, action budgets |
Command A — target Ubuntu host, Bash, ordinary user. Read-only inventory. It does not open a serial port, inspect environment/credential files, change networking or contact the iBoot. Expected result: Ubuntu/systemd/Python information and actual interface/serial candidates. A missing directory or tool is a recorded compatibility gap; do not substitute a guessed device.
(
set -eu
uname -srmo
cat /etc/os-release
systemd --version
python3 --version
id
ip -brief link
ip -brief address
ip route show
if [ -d /dev/serial/by-id ]; then
ls -l /dev/serial/by-id/
else
printf 'USB_STABLE_PATHS=NOT_PRESENT\n'
fi
)
INVENTORY_STATUS=$?
printf 'INVENTORY_STATUS=%s\n' "$INVENTORY_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Keep the full inventory private. Redact addresses, serial/MAC identifiers and user paths before publication. Confirm the BIOS setting that restores power after AC returns and test it in a separate commissioning session; its name varies by motherboard. BIOS configuration alone does not prove the host actually restarted after this event.
6.2 Physical setup and disarmed controller configuration
Trace the iBoot input to its AC source. Trace Main to the test load. Connect USB to the intended Linux host and Ethernet to the protected management segment. Initially use a harmless lamp within the ratings and keep the server outside the switched test path.
For controller setup, use the actual device UI and CLI documentation. The captured early screen above shows a disarmed state. Before connecting a production host, separately verify Device, Heartbeat, AutoPing, Schedule and Graceful Shutdown settings. Disable or account for every active automation capable of changing Main. Some setting changes require a controller reboot; do not assume saving a web form makes every setting effective immediately.
| Setting | Accepted reference design | Reason / verification |
|---|---|---|
| Initial State Main | ON | Host must receive AC on restoration; confirm BIOS behavior too. |
| Heartbeat source | USB | Separate host liveness transport from failed primary Ethernet. |
| Device Frequency / Fail Count | 60 s / 30 | Nominal 1,800 s threshold, distinct from the helper's shorter transmit interval. |
| Setup action | None | Observe communication without a heartbeat-triggered outlet action. |
| Commissioned action | Cycle | Only after the full arming and physical acceptance procedure. |
| Selected outlet | Main | Confirm by tracing the physical load, not by the screen label alone. |
| Main cycle duration | 30 s | Chosen and physically checked reference interval, not an assumed factory default. |
| Hardware cycle maximum | 1 for the accepted production design | One cycle for a continuous loss episode. A greyed field or incomplete CLI readback is not proof of an effective armed value. |
| Disable OFF | Yes in the captured reference readback | Timed-cycle behavior still requires actual verification. |
| AutoPing | All three actions None in captured inventory | Read back the target; do not assume factory defaults. |
| Legacy Graceful Shutdown | C2 accepted a disable command for Main | Setter acceptance was recorded; independent effective-state readback was not. |
| Schedule | Target inspection required | A complete current schedule capture has not been verified in this edition. |
Command B — iBoot management CLI, authenticated administrator, one session only. These are readback commands, not an arming sequence. They query settings. Save the returned values, then close the session. If a command is rejected, a prompt is ambiguous or results disagree with the UI, stop the setup and resolve the version-specific behavior. Do not issue a speculative power command to recover the session.
get heartbeat
get cycle
get initial state
get disable off
logout
Readback proves the values the controller reports, not every internal timing or retry state. In particular, the captured get heartbeat output does not expose every cycle-budget detail. Keep a genuine UI capture and physical test record for settings that lack sufficient CLI evidence. An ACK alone cannot prove Action Cycle survived a room-wide power loss.
6.3 Reproducible reference exercises
Extraction prerequisites: Ubuntu procedures use unzip and sha256sum. Check command -v unzip and command -v sha256sum first. If missing, install the Ubuntu packages unzip and coreutils using your normal package-management procedure before extracting either archive. The review-only workstation does not need serial hardware or systemd services.
The companion downloads are available on the author's website:
| Download | Contents | SHA-256 checksum |
|---|---|---|
| iboot_ubuntu_reference_v0.2.0.zip | Portable v0.2.0 reference source, tests and manuals; known live-commissioning blockers | 82240026a56e5f856f68116d82d1941be804f5d0ffb9270cf7654e3ded07477f |
| iboot_article_review_materials_v0.2.zip | Evidence matrix, compatibility list, publication handoff and article source | fb091a5b8b54fa74f48d2299ae703bf1c91190c5f20c66be88f881f59358f06f |
Both published downloads were retrieved and verified against these SHA-256 values. SHA-256 is an integrity checksum, not a digital signature. The review-materials ZIP preserves the source snapshot from before these published download links were added to the HTML.
Download the portable ZIP into a review directory or a separate Ubuntu test host. Use the extraction below for offline source review and tests. Appendices A/B document its interfaces and the corrections needed before live commissioning. The checksum verifies the published bytes; it does not certify the code as ready for unattended use.
Host/shell: Ubuntu test host, Bash, download directory. Privilege: ordinary user. Effect: checksum and extraction into a fresh directory only. Expected: checksum OK and successful extraction. Failure: retain the files/output and stop; do not overwrite an earlier deployment directory.
(
set -eu
printf '%s %s\n' \
'82240026a56e5f856f68116d82d1941be804f5d0ffb9270cf7654e3ded07477f' \
'iboot_ubuntu_reference_v0.2.0.zip' | sha256sum --check -
if [ -e iboot_ubuntu_reference_v0.2.0 ] || [ -L iboot_ubuntu_reference_v0.2.0 ]; then
printf 'EXTRACTION_REFUSED_EXISTING_DIRECTORY\n'
false
fi
unzip -q iboot_ubuntu_reference_v0.2.0.zip
)
IBOOT_EXTRACTION_STATUS=$?
printf 'IBOOT_EXTRACTION_STATUS=%s\n' "$IBOOT_EXTRACTION_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Enter the extracted directory before the Appendix A verification step. Do not run the archive's historical predecessors on a new target.
6.4 Using the review-materials ZIP
This archive is the editorial and technical audit package for the article. Use it to trace claims to evidence, inspect acceptance limits, review the publication plan, and rebuild the archived article. You do not need to install it on the monitored server. Reading its Markdown files requires only a text editor; rebuilding the HTML is optional.
Version boundary: the published v0.2 ZIP is a fixed source snapshot. It predates the published download links, this walkthrough and the reviewed article 0.3 corrections, figures and implementation prompt. Its rebuild produces that earlier article edition, not a byte-identical copy of the HTML you are reading now. Its recorded HTML checksum likewise belongs to that earlier edition. The ZIP's own checksum in the download table remains valid because the ZIP has not been changed.
Step 1 — Verify and extract into a separate review folder
Download iboot_article_review_materials_v0.2.zip using the link above. Choose the Windows or Ubuntu procedure below. The archive contains several top-level directories rather than one enclosing project folder, so extract it into the explicitly named review folder. Keep the internal directory structure and image filenames intact.
Host/shell: Windows review computer, PowerShell; ZIP in Downloads. Privilege: ordinary user. Effect: verify the downloaded bytes, then extract into a new folder in Downloads. Expected: REVIEW_MATERIALS_RESULT=PASS and the folder path. Failure: preserve the output; a checksum mismatch or existing destination stops extraction. Use the existing folder for inspection or choose a separate new folder after reviewing why it exists.
$ReviewZip = Join-Path $env:USERPROFILE 'Downloads\iboot_article_review_materials_v0.2.zip'
$ReviewFolder = Join-Path $env:USERPROFILE 'Downloads\iboot_article_review_materials_v0.2'
$ReviewExpected = 'fb091a5b8b54fa74f48d2299ae703bf1c91190c5f20c66be88f881f59358f06f'
try {
$ReviewActual = (Get-FileHash -LiteralPath $ReviewZip -Algorithm SHA256 -ErrorAction Stop).Hash
if ($ReviewActual -ine $ReviewExpected) {
throw 'SHA-256 mismatch; extraction refused.'
}
if (Test-Path -LiteralPath $ReviewFolder) {
throw 'Destination already exists; extraction refused.'
}
Expand-Archive -LiteralPath $ReviewZip -DestinationPath $ReviewFolder -ErrorAction Stop
Write-Host 'REVIEW_MATERIALS_RESULT=PASS'
Write-Host "REVIEW_FOLDER=$ReviewFolder"
} catch {
Write-Host 'REVIEW_MATERIALS_RESULT=FAIL'
Write-Host $_.Exception.Message
}
Host/shell: Ubuntu review computer, Bash, in the directory containing the downloaded ZIP. Privilege: ordinary user. Effect: verify and extract review files only. Expected: checksum OK, then REVIEW_MATERIALS_STATUS=0. Failure: stop and retain the ZIP/output; an incomplete extraction is not a verified review folder and must not be silently overwritten.
(
set -eu
printf '%s %s\n' \
'fb091a5b8b54fa74f48d2299ae703bf1c91190c5f20c66be88f881f59358f06f' \
'iboot_article_review_materials_v0.2.zip' | sha256sum --check -
if [ -e iboot_article_review_materials_v0.2 ] || [ -L iboot_article_review_materials_v0.2 ]; then
printf 'REVIEW_EXTRACTION_REFUSED_EXISTING_DIRECTORY\n'
false
fi
mkdir iboot_article_review_materials_v0.2
unzip -q iboot_article_review_materials_v0.2.zip -d iboot_article_review_materials_v0.2
)
REVIEW_MATERIALS_STATUS=$?
printf 'REVIEW_MATERIALS_STATUS=%s\n' "$REVIEW_MATERIALS_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Step 2 — Read the review documents in order
Open the extracted folder in your editor. Markdown (.md) files are ordinary text; an editor with Markdown preview makes the tables easier to read. Paths below are relative to the extracted review folder.
| Order / file | How to use it | What it does not establish |
|---|---|---|
1. review/compatibility_and_remaining_tests.md |
Identify supported components, unperformed tests and missing acceptance material before relying on the deployment instructions. | It does not certify an untested host or firmware. |
2. review/evidence_claim_source_matrix.md |
Find a claim's classification, source archive, internal locator and hash. Compare the article's wording with that evidence boundary. | The private source archives and full support correspondence are not included. Request the relevant retained evidence if an independent audit needs it. |
3. review/validation_review_v0.2.md and review/html_checks.json |
Review the recorded rendering, navigation, command-syntax and archive checks. The JSON contains machine-readable browser observations. | These are saved observations for the archived edition, not fresh checks of an edited article. |
4. review/publication_handoff.md |
Review the proposed publication locations, content types, attribution, distribution permission and post-upload checks. | Reading it does not publish anything or confirm the proposed canonical page exists. |
5. BUILD.md |
Read the short build overview, then use the complete procedure below if you want a local HTML rebuild. | A successful build does not commission hardware or approve publication. |
For a technical review, record each proposed correction with the article section, claim/source identifier, the reason for the correction and the evidence supporting it. Keep documented, measured, operator-reported and unknown results distinct. Retain the original ZIP as the reference copy.
Step 3 — Understand the source tree
| Directory / file | Role when reviewing or editing |
|---|---|
article/article.md |
Main article narrative and insertion markers. Make editorial revisions here in a working copy. |
article/insertions.json |
Reusable technical passages and evidence table inserted into the narrative. Preserve valid JSON. |
article/bundle_commands.md |
Download and executable instruction section as it existed in the archived edition. |
article/style.css |
Responsive screen and print styling. Recheck both layouts after changes. |
article/build_article.py |
Expands source passages and embeds the supplied genuine images and original diagrams. |
article/markdown_to_html.mjs |
Converts expanded Markdown to the standalone HTML, navigation and metadata. |
evidence/images/Alcubierre Drive and the Spinelli Framework/ |
Four public project images used by the article. Preserve the paths; these are not a complete private evidence archive. |
portable/iboot_ubuntu_reference_v0.2.0/ |
Software/documentation snapshot used by the appendices. Inspect its README, manuals, tests and manifest. Running an installer is unnecessary for article review. |
The portable source copy is included so the article builder can read its manuals. Inspect it without running its installer. The separately published portable ZIP has the same release boundary: its known commissioning defects must be corrected before deployment. Changes to source or manuals invalidate the existing package manifest and require a separately versioned release; rebuilding the article does not update that manifest.
Step 4 — Optionally rebuild the archived HTML
Use Python 3, Node.js 20 or newer, and npm on an Ubuntu review workstation. The builder uses the marked Markdown parser; the command pins the version used for this article, 17.0.5, in a private build-dependency folder. Internet access is needed for that dependency installation. The two environment-variable names below are required by the archived builder and simply identify your local Node executable and module directory; no ChatGPT account or service is needed.
Host/shell: Ubuntu review workstation, Bash, starting inside the extracted iboot_article_review_materials_v0.2 folder. Privilege: ordinary user; no sudo. Effect: install a local Markdown-rendering dependency and generate article/expanded.md plus a standalone HTML file in deliverables/. This does not invoke any recovery installer or device command. Expected: ARTICLE_REBUILD_STATUS=0 and an HTML file larger than zero bytes. Failure: retain the build output, fix the stated dependency/path/JSON error and retry only the build; do not modify recovery state. Rerunning this block replaces the generated HTML and expanded Markdown in this working copy.
(
set -eu
test -f article/build_article.py
test -f article/markdown_to_html.mjs
test -f portable/iboot_ubuntu_reference_v0.2.0/README.md
command -v python3
command -v node
command -v npm
node -e 'if (Number(process.versions.node.split(".")[0]) < 20) { throw new Error("Node.js 20 or newer required"); }'
mkdir -p .article-build-deps deliverables
npm install --prefix .article-build-deps --ignore-scripts --no-audit --no-fund --save-exact marked@17.0.5
export CODEX_PRIMARY_RUNTIME_NODE="$(command -v node)"
export CODEX_PRIMARY_RUNTIME_NODE_MODULES="$PWD/.article-build-deps/node_modules"
python3 article/build_article.py
test -s deliverables/iboot-g2plus-ubuntu-scientific-disaster-recovery-rc-v0.2.html
printf 'REBUILT_HTML=%s\n' "$PWD/deliverables/iboot-g2plus-ubuntu-scientific-disaster-recovery-rc-v0.2.html"
)
ARTICLE_REBUILD_STATUS=$?
printf 'ARTICLE_REBUILD_STATUS=%s\n' "$ARTICLE_REBUILD_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Open the generated HTML in a browser. Check its contents list, images, wide tables, phone-width layout and print preview. Disconnect networking and reload to check offline rendering. External references and downloads still require a connection when clicked. The archived build can still contain earlier relative download links; it does not incorporate subsequent editorial changes automatically. Before publishing a revised article, review those links, recalculate the revised HTML's checksum into a separate sidecar, and follow the publication handoff. Keep the currently published v0.2 ZIP intact; distribute future source changes as a new version with a new checksum.
Step 5 — Verify this HTML edition separately
The complete-file SHA-256 is provided in the adjacent .html.sha256 file delivered with this edition. The ZIP’s archived HTML hash refers to its older snapshot. A literal hash of all bytes of this HTML cannot be inserted into those same bytes without changing the hash; the external checksum avoids that self-reference. SHA-256 is an integrity checksum, not an author’s cryptographic signature.
Save the HTML and its checksum file together, retaining the delivered filename. On Ubuntu, run the following ordinary-user check in that folder. It only reads the files; OK means the bytes match. Stop if they differ and compare the exact edition before using its instructions.
sha256sum --check -- 'Linux_unattended_crash_recovery_system_for_off_site_Management_of Servers_Using_iBoot_G2_+.html.sha256'
On Windows PowerShell, Get-FileHash -LiteralPath '.\Linux_unattended_crash_recovery_system_for_off_site_Management_of Servers_Using_iBoot_G2_+.html' -Algorithm SHA256 prints the HTML’s hash. Compare all 64 hexadecimal characters with the first field of the supplied checksum file. Any subsequent edit requires a newly calculated checksum.
6.5 What remains before production activation
The package installs inactive units with live actions disabled and contains useful source/configuration, workload, budget and physical-acceptance gates. However, the documented source defects block the physical commissioning path and leave required lifecycle behavior incomplete. First correct these defects in a separately versioned release, then validate the corrected path on the target’s actual systemd, storage, network, workload and controller. Offline review of the published ZIP can proceed now.
The RecipeGram live Ethernet repair removed and rescanned one specifically verified PCI device and applied its network configuration. That procedure is not portable to an arbitrary NIC, USB Ethernet adapter, virtual machine, cloud interface or NetworkManager setup. Copying that PCI address into a generic guide would be technically wrong and potentially disruptive. This reference supports an explicitly selected NetworkManager connection restart; it does not remove/rescan PCI devices. Hosts using another network manager need a reviewed adapter before automatic network repair can be enabled.
For the same reason, a generic health command that prints PASS is not sufficient input to a privileged reboot worker. The adapter's evidence must describe the current workload and be independently checked by the action gate.
7. Scientific preparation, readiness and continuation
The Stage10c R2 package contains a future numerical continuity requirements document, schema spinelli_numerical_continuity_requirements_v2. It is a requirements contract for future models, not an installed generic Stage10d executor. It must be reconciled with the separate scientific-integration branch when that versioned specification is available. [E-SCI]
7.1 Preparation is a protocol
A prepare request must stop new work admission and drain or otherwise resolve in-flight work according to a defined policy. Workers must publish complete results with stable work identifiers and payload hashes. A result already committed under an identifier must not be silently overwritten by a later attempt. Durable publication requires file flushing, an atomic installation strategy and directory durability—not merely a JSON flag saying “saved.”
Readiness must refer to the exact preparation attempt, current boot/workload identity and the verified committed state. It must fail if a worker remains unaccounted for or a numerical mismatch appears. A long elapsed interval does not manufacture a valid readiness receipt. An integrity mismatch and a stalled worker are different conditions and require different responses.
For Stage10c, numerical continuation includes sealed distinct-tile canaries and exact-hash checking. Ten accepted tiles must be the prescribed distinct tiles; failed comparisons must not be replaced by opportunistically resampling until ten happen to pass. Stable inputs, code/environment identity and deterministic aggregation are part of the scientific contract. The sample workload in the companion illustrates atomic records and adapter transitions; it does not demonstrate Stage10c's numerical science.
7.2 Resume and completion
After reboot, validate the environment and accepted results before scheduling more work. A running worker is not adequate proof. Establish new progress belonging to the current service invocation, or verify a durable completion result with quiescence if the job has finished.
Completed jobs need an explicit success state so the disaster manager does not mistake “no new tiles” for a hang. Corrupt, missing, stale or contradictory evidence must keep the scientific state blocked and visible. Never reset its history merely to regain a green dashboard.
A full power cut can lose uncommitted calculations. Correct recovery can preserve all valid committed results and recompute unfinished work, but that is not a claim of zero lost in-flight computation.
8. Bounded recovery, persistent budgets and R24
The recovery sequence has two purposes: attempt a proportionate repair, and establish whether it actually worked. For a network fault, keep scientific work and USB heartbeats running during the permitted nondisruptive steps. If the configured live repair attempts fail, prepare the workload. Recheck the network before the reboot handoff; a now-healthy network can make the disruptive action unnecessary.
Application-only failure must follow a service-level path. A static HTTP 200 page or an active web server does not prove a Node/database transaction is healthy. Similarly, optional Wi-Fi telemetry should not silently override the explicitly selected primary-Ethernet recovery policy. If Wi-Fi is an emergency service path, inbound routing and external access require their own test.
8.1 R24's actual policy
Historical R24 policy. The journal-proof details below describe the accepted RecipeGram observer. The portable reference has its own stored records and unresolved lifecycle gaps; see the source review and Appendix A §13.
- Proof must belong to the exact consumed epoch and current trusted heartbeat process: boot, unit invocation, PID/UID, administrator login, durable reservation, reboot dispatch, post-reboot ACK and matching recovery result.
- At least four subsequent normal exact ACKs are required. The proof logic bounds regular-ACK gaps to 60 seconds and the most recent ACK age to 45 seconds at proof capture. Missing, duplicate, out-of-order or incomplete journal evidence cannot qualify.
- The cooldown starts when trusted root-side proof is durably observed. It uses
CLOCK_BOOTTIME, including suspend, rather than calendar time. It is therefore not necessarily exactly 24 hours after the physical controller reboot. - A host boot-ID change starts a fresh full 24-hour wait at the next accepted observation. A missing timing anchor starts a new wait; wall-clock adjustment does not shorten it.
- At the due check, accepted source/receipt/unit bindings, fresh manager status, stable identities and current health must pass. HOLD, reset INHIBIT, unresolved incident, recovery failure or unresolved heartbeat guardian blocks renewal. Science must have accepted progress or validated completed/quiescent evidence.
- A grant creates one new intent epoch linked to the spent predecessor. It preserves old spent records. Partial renewal failures retain inhibition for review. The observer does not itself reboot anything. [E-R24]
The important invariant is conservation of permission: a failure cannot generate another attempt simply by restarting the observer. The observer's timer evaluates state; time passing does not cause a reboot. The first installation check had an existing unused allowance and did not exercise a real spent-allowance renewal.
8.2 Maintenance, upgrades and removal
A software HOLD can pause recovery dispatch without disabling the external watchdog. Conversely, setting the device's heartbeat action to None can leave the host unprotected even when software health looks good. Status must show both conditions separately.
For planned maintenance, establish the intended device state first, record readback and keep the heartbeat provider running until its removal is safe. A host cannot rely on a stopped daemon to send a final protective message later. If communication fails during this procedure, retain the working provider and stop the maintenance operation; do not proceed on an assumption that the outlet is disarmed.
An upgrade needs a dependency manifest that accounts for all consumers of a pinned source, not only the component being replaced. Stage code and receipts, validate the complete chain, retain the original records and perform a health check that exercises the new observer before declaring installation success. If installation is interrupted, preserve its transaction state and contain further dispatch until reconciliation determines what changed.
Rollback is not deletion of history. Restore a reviewed code/configuration set while retaining spent or uncertain action records. Uninstalling an armed watchdog's heartbeat provider is an electrical action in effect: the external timer may eventually remove host power. Verify disarming or an accepted replacement provider before disabling/removing production services. The companion package never owns that production provider.
9. The accidental outage: a useful test and a real defect
On 19 September, room power was accidentally interrupted. The server, iBoot, modem, router and LAN switch lost power. Room power was restored before the router's power supply was reconnected. That ordering challenged recovery while the host could run but the network infrastructure was still unavailable.
| Observation | Evidence and interpretation |
|---|---|
| FIT restart validation | All 746,496 FIT tiles were accepted. |
| SCORE continuation | The examined interval progressed from 146,061 to 148,961 committed SCORE tiles: 2,900 additional tiles. |
| USB heartbeat | The O1 observation window contained 267 consecutive ACKs and no heartbeat misses. The controller-reset allowance was unspent. |
| Wi-Fi address | DHCP event at boot-monotonic 523.733 s. Address acquisition alone does not prove external connectivity. |
| Ethernet address | DHCP event at boot-monotonic 581.917 s: about 9 minutes 42 seconds from monotonic zero. The earlier 9m41 figure used the first journal event as its origin. |
| Automatic network recovery | 14 attempts encountered the same captured-source mismatch. Current network health did not mean the recovery machinery was ready. |
| Alert | SMTP accepted the outage alert at 13:25:36 EDT after retry; the operator later confirmed inbox receipt. Acceptance and receipt are distinct evidence. |
| Electrical outage duration | Unknown from host logs. The powered-off host could not measure the interval directly. |
| Retained outlet action | O1 deliberately did not query iBoot settings. Fresh ACKs do not prove the saved Cycle action survived AC loss. |
The 9m42 interval is neither an exact Internet downtime measurement nor a measured electrical-outage duration. It is one well-defined boot-to-address interval. The operator supplies the cause and delayed-router context; the logs supply their own bounded observations. [E-O1]
9.1 Why healthy services were not enough
Two older recovery components still expected a pre-T2 heartbeat-file hash, while the installed helper matched the approved T2 source. Their source-verification check correctly refused the mismatch. The deployment defect was failure to migrate all dependent bindings together. Repeating the same blocked recovery did not improve it.
O2 collected the omitted receipts and ledgers. O3 updated the coordinated source bindings, contained repeated configuration failures and made readiness reject a failed stopped recovery service. Its installation retained the heartbeat and scientific process identities, observed four fresh acknowledgments, and saw 200 more SCORE tiles during the installation interval. The original records and reset budgets were preserved. [E-O2, E-O3]
R24 O3 F1 then enrolled the renewal observer without adding an allowance or restarting existing services. The report archive hash matches the accepted value, its payload hashes verify, and its first installed observer passed with the existing allowance available. A later real-world 24-hour renewal has not yet been demonstrated. [E-R24]
9.2 What commissioning established—and what is only reported here
The accepted project account reports approximately 30 minutes to heartbeat expiry and a 30-second Main OFF interval with the server attached. The recovered C2 state identifies a deployed commissioning phase and accepted postboot scientific health. The original physical timing record and complete final device capture were not recovered in the inspected archive set, so this article labels the physical interval operator-reported, distinct from the hash-verified O1/O3/R24 evidence.
The earlier lamp account describes a 70-second heartbeat run, a subsequent OFF interval, another cycle and then sustained ON. Its reported approximate intervals do not define a firmware timing specification. A valid heartbeat followed by another induced failure is the relevant rearming test; trigger-counter growth alone is insufficient. The complete September 13 rearming transcript remains part of the release evidence request.
10. Alerts should explain the outcome
“Starting recovery” records an intention. It does not tell an absent operator whether the Ethernet interface returned, whether science restarted, whether the recovery procedure was blocked, or whether the timer could act again.
The reporting states should include fault detected, recovery blocked, and recovery verified. Each message should carry an incident identity, an observation time, what was checked, what action actually ran, and what remains unknown. Persist the queue locally when the network is down; deduplicate by event identity. Distinguish queued, attempted, SMTP-accepted and inbox-confirmed states.
The following is an illustrative report assembled from the reviewed outage facts, not a screenshot or a claim that this exact email was sent:
Subject: Services restored after room-power interruption; recovery defect identified
Operator-reported room power loss. Exact electrical-outage duration: unknown.
Ethernet obtained an address 9 min 42 s after host boot monotonic zero.
Wi-Fi obtained an address; external VPN access was not tested by this capture.
USB heartbeat: 267 consecutive acknowledgments in the observation window.
Science: startup validation passed; 2,900 additional SCORE tiles committed.
Automatic Ethernet recovery: blocked by stale source bindings on 14 attempts.
Follow-up: O3 later repaired the bindings and passed readiness checks.
A failed-check/restored-check interval is a measured observation interval, not necessarily the exact duration of Internet loss. A power-off cause should be labeled operator-reported or inferred unless independently measured. Avoid alarming precision that the instrumentation cannot support.
11. Commissioning and troubleshooting
11.1 Acceptance sequence
| Phase | Required evidence | Stop condition |
|---|---|---|
| Offline | Integrity checks; meaningful policy/adapter tests; varied ordinary-user permission masks | Any failure, missing dependency or code/config mismatch |
| Disarmed bench | Physical load identity, firmware/readback, exact USB exchange, exclusive ownership | Unexpected switching, ambiguous ACK or management state |
| Read-only host health | Primary network, backend, current science, complete recovery-readiness check | HOLD, open failed/uncertain incident, corrupt/stale evidence |
| Protected maintenance | Explicit device/host state; known heartbeat provider; recovery dispatch held appropriately | Unable to verify device state or resolve a pending action |
| Controlled arming | Accepted settings, legitimate boot bound, planned physical test, observer ready | Incomplete saved/effective configuration or workload gate |
| Deliberate disruption | An explicitly identified lamp/host test with physical observation | Unplanned load, unbounded retry, incomplete scientific readiness |
| Postboot | Fresh boot/invocation evidence, validated numerical continuation or completion, endpoint checks | PID-only success, stale checkpoints, current integrity block |
| Final acceptance | Archived source/config hashes, timing observations, all limits and outstanding tests | Any attempt to call unperformed tests passed |
Appendix B describes the finite watchdog test and the repairs needed before it is executable. Its power-test, acceptance and abort commands are blocked for the unchanged v0.2.0 ZIP. In a corrected release the test must stop heartbeat only after workload/device gates pass; the identified target then loses AC and SSH. No such test was executed for this article. Do not repeat a production outage merely to take a screenshot.
11.2 Troubleshooting decisions
| Symptom | First distinction | Appropriate response |
|---|---|---|
| One ACK missing | USB present? Exact later ACK? Stale/fragmented response? | Bounded retry/reopen; record the incident. Do not declare host lockup. |
| Repeated ACK failure | Exclusive port/management ownership? Valid login available? | Apply the accepted bounded controller-recovery gate; otherwise retain uncertainty and report. |
| iBoot login immediately closes | Another interactive management session active? | Close conflicting session and re-evaluate; do not reset allowances. |
| Network eventually returns after boot | Router/DHCP late versus NIC failure? | Record timed observations and avoid attributing all delay to the host. |
| Website down, NIC healthy | Backend/database functional transaction versus static page? | Use bounded service recovery; no new PCI repair solely for this fault. |
| Science active but progress absent | Current invocation? Expected tile duration? Job completed? | Check the workload contract, completion and runtime evidence. |
| Recovery service stopped with failure | Failed unit result versus intentional idle state? | Block readiness; surface its result and source/receipt cause. |
| Source hash changed after upgrade | All dependent bindings migrated? | Validate the coordinated release; preserve ledgers. Never bypass the check by deleting state. |
| Allowance still spent after 24 h | Verified recovery/cooldown/health predicates satisfied? Host rebooted? | Inspect R24 evidence. Elapsed wall time alone earns no allowance. |
| Email absent | Queue, transport acceptance or inbox filtering? | Inspect each delivery state; do not equate send-function return with receipt. |
The F1 testing lesson is equally practical. Temporary test directories inherited group-write permissions under some user permission masks. Security checks correctly rejected them before installation. The fixtures were corrected without changing the accepted live code, and 125 tests passed under four masks—500 executions. Reproducibility includes the environment that creates test files. [E-R24]
12. Publication, attribution and distribution
Author: Dr. Julio C. Spinelli, as identified on juliospinelli.com. Dataprobe is the manufacturer and source of cited technical information. Neither the support reply nor this note implies Dataprobe authorship or endorsement.
Proposed canonical article address, not yet published: https://juliospinelli.com/iboot-g2plus-ubuntu-scientific-disaster-recovery/.
Distribution permission: Dataprobe/iBoot may share and distribute links to the canonical article on juliospinelli.com, provided Dr. Julio C. Spinelli is acknowledged as the author. This permission concerns distribution through links to the author's website. It does not grant an unrestricted license to republish the article elsewhere. Rights in cited third-party material remain with their respective owners.
This standalone UTF-8 edition embeds its CSS, original diagrams and selected genuine project images. It requires no external font, script, image host or rendering service to read offline. Normal references remain links. The companion source bundle has its own version and checksums; it is not embedded as executable content inside the HTML.
The website currently serves other standalone technical-note HTML files with embedded images. The publisher must still verify the new route's content type, CSP and mobile/print behavior after upload. The existence of a working older article is not proof that a WordPress editor or new route will preserve every embedded asset. The handoff includes the required checks. No publishing or support message has been performed in this branch.
13. Evidence, limitations and references
13.1 Evidence classification
D = vendor documentation. S = dated support clarification. E = inspected source/receipt/log with archive integrity checking where available. H = operator historical report. I = engineering interpretation. U = unresolved or unperformed.
A source checksum establishes byte identity, not truth of every claim inside the file. A unit-test success establishes the exercised software behavior, not a physical relay operation. A configuration image establishes what was shown at capture time, not what remains effective after a later reboot.
| ID | Inspected source | What it supports / boundary |
|---|---|---|
| E-HB | C2 F1 reference iboot-heartbeat-t2.py; T2 and D2 F1 recovery reports |
Exact byte exchange, implementation flags and live communication restoration; no new outlet-cycle proof |
| E-C2 | O1 files/c2-state-selected.json; C2 F1 commissioning source |
Selected final DEPLOYED receipt dated 19 September 15:21:01 UTC; physical timing field not included in selected capture |
| E-O1 | recipegram_power_outage_o1_20260919.tar.gz |
137 payload entries verified; outage startup/progress/network/heartbeat observations; effective outlet action not queried |
| E-O2 | recipegram_recovery_bindings_o2_20260919.tar.gz |
35 payload entries verified; original source pins, installation receipts and preserved ledgers |
| E-O3 | O3 source ZIP and recipegram_recovery_repair_o3_20260919T193426Z_eec0e4f8.tar.gz |
21 report payload entries verified; coordinated source repair, current readiness and process continuity |
| E-R24 | R24 O3 F1 source ZIP and final ...20260919T204257Z_12bb3e13abca.tar.gz |
Three report payload entries verified; enrollment, unchanged allowance and first observer acceptance; real subsequent renewal unperformed |
| E-SCI | stage10c_recovery_r2_20260908(3).zip |
All 20 source manifest members verified; manifest matches O2 installed evidence; Stage10c-specific contracts and future requirements, not a universal ABI |
| H1/H2 | Earlier operator account and accepted branch prompt | Seq327 incident, rearming history and approximately 30 min / 30 s physical commissioning; raw aligned records still requested |
| U-PORT | Companion version 0.2.0 | Parameterized reference with known commissioning and lifecycle defects; 840 recorded offline executions do not establish operational acceptance |
Accepted final R24 archive SHA-256: c8afe6258f01d753ce333b2371ae1369f9adaba09ec8d7f179ec35245b410ba8. The full source matrix supplies the other archive hashes and precise internal locators.
13.2 Remaining acceptance work
The publication handoff groups the essential missing material: current read-only device screens, the complete physical commissioning/rearming timing record if retained, and the final versioned scientific-integration contract when available. Obtain them only through a coordinated brief management session: even read-only management can interact with the communication path. Do not change settings, restart services or interrupt power merely for a picture.
The portable 0.2.0 source includes the listed components, but the new source review establishes that its advertised lifecycle is incomplete. A corrected release must resolve those defects before ordinary-user execution, target installation, real SMTP delivery and physical recovery/renewal/migration validation can establish acceptance. The separate scientific-integration branch contract has not been supplied; the package implements only its own sample-v1 adapter.
References
- Dataprobe, iBoot-G2S / iBoot-G2+ manual, V210312W, March 2021. A project-uploaded copy and the support CLI extract were inspected.
- Dataprobe, iBoot Utility manual V130917E, software 2.10.09, §3.5 for heartbeat tokens. Retrieved 20 September 2026.
- Dataprobe, current iBoot-G2+ product page, model specifications and linked downloads. Retrieved 20 September 2026.
- Dataprobe, iBoot-G2 revision history. Retrieved 20 September 2026; firmware link/notes discrepancy recorded above.
- Dataprobe, iBoot-G2+ Quick Start V210312W. Connection reference; vendor figures are not reproduced here.
- Dr. Julio C. Spinelli, author information. Byline verified 20 September 2026.
- S1: Dataprobe support correspondence, question dated 9 September and reply dated 10 September 2026; MIME decoded and attached CLI PDF inspected. Paraphrased with private headers/signatures excluded.
- Project evidence: Source and archive identifiers in the evidence table and companion claim/source matrix. Private raw archives are retained separately from public downloads.
Revision history
| Revision | Date | Change |
|---|---|---|
| 0.1 — technical review | 20 September 2026 | Original correspondence audit; verified outage/repair/renewal case study; embedded original diagrams and genuine images; disarmed reference companion; explicit publication and production-port gaps. |
| 0.2 — release candidate | 20 September 2026 | Parameterized implementation, 210 tests across four masks, operational and commissioning appendices added; target validation boundaries retained. Later review found the operational gaps documented in revision 0.3. |
| 0.3 — technical and readability review | 20 September 2026 | Documented real-Context timer failure, absent heartbeat-loss drain and lifecycle gaps; corrected postboot resume/evidence ownership, added activation requirements, reader guide, eight new diagrams and GPT-6 Astra Max implementation prompt. Published ZIPs unchanged. |
Appendix A. Installation interfaces and operation requirements
Appendix A step index
- 1. Verify the extracted release
- 2. Create the target inventory
- 3. Inspect the installation plan
- 4. Install without activating recovery
- 5. Enter the management credential privately
- 6. Inspect before authorizing actions
- 7. Authorize the candidate, then establish disarmed hardware
- 8. Initialize the new installation
- 9. Start heartbeat and sample services
- 10. Exercise the sample's controlled continuation
- 11. Physical commissioning and arming
- 12. Routine observations and reports
- 13. Renewal, email and incident reconciliation
- 14. Enter protected maintenance
- 15. Source upgrade transaction
- 16. Roll back a source transaction
- 17. Uninstall only after verified external disarming
These procedures document the portable 0.2.0 reference and its intended operation on a separate target. Source review and offline tests can proceed. Because the known release defects block a complete deployment, treat live installation, arming and automatic operation as work for a corrected release. The historical RecipeGram installation has different accepted source and receipts.
Each command states its privilege, effect, expected result and failure action. Paths such as /opt/iboot-reference/0.2.0/ identify the archived candidate; use the accepted paths from a corrected release’s own runbook. Keep release source immutable and create target configuration outside it. Initial hardware tests need an identified harmless load and separate commissioning host.
What the four recovery actions mean
| Mechanism | Trigger and consequence |
|---|---|
| Hardware outlet watchdog | Missing heartbeats for the configured interval cause the iBoot to cycle Main, subject to its configured cycle limit |
| Ubuntu warm reboot | Host software requests an orderly reboot only after scientific preparation and its governed recovery gates |
| iBoot controller reboot | Management software restarts the controller to recover communication; it is not an outlet-cycle request |
| Permission renewal | Verified successful controller recovery plus the full cooldown and healthy checks can restore one future controller-reset allowance; renewal does not itself reboot anything |
The one-cycle hardware setting and the software reset allowance are different budgets. Never substitute the device command set main cycle for the controller command reboot. Preserve spent records and ambiguous-action evidence; deleting them to obtain another attempt defeats the design.
Staged workflow
- Review source defects, compatibility, wiring, load and host AC-restore behavior.
- Unpack into a fresh directory, verify the manifest and run offline tests. Repair release blockers in a new version before proceeding to live work.
- Discover target values and complete one configuration inventory.
- Plan and install inactive units with live actions disabled; then enter credentials through the installed local procedure.
- Inspect the identified controller and verify effective action None before starting the new heartbeat provider.
- Start the sample while disarmed and prove its preparation, readiness and continuation through a corrected commissioning-health interface.
- Prepare the actual workload and commission the named target. Keep the manager inactive.
- Explicitly arm; persist any consumed-allowance recovery proof before the finite test; observe one planned cycle and automatic boot.
- After boot, verify readiness, resume once and prove current progress/completion. Accept the physical evidence before activating manager and timers.
- Operate only through accepted lifecycle paths. Verify effective disarming before stopping a provider, migrating source or removing services.
Installation, disarmed observations, controller recovery, arming and deliberate outage testing are separate phases. A lamp trial does not commission server checkpointing, and a server restart does not measure the Main outlet's actual OFF duration.
One configuration inventory
Keep target values in the package's configuration file. Do not use broad find-and-replace operations on the source. The wizard and validator must discover or check values before installation.
| Group | Values to record | Validation purpose |
|---|---|---|
| Installation | New installation ID; absolute code, private state, workload, control and credential paths | Prevent cross-installation receipts and unsafe path reuse |
| Account | Existing service user and exact UID/GID | Enforce ownership and the expected worker identity |
| USB | Stable serial identity, permissions, transmit interval, ACK deadline | Keep one writer and distinguish a stale reply from fresh communication |
| Management | Controller address/port, firmware, protected credential path | Recover the controller using its management interface |
| Controller | Frequency, fail count, cycle count, Main, OFF duration, initial state, Disable OFF | Keep timing, power target and restart behavior explicit |
| Network | Primary interface, gateway, optional connection profile, optional Wi-Fi and external probe hosts | Separate local path health from external reachability |
| Application | Unit, health URL, expected HTTP status and optional semantic marker | Avoid treating a running PID as useful application service |
| Science | Adapter, unit, data/control paths, timeouts, progress evidence | Prove safe quiescence and valid continuation |
| Recovery | Confirmation count, infrastructure grace, enabled bounded actions | Avoid immediate power changes when a router starts late |
| Server/TLS mode, account, local credential, sender/recipients | Preserve private credentials and queue delayed notifications |
Optional Wi-Fi and WireGuard are not prerequisites. A local Wi-Fi address and route do not establish external VPN access. The iBoot's management address is not the public website address unless the actual network has explicitly arranged that relation.
Hardware preparation
Use external standard power, USB and Ethernet interfaces. Identify physically which device is connected to Main. Keep the modem, router and switch on their intended independently understood supply path; do not assume a host outlet cycle also resets network infrastructure. Record the host BIOS/UEFI AC-restore setting and test it during commissioning.
Manufacturer documentation lists 12 A at 105–125 VAC and 10 A at 210–240 VAC for the iBoot-G2+. The actual controller, cables, outlets and load all need suitable ratings. This reference does not control internal ATX wiring. With no UPS, it can recover from an outage but cannot checkpoint during a complete absence of power.
Read the device's current firmware and configuration. Historical values from RecipeGram are evidence for that installation, not the new target's saved/effective settings. The nominal reference timeout is 60 seconds × 30 missed intervals, with one Main cycle and a 30-second OFF duration; physical timing and restart margin still require measurement on the target.
The controller may begin heartbeat timing when it boots. Do not assume it waits for the first host heartbeat or has an independent boot-grace setting. action none is a maintenance setting with no promised automatic restoration timer. Record which other automations could switch Main, including AutoPing, schedules, linked expansion behavior and manual control.
Failure handling and evidence
If an operation returns a blocked or uncertain result, preserve its printed result and report. Do not rerun a mutating command, delete a lock/budget, erase a HOLD/fence or refund an allowance to force progress. A follow-up status/report is observational; a repair or retry must follow the documented transition for the specific state.
The manager treats a working local network path, application health, scientific integrity and fresh progress as separate observations. Router/DHCP unavailability can warrant an infrastructure wait. Source mismatches and failed recovery readiness must remain visible even when services currently appear healthy.
Email distinguishes detection, blocked recovery and verified recovery. Queue creation is not delivery; SMTP acceptance is not inbox confirmation. Report measured observation intervals and boot-to-network timing using their precise definitions. A powered-off host cannot by itself establish the exact duration or electrical cause of its outage.
Timing and budget rules
The heartbeat sends case-sensitive protocol bytes and requires the expected ACK according to the implementation's bounded parser. One missed ACK does not by itself establish a periodic firmware defect or failed science. Reopening serial, managing controller sessions and any permitted controller reset must preserve exclusive access and evidence.
Management sessions can interfere with automated recovery. Close interactive sessions cleanly before automation is allowed to recover the controller. Telnet is plaintext and belongs on a restricted management path; credentials are read from protected local files or systemd credentials and are never embedded in article commands, reports or source archives.
Host warm reboot and iBoot controller reboot are permissioned separately. An ambiguous reboot attempt remains spent until the actual implementation supplies sufficient success evidence. Renewal does not erase spent history, issue a daily reset or extend the hardware outlet timeout by 24 hours.
Maintenance, upgrade and uninstall principles
Maintenance first prevents automated dispatch and establishes the documented safe controller condition while preserving evidence. Only then stop or replace providers. The software's receipt of an ACK establishes communication; it does not by itself prove the configured power action, physical wiring or retained settings.
Upgrades are source-pin transactions. Record the old manifest/configuration/state, enter the installation fence, validate new bytes and unit definitions, publish coherently, require the post-install checks and preserve incident/reset history. Rollback must restore a coherent version without inventing allowances or erasing uncertain actions. A release number alone does not authorize source-pin migration.
Uninstalling a heartbeat daemon while its external watchdog remains armed can cause a later power interruption. Verified controller disarming is an uninstall prerequisite. If the controller is unreachable or readback is ambiguous, removal must block and require an independently safe physical arrangement; do not bypass that block by deleting service files.
Target prerequisites
The numbered procedures below describe the archived candidate. Corrected-release paths and validated operational gates take precedence over these version-specific examples.
Use an Ubuntu installation that actually runs systemd, not a container or subsystem assumed to control the host's power and services. Install the serial/runtime and observation dependencies on the commissioning host. NetworkManager connection recovery is optional and requires an already configured NetworkManager connection; this guide does not replace the host's networking stack automatically.
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root for package management; ordinary privilege for version checks. Effect: refresh Ubuntu package indexes and install the named prerequisites; no iBoot setting or power action. Expected: package installation succeeds and versions print. Failure: resolve the host's package/network problem before proceeding; do not disable package verification or controller safety gates.
(
set -eu
sudo apt-get update
sudo apt-get install --no-install-recommends \
python3 python3-serial curl iproute2 iputils-ping unzip coreutils
python3 --version
systemctl --version
python3 -I -B -c 'import serial; print("PYSERIAL_VERSION=" + serial.__version__)'
test -d /run/systemd/system
)
IBOOT_PREREQUISITE_STATUS=$?
printf 'IBOOT_PREREQUISITE_STATUS=%s\n' "$IBOOT_PREREQUISITE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Record the precise Ubuntu/kernel/systemd/Python/pyserial versions in the commissioning report. Presence of a binary or package version alone is not a tested-platform claim. Choose the supported firmware profile only after a genuine device capture confirms it.
1. Verify the extracted release
Host/shell: commissioning Ubuntu host, Bash, in the extracted release directory. Privilege: ordinary user. Effect: validate package bytes and run offline tests; no service, USB, controller or power action. Expected: verification and the offline suite return status 0, with their result records. Failure: stop, preserve output and obtain an intact matching release; do not install around a failed test.
(
set -eu
test -f install.py
test -f configure.py
test -f tests/run_tests.py
python3 -I -B install.py --verify
python3 -I -B tests/run_tests.py
)
IBOOT_OFFLINE_STATUS=$?
printf 'IBOOT_OFFLINE_STATUS=%s\n' "$IBOOT_OFFLINE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Record the actual runtime and permission-mask results produced by this version. Do not replace ordinary-user verification with a root-run suite and assume they are equivalent. A permissions failure must be corrected in the package/test setup, not by weakening private-state checks.
2. Create the target inventory
Host/shell: commissioning Ubuntu host, Bash, same extracted directory. Privilege: ordinary user. Effect: interactive discovery/configuration; writes a new target configuration with live recovery disabled. Expected: ../iboot-reference-target-config.json is created outside the release directory and the command returns status 0. Failure: preserve output, correct the stated input or missing prerequisite, and use a new configuration path if the prior file was partially created. Do not copy a production credential into this JSON.
(
set -eu
test ! -e ../iboot-reference-target-config.json
python3 -I -B configure.py --output ../iboot-reference-target-config.json
)
IBOOT_CONFIGURE_STATUS=$?
printf 'IBOOT_CONFIGURE_STATUS=%s\n' "$IBOOT_CONFIGURE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Use an existing non-root account for the scientific workload. The wizard's USB path must identify the controller on this host, not a renamed example from the article. Specify an actual primary interface and, if enabled, the exact NetworkManager profile and application service. The optional semantic marker is text the intended endpoint must return, not merely any HTTP 200 page.
3. Inspect the installation plan
Host/shell: commissioning Ubuntu host, Bash, extracted directory. Privilege: ordinary user unless the planner reports a specific read-permission requirement. Effect: inspect the proposed files, units, paths and ownership without applying them. Expected: a valid plan and status 0; configuration remains disarmed. Failure: correct the configuration or missing capabilities and plan again. No failed planning gate may be skipped.
(
set -eu
python3 -I -B install.py --plan --config ../iboot-reference-target-config.json
)
IBOOT_PLAN_STATUS=$?
printf 'IBOOT_PLAN_STATUS=%s\n' "$IBOOT_PLAN_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
4. Install without activating recovery
Host/shell: commissioning Ubuntu host, Bash, extracted directory. Privilege: sudo/root for the installer. Effect: create the configured code/state/configuration and unit files, with live actions disabled; this is not arming or service activation. Expected: installer returns status 0 and records its receipt; no power action. Failure: preserve the report and installation-fence state. Do not repeatedly apply or delete the fence; use the diagnosed interrupted-install procedure.
(
set -eu
python3 -I -B install.py --verify
sudo python3 -I -B install.py --apply --config ../iboot-reference-target-config.json
)
IBOOT_INSTALL_STATUS=$?
printf 'IBOOT_INSTALL_STATUS=%s\n' "$IBOOT_INSTALL_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
5. Enter the management credential privately
The following installed commands use the wizard's documented default code directory /opt/iboot-reference/0.2.0. If the inventory deliberately uses another code directory, use the installed cli.py at that exact recorded directory. The installed configuration remains /etc/iboot-reference/config.json. Credential values belong only in the private prompt; they must not appear in command arguments, shell history, screenshots or this configuration file.
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: prompt twice without echo and save the iBoot password to the configured protected credential file; no device action. Expected: matching input, protected file creation and status 0; no password is printed. Failure: inspect the stated local path/mode or mismatch error; do not weaken permissions or put the password into source code.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json credential-set --kind iboot
)
IBOOT_CREDENTIAL_STATUS=$?
printf 'IBOOT_CREDENTIAL_STATUS=%s\n' "$IBOOT_CREDENTIAL_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
If authenticated SMTP was enabled in the inventory, create its distinct password using the same local procedure.
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: save only the configured SMTP credential; sends no email. Expected: status 0 with no password output. Failure: leave mail disabled until the protected configuration is corrected.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json credential-set --kind smtp
)
IBOOT_SMTP_CREDENTIAL_STATUS=$?
printf 'IBOOT_SMTP_CREDENTIAL_STATUS=%s\n' "$IBOOT_SMTP_CREDENTIAL_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
No .env file is used or inspected. The installed services receive their needed protected credentials through the deployed mechanism; do not add a password to Environment= in a unit.
6. Inspect before authorizing actions
Complete the disarmed device-preparation procedure in DEVICE_COMMISSIONING.md on the identified safe target. This reference validates a deliberately narrow controller profile: firmware 1.62.117, USB source, 60-second frequency, 30 failures, Main, 30-second cycle time, one cycle, initial ON, Disable OFF enabled and independent switching disabled. The generic installer does not silently replace arbitrary existing controller settings. In particular, device-disarm changes heartbeat action to None; it does not configure all other fields required by the profile.
Close other management sessions, including an interactive Telnet login. Verify physically that Main's connected load is the intended lamp or separate commissioning target. The explicit exclusive-management flag is an operator statement that concurrent management access has been cleared; it is not a mechanism that can evict unknown outside clients.
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: authenticate and read controller configuration; no requested reset, outlet cycle or settings change. Expected: a structured settings readback; firmware identity still requires the genuine operator-reviewed capture because the queried CLI fields do not establish it. Failure: keep recovery inactive, preserve the error and resolve address, access, firmware or session ownership before proceeding.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json device-inspect \
--exclusive-management
)
IBOOT_INSPECT_STATUS=$?
printf 'IBOOT_INSPECT_STATUS=%s\n' "$IBOOT_INSPECT_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Readback must be reconciled with the device's actual web screens: identity/firmware; heartbeat source/frequency/fail count/action/outlet/cycles; Main cycle time and initial state; Disable OFF; and every other active switching mechanism. A genuine screen is evidence only for its capture date/state. Do not describe an earlier test screen as the current final settings.
7. Authorize the candidate, then establish disarmed hardware
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: explicitly opt this installation into its guarded live operations, then disable the controller's heartbeat power action and verify readback. Authorizing alone performs no hardware action; disarming changes a controller setting. Expected: successful authorization, verified disarmed receipt and status 0. Failure: stop; assume the prior external-watchdog state may remain in effect. Keep any existing heartbeat provider running until the controller state is established.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json authorize-live --confirm
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json device-disarm \
--exclusive-management
)
IBOOT_DISARM_STATUS=$?
printf 'IBOOT_DISARM_STATUS=%s\n' "$IBOOT_DISARM_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Disarmed heartbeat action does not prove that another AutoPing detector, schedule, expansion link or outside operator cannot switch the outlet. Resolve those independent actors in the commissioning inventory.
8. Initialize the new installation
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: initialize the harmless sample and the new installation's initial controller allowance under the disarmed gate. Expected: fresh initialization and status 0, with one initial allowance as identified by the result. Failure: preserve existing records. This command is not a way to replenish a spent allowance and must not be forced by deleting state.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json initialize
)
IBOOT_INITIALIZE_STATUS=$?
printf 'IBOOT_INITIALIZE_STATUS=%s\n' "$IBOOT_INITIALIZE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
This new-installation initialization is different from the historical R24 upgrade, which preserved an existing allowance and added none at enrollment. Never transplant the historical deployment's initial/spent records into this installation.
9. Start heartbeat and sample services
Start only after verified disarming, source checks and initialization. The service names below are the configured default names; use the exact inventory if intentionally changed.
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: enable/start heartbeat and the harmless workload, with the controller disarmed. Expected: both units active, fresh ACKs and advancing valid sample work; status command returns structured observations. Failure: preserve the journal/report. Confirm controller disarming before stopping or changing its heartbeat provider; do not start the manager to override failed prerequisites.
(
set -eu
sudo systemctl daemon-reload
sudo systemctl enable --now \
iboot-reference-heartbeat.service \
iboot-reference-workload.service
sudo systemctl status --no-pager \
iboot-reference-heartbeat.service \
iboot-reference-workload.service
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json status
)
IBOOT_START_STATUS=$?
printf 'IBOOT_START_STATUS=%s\n' "$IBOOT_START_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
The continuously running heartbeat requires an exact fresh-identity ACK sequence. A daemon's active state alone is insufficient. Starting the sample is not yet proof of its pause/resume or power-loss continuity.
Health interpretation: the CLI status command reports stored records; it does not independently prove current scientific progress. Ordinary CLI health uses purpose=observe and requires an active manager, which must remain inactive before physical acceptance. A corrected release must expose and test a commissioning-health observation using the already implemented commission-purpose policy. Do not start the manager just to obtain a green health result.
10. Exercise the sample's controlled continuation
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root for the adapter commands. Effect: ask the sample to prepare, validate its ready state, then resume; no host reboot or outlet cycle. Expected: preparation and ready proof pass, resume succeeds and subsequent work advances from valid state. Failure: leave the sample contained at its reported phase, preserve evidence and repair the adapter/integrity failure before arming.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json prepare
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json ready
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json resume
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json status
)
IBOOT_SAMPLE_CONTINUITY_STATUS=$?
printf 'IBOOT_SAMPLE_CONTINUITY_STATUS=%s\n' "$IBOOT_SAMPLE_CONTINUITY_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Require fresh progress after resume. Completed-and-quiescent work must be treated separately from a stalled unfinished sample. A future Stage10d/StageXX adapter must satisfy the released scientific contract and its own numerical/integrity tests; renaming a sample success file is not integration.
resume returning exit status zero means the start request was accepted. Its structured result is still STARTED_GATE_PENDING, healthy:false. Verify fresh numerical progress or validated completion separately; do not call a repeated resume to obtain a more favorable result.
11. Physical commissioning and arming
Blocked for the published v0.2.0 ZIP. The physical-test and abort paths cannot pass their real timer query. Appendix B §§4–6 documents the intended order and contains guarded historical command examples. Use them to review a corrected release, not to arm the unchanged package. Arming is separate from physical acceptance, and the manager must remain inactive until acceptance.
12. Routine observations and reports
Host/shell: commissioned Ubuntu host, Bash. Privilege: sudo/root. Effect: observe status/health and write a diagnostic report; no requested reboot, relay action or allowance reset. Expected: structured results and a report location; a health failure remains a meaningful finding even if services are running. Failure: preserve partial output/report. Do not turn a failed or unknown health observation into success because report creation completed.
(
set -u
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json status
IBOOT_STATUS_CHECK=$?
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json health
IBOOT_HEALTH_CHECK=$?
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json report
IBOOT_REPORT_CHECK=$?
printf 'IBOOT_STATUS_CHECK=%s\n' "$IBOOT_STATUS_CHECK"
printf 'IBOOT_HEALTH_CHECK=%s\n' "$IBOOT_HEALTH_CHECK"
printf 'IBOOT_REPORT_CHECK=%s\n' "$IBOOT_REPORT_CHECK"
)
IBOOT_OBSERVATION_STATUS=$?
printf 'IBOOT_OBSERVATION_SHELL_STATUS=%s\n' "$IBOOT_OBSERVATION_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
The three inner statuses show whether their command completed; inspect each JSON result and its health/reasons as well. The CLI can successfully produce a failed-health observation. The outer shell merely confirms the observation script returned. Share a report only after checking that its public-export policy excludes secrets and unnecessary infrastructure identifiers.
Host/shell: commissioned Ubuntu host, Bash. Privilege: sudo/root. Effect: read recent service logs and timer schedules; no changes. Expected: bounded log output and configured timers. Failure: retain the missing-unit/permission error and check the installation receipt; do not create guessed unit names.
(
set -u
sudo journalctl --no-pager -n 120 \
-u iboot-reference-heartbeat.service \
-u iboot-reference-manager.service \
-u iboot-reference-renew.service \
-u iboot-reference-mail.service
sudo systemctl list-timers --all --no-pager \
iboot-reference-renew.timer iboot-reference-mail.timer
)
IBOOT_LOG_STATUS=$?
printf 'IBOOT_LOG_STATUS=%s\n' "$IBOOT_LOG_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
13. Renewal, email and incident reconciliation
Proof before disruption: if arming or communication recovery consumed an allowance, run the renewal observer after four valid regular ACKs while the same heartbeat invocation and host boot remain alive. Check that trusted success proof and its timing anchor were actually persisted. Do this before a deliberate heartbeat stop or host reboot. A missing proof cannot later be reconstructed by editing timestamps or substituting another invocation. A host reboot after proof capture starts a fresh full cooldown.
Ordinary scheduling belongs to the installed timers. The following manual observation is useful after reviewing a recovery report.
Host/shell: commissioned Ubuntu host, Bash. Privilege: sudo/root. Effect: evaluate the renewal observer now; it may update cooldown/evidence or restore one permitted allowance when all gates pass. It never sends a controller reset or outlet cycle. Expected: an explicit allowance-available, cooldown, renewed-one-allowance or blocked result. Failure: preserve the reason and spent history; do not initialize again or change timestamps.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json renew
)
IBOOT_RENEWAL_STATUS=$?
printf 'IBOOT_RENEWAL_STATUS=%s\n' "$IBOOT_RENEWAL_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Host/shell: commissioned Ubuntu host, Bash. Privilege: sudo/root. Effect: attempt delivery of already queued messages using configured recipients and TLS; this block sends email. Expected: delivery-attempt/SMTP-acceptance or retry information without secrets. Failure: retain the queued records and correct network/TLS/account configuration; do not discard evidence merely to suppress alerts.
(
set -eu
sudo systemctl start iboot-reference-mail.service
sudo journalctl --no-pager -n 60 -u iboot-reference-mail.service
)
IBOOT_MAIL_STATUS=$?
printf 'IBOOT_MAIL_STATUS=%s\n' "$IBOOT_MAIL_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Host/shell: commissioned Ubuntu host, Bash. Privilege: sudo/root. Effect: reobserve a pending incident and close it only if the implementation's evidence gates pass; does not refund action budgets or force an attempted action to be successful. Expected: a reconciled result or a preserved blocked incident. Failure: keep the incident/report for diagnosis.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json reconcile
)
IBOOT_RECONCILE_STATUS=$?
printf 'IBOOT_RECONCILE_STATUS=%s\n' "$IBOOT_RECONCILE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
14. Enter protected maintenance
Close other controller management sessions. Maintenance must return verified action None before providers can be stopped. It has no automatic rearm timer.
Host/shell: commissioned Ubuntu host, Bash. Privilege: sudo/root. Effect: disarm and independently verify the controller, enter MAINTENANCE, then stop automatic manager service and renewal/mail timers dispatch while leaving the heartbeat and workload running. Expected: verified maintenance with automatic_rearm:false; service stops occur only after that succeeds. Failure: keep the heartbeat provider running, preserve the blocked state and resolve controller uncertainty before proceeding.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json maintenance \
--exclusive-management
sudo systemctl stop \
iboot-reference-manager.service \
iboot-reference-renew.timer \
iboot-reference-mail.timer
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json status
)
IBOOT_MAINTENANCE_STATUS=$?
printf 'IBOOT_MAINTENANCE_STATUS=%s\n' "$IBOOT_MAINTENANCE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Rearming limitation in v0.2.0: arming always requires an unused controller-reset allowance, while renewal requires ARMED mode. Entering MAINTENANCE with a spent allowance can therefore leave no supported route back to ARMED. Continue to disarm when safety requires it; do not preserve arming merely to avoid this defect. Stay verifiably disarmed until a reviewed lifecycle correction is available. Waiting 24 hours in maintenance, editing deployment.json or deleting spent records does not solve the problem.
15. Source upgrade transaction
The expandable command examples in this section deliberately exit before any action. They preserve the old interface for review. Use a tested runbook from a corrected release; deleting the guard does not repair the implementation.
This procedure applies to a different, reviewed package version after its offline tests pass. Execute the upgrade command from that new package's extracted directory. The installer preserves installation identity, workload/state/credential paths, incidents and budgets; it derives the new versioned code directory. A same-version/same-directory upgrade is refused. Do not generate a new installation ID to evade this check.
First enter verified maintenance using the preceding block. Then make the sample quiescent and stop all owned units before touching source bindings.
Host/shell: commissioned Ubuntu host, Bash. Privilege: sudo/root. Effect: seal/verify the sample, then stop its service, heartbeat and owned manager/observer/mail services while the external watchdog is verifiably disarmed. Expected: ready proof and stopped units. Failure: preserve the seal/fence and actual service state; never stop heartbeat after an unverified maintenance result.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json prepare
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json ready
sudo systemctl stop \
iboot-reference-manager.service \
iboot-reference-renew.timer iboot-reference-renew.service \
iboot-reference-mail.timer iboot-reference-mail.service \
iboot-reference-workload.service \
iboot-reference-heartbeat.service
)
IBOOT_UPGRADE_STOP_STATUS=$?
printf 'IBOOT_UPGRADE_STOP_STATUS=%s\n' "$IBOOT_UPGRADE_STOP_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Host/shell: same Ubuntu host, Bash, now in the extracted new release directory. Privilege: ordinary user for offline verification; sudo/root for migration. Effect: validate new source, create an immutable rollback record, fence publication and migrate the source/configuration/unit bindings while preserving persistent history. Expected: coherent new installed version and passing migration receipt. Failure: preserve the transaction report and fence; do not activate mixed units/sources, recopy stale configuration or delete spent records.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
python3 -I -B install.py --verify
python3 -I -B tests/run_tests.py
sudo python3 -I -B install.py --upgrade \
--config /etc/iboot-reference/config.json
)
IBOOT_UPGRADE_STATUS=$?
printf 'IBOOT_UPGRADE_STATUS=%s\n' "$IBOOT_UPGRADE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
A corrected migration runbook must use the returned code directory, preserve numerical continuity and all budgets, and create a new versioned physical acceptance linked to the previous records. The v0.2.0 commands cannot provide that reacceptance. Do not restart automatic operation or delete old test files to get past the gate. A scientific adapter migration also needs its own schema and integrity checks; changing a package version does not authorize discarding an incompatible seal.
16. Roll back a source transaction
The expandable command examples in this section deliberately exit before any action. They preserve the old interface for review. Use a tested runbook from a corrected release; deleting the guard does not repair the implementation.
Rollback also needs the corrected versioned acceptance lifecycle before automatic operation can resume. A restored code receipt alone is not permission to reactivate recovery.
Rollback is a versioned installer action; it is not recursive copying over live code. Use the current/new package's installer with the installed configuration, after verified maintenance and stopped owned units. The immutable previous configuration/units and source receipts must be present. Existing allowance consumption and ambiguous-action history must survive rollback.
Host/shell: same Ubuntu host, Bash, extracted current/new package directory. Privilege: sudo/root. Effect: restore the previous coherently pinned code/configuration/unit version from its installation backup; preserve state/budgets. Expected: a verified rollback receipt, with services awaiting deliberate activation and renewed commissioning. Failure: leave the installation fenced/disarmed and retain both versions and reports for diagnosis; do not roll back by erasing state.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
sudo python3 -I -B install.py --rollback \
--config /etc/iboot-reference/config.json
)
IBOOT_ROLLBACK_STATUS=$?
printf 'IBOOT_ROLLBACK_STATUS=%s\n' "$IBOOT_ROLLBACK_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
17. Uninstall only after verified external disarming
Run from the installed version's matching extracted package directory. First enter protected maintenance and verify the recorded external-controller state. The removal gate additionally requires fresh verified disarming; an unreachable controller is a blocking condition. Preserve scientific data, incident/reset history, captures and installation receipts separately.
Host/shell: commissioned Ubuntu host, Bash, matching extracted package directory. Privilege: sudo/root. Effect: obtain fresh independent action-None readback, prepare/verify the sample, stop owned units and invoke the gated uninstall. Expected: unit removal succeeds only with current disarm proof, prior verified other-automation inventory and stopped units; preserved evidence is identified. Failure: keep the heartbeat provider available until external power enforcement is confirmed safe; do not delete units manually to bypass the check. The disarm proof must remain less than 600 seconds old; if preparation exceeds that window, obtain another readback before retrying the observational gate.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json device-disarm \
--exclusive-management
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json prepare
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json ready
sudo systemctl stop \
iboot-reference-manager.service \
iboot-reference-renew.timer iboot-reference-renew.service \
iboot-reference-mail.timer iboot-reference-mail.service \
iboot-reference-workload.service \
iboot-reference-heartbeat.service
sudo python3 -I -B install.py --uninstall \
--config /etc/iboot-reference/config.json
)
IBOOT_UNINSTALL_STATUS=$?
printf 'IBOOT_UNINSTALL_STATUS=%s\n' "$IBOOT_UNINSTALL_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
No uninstall command here deletes historical budgets or scientific outputs. The installer removes only its verified owned unit files and retains source, state, configuration, scientific outputs and credentials for separate archival/removal. A powered-off or disconnected iBoot is not itself proof that a future reconnection will be disarmed; retain the documented safe physical arrangement until its effective settings are verified.
Troubleshooting decisions
| Observation | Interpretation | Action |
|---|---|---|
| One missed ACK | Communication observation, not proof of a host/science failure | Inspect bounded retry/reopen evidence and current provider identity |
| Manager idle but failed result/source mismatch | Recovery is not ready even if network/site is healthy | Preserve incident and source receipts; use reviewed migration/repair |
| Delayed router/DHCP | Infrastructure may not yet be available | Inspect configured boot grace and local path evidence; avoid inventing an electrical duration |
| Device readback or prompt ambiguous | Effective configuration unverified | Keep provider running; close outside sessions and collect bounded evidence |
| Controller allowance spent without recovery proof | Attempt is not eligible for automatic replenishment | Preserve history; do not reset the budget manually |
| R24 cooldown after a host reboot | Prior host monotonic interval cannot establish the new continuous cooldown | Let the observer establish its current-boot evidence and full cooldown |
| Wi-Fi address exists but remote VPN fails | Local interface observation does not prove external access | Diagnose the optional route/tunnel from an independent endpoint |
| SMTP accepted, no inbox message | Transport acceptance is not final inbox receipt | Inspect recipient mailbox/filtering separately; avoid duplicate sends |
| Completed sample | No new-progress requirement for correctly completed/quiescent work | Require authenticated completion and quiescence, not arbitrary PASS text |
| Uninstall blocked | External power safety or provenance not established | Preserve service and evidence; resolve actual controller state |
Appendix B. Device commissioning and finite physical test
The intended test deliberately interrupts Main power on a separately identified target, with an operator present. Genuine target evidence must replace historical article screenshots. No physical action was executed while preparing this edition. A corrected runbook must retain explicit stop/abort handling and distinguish operator timing from machine-observed reboot and workload evidence.
The controller, heartbeat provider, workload and recovery manager have separate acceptance gates. PASS_ARMED_CONFIGURATION means effective CLI settings and new acknowledgements passed; it does not mean a relay operation was observed. The final physical receipt expressly attributes timing to the operator and combines it with a changed host boot, fresh USB acknowledgements and postboot workload validation.
1. Prepare the actual device while Main carries a harmless load
Before connecting a host whose AC may be interrupted, use a suitably rated lamp or comparable harmless load. Identify Main physically. Verify actual device model and firmware 1.62.117, voltage/current ratings, protected management connectivity and the host's later BIOS/UEFI AC-restore behavior.
In the local device web interface, inspect and save the values below. Screen labels can differ; use the actual firmware UI and official manual. Do not invent a missing setting or assume a displayed default was saved.
| Setting | Required value | Reason and evidence |
|---|---|---|
| Firmware | 1.62.117 | This candidate deliberately recognizes only this reviewed banner |
| Heartbeat source | USB | Host token exchange over serial |
| Frequency | 60 seconds | Controller interval; sender uses20 seconds |
| Fail Count | 30 | Nominal1800-second timeout; measure actual behavior |
| Heartbeat action during setup | None | Prevent setup from arming Main |
| Heartbeat outlet | Main | The physically identified load |
| Heartbeat maximum cycles | 1 | Limit continuous-loss episode; a subsequent heartbeat can rearm it |
| Main cycle time | 30 seconds | Actual OFF duration is measured separately |
| Main initial state | On | Restore the load after controller/utility restart |
| Disable OFF | Yes | Prevent several persistent-OFF paths; verify timed-cycle compatibility |
| Main Graceful Shutdown | Disabled | This implementation performs workload preparation on Ubuntu |
| AutoPing 1, 2, 3 actions | None | Avoid competing ping-triggered power actions |
| Scheduled power events | Disabled | Avoid an independent scheduled interruption |
| Other power automation | Disabled or positively inapplicable | Check expansion links, I/O rules and cloud/external automation against the actual setup |
If the controller says a restart is required, complete that activation while the harmless load is attached, then independently reopen the settings and verify the effective values. A controller restart is distinct from a Main outlet cycle. Do not perform a manual outlet cycle or substitute set main cycle for controller reboot. This procedure supplies no guessed schedule/GSD query command; the local UI is the evidence for fields the verified CLI cannot expose.
Create genuine local screenshots for firmware, heartbeat maximum cycles, disabled Main graceful shutdown, disabled schedules and other automation. Photograph the actual Main load connection. Capture no passwords, login input, browser session tokens or unrelated infrastructure identifiers. Close every management browser/Telnet session afterward, including sessions on other computers.
The CLI checks use only get heartbeat, get cycle, get initial state, get disable off, and get autoping. A get heartbeat response does not expose every gate; maximum cycles and other absent fields remain separately labeled operator evidence.
2. Installation, credentials and disarmed readback
Complete installation, protected credential entry and authorize-live as described in INSTALL_OPERATION.md. That authorization changes local software policy; it does not itself arm the device. Controller credentials are read by the shared Telnet transport through protected local/systemd credentials; no command here takes a password argument.
For every management command, --exclusive-management asserts that other sessions are closed. The process takes controller/guardian.lock; the continuous heartbeat process defers controller-reset escalation while that lock is held. The provider stays running during these brief transactions. That differs from the historical stopped-provider commissioning transaction and must be validated on this new target. If management causes missed ACKs, stop commissioning, preserve the report and recover the proven disarmed state before proceeding. Telnet remains plaintext on the controller's LAN hop.
Host/shell: separate Ubuntu commissioning host, Bash. Privilege: sudo/root. Effect: get readback, set heartbeat action None, close that session, then independently get readback again. No provider is stopped. Expected: PASS_EFFECTIVELY_DISARMED. Failure: leave the heartbeat provider running if already active; do not stop/remove it, rerun an uncertain mutation or infer that a reboot-required setting is effective.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json device-inspect \
--exclusive-management
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json device-disarm \
--exclusive-management
)
IBOOT_DISARM_STATUS=$?
printf 'IBOOT_DISARM_STATUS=%s\n' "$IBOOT_DISARM_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
The default code/config paths shown are produced by the default installation. If the accepted inventory intentionally selects different paths, use those exact installed paths consistently. The configuration gates also require the reviewed USB 60/30/Main profile,30-second OFF interval, Initial On, Disable Off Yes and all AutoPing actions None. A fresh install's software label cannot establish physical disarming; the independent query is required.
3. Build the genuine UI evidence inventory
Stage genuine evidence with the correct owner. Preserve your original captures. Have the administrator copy only the specifically reviewed, nonsecret PNG/JPEG files into a newly created root-owned private directory, with directory mode 0700 and file mode 0600. Check each source is a regular, single-link file rather than a symlink; check its size and compare its SHA-256 before and after copying. Use those absolute root-owned copy paths in the inventory. Apply the same process to the actual timing and postboot records in §6. Do not change ownership of the operator’s originals or copy credential directories.
The inventory is an operator attestation bound to actual capture bytes and this installation. Hash checking does not interpret the screenshot or prove the operator's observation. It prevents silently substituting another capture or target after review. Initial inventory captures must be PNG/JPEG files, owned by root under the documented sudo invocation, single-link regular files, not symlinks, not group/world-writable, and within 64 MiB. The JSON is bounded to 256 KiB, unique-key, source/target-bound, no more than 24 hours old, and protected by its record hash. Do not fabricate screen images or reuse a different installation's captures.
Host/shell: commissioning Ubuntu host, Bash. Privilege: sudo/root to read its installed nonsecret configuration and protect the resulting inventory. Effect: interactive prompts create a new JSON inventory from six existing actual capture files; no device or service call. Expected: DEVICE_EVIDENCE_FILE= names a new private JSON file. Failure: correct the capture/permission/input issue; no commissioning authority has been granted by a partial inventory.
(
set -eu
sudo python3 -I -B - <<'PY'
import datetime, hashlib, json, os, pathlib, stat
def canonical(value):
return (json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False, allow_nan=False)+'\n').encode()
def ask(tty, prompt):
tty.write(prompt+' '); tty.flush()
value=tty.readline().strip()
if not value:
raise RuntimeError('Required value missing')
return value
with open('/dev/tty','r+') as tty:
config_path=pathlib.Path(ask(tty,'Full installed config path:'))
cfg=json.loads(config_path.read_text())
target=ask(tty,'Specific physical Main load identity (host or lamp identifier):')
operator=ask(tty,'Name of operator who personally reviewed the six captures:')
for statement in (
'I physically identified this Main load',
'Heartbeat maximum cycles is 1',
'Main Graceful Shutdown is disabled',
'All scheduled power events are disabled',
'Other controller/external power automation is disabled or positively inapplicable',
'All other management sessions are closed'):
if ask(tty,statement+'; enter YES only if verified:') != 'YES':
raise RuntimeError('Observation not verified; no inventory created')
captures=[]
for role in ('firmware','heartbeat_cycle_count','graceful_shutdown','schedule','other_automation','physical_main_load'):
path=pathlib.Path(ask(tty,'Absolute actual PNG/JPEG capture path for '+role+':'))
if not path.is_absolute() or path.name=='.env' or path.name.startswith('.env.'):
raise RuntimeError('Invalid capture path')
if pathlib.Path(cfg['paths']['credentials']) in path.parents:
raise RuntimeError('Credential directories are not capture sources')
for parent in (path,)+tuple(path.parents):
if parent.is_symlink():
raise RuntimeError('Symlink captures are refused')
info=path.stat()
if not stat.S_ISREG(info.st_mode) or info.st_nlink!=1 or info.st_mode&0o022 or not 16<=info.st_size<=64*1024*1024:
raise RuntimeError('Unsafe capture metadata')
if path.lstat().st_uid!=0:raise RuntimeError('Root-owned staging copy required under sudo')
raw=path.read_bytes()
media='image/png' if raw.startswith(b'\x89PNG\r\n\x1a\n') else 'image/jpeg' if raw.startswith(b'\xff\xd8\xff') else None
if media is None:
raise RuntimeError('Actual PNG/JPEG capture required')
captures.append({'role':role,'path':str(path),'sha256':hashlib.sha256(raw).hexdigest(),'media_type':media})
doc={'schema':'iboot-device-evidence-v1','installation_id':cfg['installation_id'],
'firmware':'1.62.117','physical_target':target,'captured_by':operator,
'captured_utc':ask(tty,'Actual capture time, ISO8601 with UTC offset:'),
'source':'operator-reviewed-local-device-ui',
'observations':{'heartbeat_max_cycles':1,'main_gsd_disabled':True,'schedules_disabled':True,
'other_power_automation_disabled':True,'physical_main_load_confirmed':True,
'exclusive_management_confirmed':True},'captures':captures}
timestamp=datetime.datetime.fromisoformat(doc['captured_utc'].replace('Z','+00:00'))
if timestamp.tzinfo is None:
raise RuntimeError('A timezone offset is required')
output=pathlib.Path(ask(tty,'New absolute private inventory JSON path:'))
if not output.is_absolute() or output.exists() or output.is_symlink():
raise RuntimeError('New absolute output path required')
doc['record_sha256']=hashlib.sha256(canonical(doc)).hexdigest()
fd=os.open(output,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600)
with os.fdopen(fd,'wb') as stream:
stream.write(canonical(doc));stream.flush();os.fsync(stream.fileno())
print('DEVICE_EVIDENCE_FILE='+str(output))
PY
)
IBOOT_EVIDENCE_STATUS=$?
printf 'IBOOT_EVIDENCE_STATUS=%s\n' "$IBOOT_EVIDENCE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Run the disarmed heartbeat/sample initialization and lifecycle checks in INSTALL_OPERATION.md. They must establish an unused controller allowance, a current heartbeat, valid scientific work and healthy configured network/application layers. Then prepare the sample and leave it sealed/quiescent for arming. A completed and validated sample can also be ready; a live PID alone is insufficient.
4. Commission and arm the named target
The entire host is the test target. Main power loss and host reboot affect every process and storage writer on that machine. A prepared sample does not protect unrelated work. Use a separate test host, or identify, stop and secure all other workloads with the operator’s authorization before any disruptive test.
The expandable command examples in this section deliberately exit before any action. They preserve the old interface for review. Use a tested runbook from a corrected release; deleting the guard does not repair the implementation.
Before attaching the commissioning host to Main, verify disarmed effective action None and suitable physical wiring/BIOS AC restore. Update the physical-load capture/inventory for the host actually on Main; a lamp photograph cannot authorize a different host. Keep iboot-reference-manager.service inactive through the physical test; the generated units include no manager timer. The corrected inactivity gate must account for that actual unit set.
Host/shell: separate commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: prepare/validate workload, record independently verified disarmed settings, then explicitly arm the external watchdog. Arming may require one budgeted controller reboot, performed by the continuous heartbeat daemon after recording the request; it never substitutes an outlet-cycle command. Expected: PASS_DISARMED_COMMISSIONING, then PASS_ARMED_CONFIGURATION, with physical test still incomplete. Failure: retain receipts and keep provider running; a partially applied arm transaction enters BLOCKED and must not be blindly replayed.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
read -r -p 'Specific Main load identity matching the inventory: ' IBOOT_MAIN_LOAD
read -r -p 'Absolute genuine device evidence JSON path: ' IBOOT_EVIDENCE
test -n "$IBOOT_MAIN_LOAD"
test -f "$IBOOT_EVIDENCE"
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json prepare
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json ready
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json commission \
--evidence "$IBOOT_EVIDENCE" --main-load "$IBOOT_MAIN_LOAD" \
--exclusive-management
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json arm \
--main-load "$IBOOT_MAIN_LOAD" --exclusive-management \
--deliberate-power-test
)
IBOOT_ARM_STATUS=$?
printf 'IBOOT_ARM_STATUS=%s\n' "$IBOOT_ARM_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Arming's verified setters are exactly set gsd main enabled no, set heartbeat cycle 1, set heartbeat action cycle. Each must return one successful acknowledgement and a complete admin prompt. A separate session reads the effective configuration. At least four subsequent regular exact ACKs are required. GSD/cycle count are labeled accepted setters plus operator captures; the software does not invent missing readback fields. If a pending controller reset spends the initial allowance, its spent intent remains intact and is governed by renewal rules; do not delete it to create immediate permission for another reset.
Required addition before the test: if a controller reboot spent the initial allowance during arming, capture and persist its trusted recovery proof after four normal ACKs while the original invocation is still alive. Check the recorded result before proceeding. See Appendix A §13. A corrected release must test this ordering as part of the same commissioning transaction.
5. Deliberate finite watchdog test
The expandable command examples in this section deliberately exit before any action. They preserve the old interface for review. Use a tested runbook from a corrected release; deleting the guard does not repair the implementation.
The following stops the heartbeat provider after verifying scientific readiness, effective arming, source pins, manager inactivity, validated unit handling and boot enablement. It deliberately allows the external Main watchdog to expire. Expect the connected host to lose AC and SSH. The workload is already quiescent. An operator must observe the actual Main load, record heartbeat-stop-to-OFF and actual OFF duration, and ensure no manual power action obscures the result.
Host/shell: commissioning Ubuntu host, Bash over its console/SSH. Privilege: sudo/root. Effect: persist a unique nonreplayable intent and stop heartbeat once; no explicit outlet-cycle command. Expected: WAITING_FOR_ACTUAL_WATCHDOG_EXPIRY, nominal 1800 s expiry and 30 s OFF, followed by automatic host startup. The shell returns without waiting 35 minutes. Failure: keep the intent and report; an uncertain stop is not replay permission. If no actual interruption occurs within 35 minutes, follow the containment paragraph below rather than repeatedly issuing this command.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
read -r -p 'Specific actual Main host identity to interrupt deliberately: ' IBOOT_MAIN_LOAD
test -n "$IBOOT_MAIN_LOAD"
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json power-test \
--main-load "$IBOOT_MAIN_LOAD" --exclusive-management \
--deliberate-power-test
)
IBOOT_POWER_TEST_STATUS=$?
printf 'IBOOT_POWER_TEST_STATUS=%s\n' "$IBOOT_POWER_TEST_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Retain the printed test_id and actual stopped UTC time. Record one Main OFF/ON cycle, OFF duration in seconds and stop-to-OFF duration in seconds. SSH downtime includes boot and network startup, so it is not the OFF measurement. The reference acceptance bands are 25–40 s OFF and 1680–1980 s stop-to-OFF; these are commissioning policy tolerances around the configured values, not claims about firmware precision.
No-interruption containment — required behavior for a corrected release: after 35 minutes without the expected interruption, preserve the test record, restore heartbeat feeding without spending another reset allowance, verify fresh ACKs and then verify effective disarming while an operator remains at the load. Record an aborted test, not a pass. The published v0.2.0 abort-test command fails at its timer gate and cannot be relied on for this containment. That is a reason to stop before arming this release. A failed ACK wait never establishes action None, and no uncertain attempt grants another trial.
Host/shell: same commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: abandon the unaccepted test, preserve its records, and start only the USB heartbeat provider. No controller/outlet reset. Expected: PHYSICAL_TEST_ABORTED_FEED_ONLY and fresh ACKs, with deployment BLOCKED. Failure: preserve the result and keep an operator at the known physical load; a failed ACK wait does not make the controller disarmed.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
read -r -p 'Specific Main host identity of the test being aborted: ' IBOOT_MAIN_LOAD
test -n "$IBOOT_MAIN_LOAD"
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json abort-test \
--main-load "$IBOOT_MAIN_LOAD" --exclusive-management \
--deliberate-power-test
)
IBOOT_ABORT_TEST_STATUS=$?
printf 'IBOOT_ABORT_TEST_STATUS=%s\n' "$IBOOT_ABORT_TEST_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
6. Record observed timing and accept after actual boot
The expandable command examples in this section deliberately exit before any action. They preserve the old interface for review. Use a tested runbook from a corrected release; deleting the guard does not repair the implementation.
After the actual automatic boot, the enabled heartbeat unit should restart. The prepared sample does not resume automatically: its durable root HOLD still prevents new work. Keep the manager inactive. A corrected runbook must perform one fresh ready check, call resume once, then independently prove current progress or validated completion before accepting the test. Do not delete HOLD by hand. The ordinary CLI health is not the commissioning gate; see the required interface correction below.
- Check the changed host boot identity, current heartbeat invocation, exact fresh ACKs and matching test intent.
- Validate the sealed workload with
ready. On any mismatch, preserve HOLD and the evidence; do not resume. - Submit one
resumefor that accepted seal. TreatSTARTED_GATE_PENDINGas pending even when the command exits zero. - Use a reviewed commissioning-health entry point that calls
recovery.health(ctx, purpose='commission'), with source checks and structured output. Require fresh progress or validated completion while the manager remains inactive. This entry point is missing from the published CLI and must be added in the corrected release. - Bind genuine operator timing records and postboot console evidence to the exact test, then seal acceptance. The timing record is an operator account; hashing it does not turn it into an independent relay sensor.
Host/shell: commissioning Ubuntu host after boot, Bash. Privilege: sudo/root. Effect: create a new observation JSON from actual timing records; no hardware/service action. Expected: new PHYSICAL_EVIDENCE_FILE. Failure: keep the pending test and correct evidence omissions; never fill timing fields with planned values merely to pass.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
sudo python3 -I -B - <<'PY'
import datetime, hashlib, json, os, pathlib, stat
def canonical(obj):
return (json.dumps(obj,sort_keys=True,separators=(',',':'),ensure_ascii=False,allow_nan=False)+'\n').encode()
def ask(tty,prompt):
tty.write(prompt+' ');tty.flush()
value=tty.readline().strip()
if not value:raise RuntimeError('Required value missing')
return value
with open('/dev/tty','r+') as tty:
cfg=json.loads(pathlib.Path(ask(tty,'Full installed config path:')).read_text())
pending=json.loads((pathlib.Path(cfg['paths']['state'])/'device/power-test-pending.json').read_text())
if pending['phase']!='WAITING_FOR_PHYSICAL_EXPIRY':raise RuntimeError('No eligible pending test')
print('TEST_ID='+pending['test_id'])
print('PHYSICAL_TARGET='+pending['physical_target'])
for text in ('Exactly one Main OFF/ON cycle was physically observed',
'The host started automatically without manual power action',
'No manual reboot or power command obscured the test'):
if ask(tty,text+'; enter YES only if actually observed:')!='YES':raise RuntimeError('Acceptance not supported')
observations={'one_main_off_on_cycle':True,'host_started_automatically':True,'no_manual_power_action':True,
'off_seconds':float(ask(tty,'Measured Main OFF duration in seconds:')),
'heartbeat_stop_to_off_seconds':float(ask(tty,'Measured heartbeat-stop-to-Main-OFF interval in seconds:'))}
captures=[]
for role in ('physical_observation_record','postboot_console'):
path=pathlib.Path(ask(tty,'Absolute actual record path for '+role+':'))
if not path.is_absolute() or path.is_symlink() or path.name=='.env' or path.name.startswith('.env.'):
raise RuntimeError('Invalid record path')
if pathlib.Path(cfg['paths']['credentials']) in path.parents:raise RuntimeError('Credential source refused')
record_stat=path.lstat()
if not stat.S_ISREG(record_stat.st_mode) or record_stat.st_nlink!=1 or record_stat.st_uid!=0:
raise RuntimeError('A root-owned, single-link regular staging copy is required')
if record_stat.st_mode&0o022:raise RuntimeError('Record must not be group/world-writable')
raw=path.read_bytes()
if not 16<=len(raw)<=64*1024*1024:raise RuntimeError('Record size invalid')
media='image/png' if raw.startswith(b'\x89PNG\r\n\x1a\n') else 'image/jpeg' if raw.startswith(b'\xff\xd8\xff') else 'text/plain'
captures.append({'role':role,'path':str(path),'sha256':hashlib.sha256(raw).hexdigest(),'media_type':media})
doc={'schema':'iboot-physical-test-evidence-v1','installation_id':cfg['installation_id'],
'test_id':pending['test_id'],'physical_target':pending['physical_target'],
'captured_by':ask(tty,'Name of operator who observed the load:'),
'captured_utc':ask(tty,'Actual observation time ISO8601 with timezone offset:'),
'observations':observations,'captures':captures}
if datetime.datetime.fromisoformat(doc['captured_utc'].replace('Z','+00:00')).tzinfo is None:
raise RuntimeError('Timezone offset required')
output=pathlib.Path(ask(tty,'New absolute private observation JSON path:'))
if not output.is_absolute() or output.exists() or output.is_symlink():raise RuntimeError('New absolute path required')
doc['record_sha256']=hashlib.sha256(canonical(doc)).hexdigest()
fd=os.open(output,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600)
with os.fdopen(fd,'wb') as stream:stream.write(canonical(doc));stream.flush();os.fsync(stream.fileno())
print('PHYSICAL_EVIDENCE_FILE='+str(output))
PY
)
IBOOT_PHYSICAL_EVIDENCE_STATUS=$?
printf 'IBOOT_PHYSICAL_EVIDENCE_STATUS=%s\n' "$IBOOT_PHYSICAL_EVIDENCE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Host/shell: postboot commissioning Ubuntu host, Bash. Privilege: sudo/root. Effect: require changed boot, matching nonreplayed intent/target/configuration, healthy resumed workload, independent effective device readback and four fresh ACKs, then seal the physical acceptance receipt. Expected: PASS_OPERATOR_OBSERVED_PHYSICAL_TEST, hardware_test_completed:true, automatic_outlet_telemetry:false. Failure: retain intent, captures and provider; do not start automatic management to override failed commissioning.
Guarded v0.2.0 reference example — stops immediately; do not remove the guard
(
set -eu
printf '%s\n' 'BLOCKED: this v0.2.0 example requires a corrected, reviewed release.' >&2
false
read -r -p 'Specific actual Main host identity matching this test: ' IBOOT_MAIN_LOAD
read -r -p 'Absolute actual physical observation JSON path: ' IBOOT_PHYSICAL_EVIDENCE
test -n "$IBOOT_MAIN_LOAD"
test -f "$IBOOT_PHYSICAL_EVIDENCE"
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json accept-test \
--evidence "$IBOOT_PHYSICAL_EVIDENCE" --main-load "$IBOOT_MAIN_LOAD" \
--exclusive-management --deliberate-power-test
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json status
)
IBOOT_ACCEPT_TEST_STATUS=$?
printf 'IBOOT_ACCEPT_TEST_STATUS=%s\n' "$IBOOT_ACCEPT_TEST_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Only a corrected release that has passed this physical acceptance can proceed to automatic service activation below. Physical acceptance does not prove a later real 24-hour renewal or every failure mode. Operator timing and machine-observed reboot/health remain separate evidence classes.
6.1 Activate automatic operation only after corrected-release acceptance
This sequence fills an omission in the original manual. It is a release requirement, not a workaround for v0.2.0’s failed commissioning. No activation command is provided for the blocked release. The implementation agent must deliver a checked target-specific command block that carries out these stages and stops on a failed structured result.
| Order | Gate or action | Required observation |
|---|---|---|
| 1. Accept the target | Validate the corrected release’s source/config/unit receipts, named physical load, current boot, fresh ACKs, verified workload and new physical acceptance. | Actual acceptance record, no unresolved fence/incident, manager still inactive. |
| 2. Prepare alert delivery | If email is enabled, configure the protected SMTP credential and approved recipient locally before starting the mail timer. | TLS/account configuration passes local validation; messages are not sent before that scope is authorized. |
| 3. Start automatic recovery | Enable and start iboot-reference-manager.service, then iboot-reference-renew.timer, using the installed names if customized. | Correct source-bound manager invocation; timer active and its next trigger visible. There is no generated manager timer. |
| 4. Start optional mail | Enable/start iboot-reference-mail.timer only when configured and authorized. | Queue, delivery attempts and SMTP acceptance remain distinct from confirmed inbox receipt. |
| 5. Verify ordinary operation | Run ordinary live health after the manager starts; inspect structured fields, service results and current scientific evidence. | All applicable health gates pass. An inactive timer, stale status or a mere exit-zero start request is not acceptance. |
| 6. Record and hand over | Capture status/report, versions, source hashes, schedules, observed limits and maintenance/abort instructions. | A reviewable acceptance report; real 24-hour renewal remains pending until observed. On failure, pause automatic dispatch while preserving heartbeat and the known safe device state. |
The exact unit names above come from units.py; the final command sequence must be tested against the corrected package. It must not print PASS based only on systemctl enable --now returning zero.
7. Maintenance and removal
Host/shell: target Ubuntu host, Bash. Privilege: sudo/root. Effect: request heartbeat action None and independently verify it before entering MAINTENANCE. No provider stop. Expected: PASS_EFFECTIVELY_DISARMED, mode MAINTENANCE, automatic_rearm false. Failure: keep provider running and preserve uncertainty; do not proceed to uninstall.
(
set -eu
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json maintenance \
--exclusive-management
sudo python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json status
)
IBOOT_MAINTENANCE_STATUS=$?
printf 'IBOOT_MAINTENANCE_STATUS=%s\n' "$IBOOT_MAINTENANCE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Follow INSTALL_OPERATION.md for the subsequent service stops and gated uninstall. A setting that is merely accepted but pending controller restart is not a safe uninstall proof. This candidate refuses ambiguous disarming instead of using a possibly unavailable reset allowance to force removal. Resolve the effective device state on the identified harmless commissioning arrangement and preserve the failed transaction before removal. Manual restoration is required after maintenance; there is no automatic time-limited action None lease.
Appendix C. Scientific adapter and email contract
Version: iboot-sample-v1, reference release 0.2.0.
This is a real, harmless integer-square computation with independently verifiable durable output. It is a fresh reference adapter. It does not accept Stage10c/R2 seals, implement a Stage10d executor, or establish compatibility with the separate scientific-integration branch. The recovered R2 requirements remain scientific-design evidence; translating a real scientific code to this interface requires that code's own numerical and environmental validation.
Authority and identity
The installed root configuration selects the service UID/GID, unit, source directory, control/output paths, total work count and per-tile delay. The installed root-owned systemd unit passes only public workload parameters to the worker. Credentials and private root configuration are not passed to the unprivileged worker.
control/: root-owned, mode 0755.RUN.jsonandHOLD.jsonare root-owned, mode 0644, so the worker can read them but cannot alter them.workload/: configured service-UID-owned, mode 0700. Tile, status, worker-lock and completion files are mode 0600.state/science/: root's privateStore; immutable preparation seals, resume intents and migration transactions are retained. A preparation pointer selects the current immutable seal but does not replace its evidence.RUN.jsonbinds a random run identity to sample-v1 parameters, installation identity, control/output locations, service UID/GID, exact unit, exactworkload.pydigest, and the Python/platform execution fingerprint.- Scientific run identity comprises sample-v1, run UUID, fixed integer-square algorithm and total work count. It is stable through an explicitly reviewed source migration. Every preparation separately binds the full source/configuration specification.
The production module requires root for its administrative interface. Offline tests inject a fake Context and use their actual user's UID as the fixture owner; that substitution exists in test code, not an environment-variable override in the production worker.
Actual computation and durable publication
Tile i is exactly {schema, run_id, index:i, value:i*i}, encoded as canonical JSON. Indices start at zero and commits form one contiguous prefix. Root independently recomputes the expected value for every committed tile; a tile's self-reported status or hash is not trusted.
Each publication writes a new temporary regular file in the output directory, flushes it with fsync, installs its permanent tile name with a no-overwrite hard-link operation, removes the temporary name and fsyncs the containing directory. A previously committed name is never overwritten. Directory sync depends on the target filesystem honoring Linux durability semantics; storage hardware and filesystem commissioning remain necessary.
Uncommitted .tmp-<uuid> fragments are ignored as scientific output. They remain available for diagnostics. Malformed, missing-in-the-middle, duplicated-index, symlinked, hardlinked, wrongly owned or wrong-mode committed records are rejected. A failed result is not replaced by recomputing it until the dashboard becomes green.
The single worker holds an exclusive flock on worker.lock. It validates the complete existing committed prefix before continuing. A completion receipt is published only after all expected tiles validate. If a process dies after publishing the last tile but before the completion record, restart recomputes the complete prefix and can publish the missing completion record. An existing corrupt completion record blocks continuation.
This demonstration recomputes all simple tiles during validation. Its full scans are deliberately straightforward and can grow expensive for very large runs. They are not an optimized scientific index. A production numerical adapter needs an equally strong, reviewed validation strategy and a progress window appropriate to its actual work duration.
Worker entry point
The installed unit uses Python isolation and this exact argument order:
/usr/bin/python3 -I -B /absolute/versioned/code/workload.py --workload /absolute/output --control /absolute/control --total 200000 --delay 1
The paths and numeric text are generated from the accepted configuration. This is unit syntax, not a command to paste with the example paths. The installer supplies the actual unit. Python's isolated mode requires the module's explicit insertion of its own verified code directory before importing shared local modules.
The service must use KillSignal=SIGTERM, SendSIGKILL=no and an unbounded systemd stop timeout; the administrative preparation request has its own bounded wait and never manufactures readiness on expiration. A TERM request stops admission at the next tile boundary. A tile already admitted completes and publishes before the worker returns.
Administrative API
| Function | Success result | Failure behavior |
|---|---|---|
initialize(ctx) |
INITIALIZED or INITIALIZED_EXISTING; creates only file state |
Existing output without enrollment or changed run bindings refuses initialization; no services start |
prepare(ctx) |
READY, exact preparation identity and verified committed count |
Leaves protected HOLD; no automatic SIGKILL, no unverified READY, no host reboot |
ready(ctx) |
Fresh READY result with current boot and preparation boot |
Recomputes state and rejects old/corrupt/changed evidence, active worker or busy lock |
resume(ctx) |
STARTED_GATE_PENDING with healthy=false |
Retains immutable dispatch intent; failed/uncertain start is not erased or blindly replayed |
health(ctx, window=None) / observe(...) |
PROGRESS or COMPLETE with independent evidence |
Returns healthy=false, a phase and explicit reasons; protected HOLD is separately visible |
run_worker(output, control, total, delay) / worker_main(...) |
Returns zero after clean drain or valid completion | Raises Failure for integrity/identity failure; CLI prints reason and returns its status |
migrate(ctx, previous_config, previous_receipt, rollback=False) |
MIGRATED_READY; workload remains held |
Rejects unverified old authority, changed scientific parameters/output, concurrent workers or unresolved conflicting transaction |
Root's ctx.verify_sources() must check installed source/config/unit authority before these interfaces run. prepare and resume additionally require ctx.command_allowed(); being root alone does not authorize service mutation.
Preparation and independent readiness
- Observe the actual service. If active, validate its
MainPID, invocation ID, boot, OS UID,/procstart ticks and exact isolated Python worker command line before requesting a stop. - Publish protected root HOLD durably. It contains the run, parameter/source digest and unique preparation identity. The worker cannot admit another tile after observing it.
- Request the configured service stop. Its current tile drains according to the unit's no-SIGKILL policy. A timeout or failed stop leaves HOLD and refuses readiness.
- Independently require a clean inactive unit, PID zero, successful unit result and free exclusive worker lock. Inspect all committed results. A status file alone cannot satisfy this gate.
- Publish an immutable root seal binding the preparation/run, full source/config specification, HOLD bytes, control/output directory device/inode identities, committed count, per-file digest aggregate and independently recomputed sum.
- Publish the root pointer to that seal. Every subsequent
readyrepeats source validation, current unit and lock checks, tile validation, digest comparison and directory identity checks.
After a host reboot the old seal is not accepted merely because it says READY. The same independent checks run again against the current boot and quiescent service. If unchanged state validates, ready reports both preparation_boot and current_boot. The manager can then resume the exact prepared incident. The unprivileged worker never removes a previous-boot HOLD by itself.
Resume, interrupted dispatch and verified progress
Resume first repeats readiness, then writes a no-overwrite root resume intent, removes HOLD durably, and asks systemd to start the accepted unit. Returning from systemctl start is only STARTED_GATE_PENDING.
If the operation fails or stops between those steps, the intent is retained. A repeated resume does not silently reset history or turn ambiguity into a new dispatch. The manager/operator must reconcile the current HOLD, unit and health evidence. An already-running valid worker can be observed without another start; an incomplete transition may require a fresh controlled preparation before a new resume.
To report progress, health observes two validated tile snapshots across a configured window. It verifies the same live boot/PID/start-ticks/UID/invocation identity and matching current worker status, requires growth, and checks the earlier committed prefix is unchanged. A live PID, old invocation's record, worker-supplied count or start-request receipt cannot produce a pass.
Completion requires every tile and completion receipt to validate, plus clean inactive service and available exclusive worker lock. A complete numerical result with an active or failed service is not reported as accepted complete/quiescent.
For abrupt power loss with no HOLD, startup verifies the existing prefix and continues. This preserves accepted committed output and recomputes uncommitted work. It does not claim that every calculation in flight survived loss of electrical power.
Source migration and rollback
The installer calls migration only while the upgrade fence is active, the old workload is prepared and all relevant units are stopped. It must authenticate previous_receipt against its own root-owned installation archive before calling.
The receipt contains:
config_sha256: canonical previous configuration digest;workload_sha256: previous pinned worker source digest;manifest_sha256: previous accepted release-manifest identity;previous_root_install_receipt: the installer-authenticated prior receipt with matching configuration/worker digests.
This API accepts only a code-directory change in configuration. Scientific parameters, identities, paths, workload unit and execution fingerprint must remain unchanged. It checks the previous source bytes, previous control/specification, old protected HOLD and old root seal, independently validates all output, and retains the same scientific run identity.
Before changing control records, it creates an immutable transaction with the old and proposed new RUN/HOLD/seal. It then publishes the new root control generation, new immutable seal and selected pointer. All old seals and tile bytes remain. A process interrupted between publications leaves the installer fence in effect; calling the exact migration again reconciles only its recorded old/new states. Any third state blocks.
Rollback uses the same procedure with rollback=True and its own authenticated current/target installation authorities. It is a validated source rebind, never removal of output, spent action budgets, incidents, HOLD or scientific history. A Python/platform fingerprint or mathematical contract change requires separately reviewed scientific validation; this code-only migration does not authorize it.
Offline validation boundary
The workload tests exercise real temporary files and worker threads, durable commits, continuation, corruption, same-invocation progress, drain/ready/resume, previous-boot readiness, interrupted completion, no-overwrite behavior, symlink/hardlink rejection and migration reconciliation. Their fake systemd runner never contacts a host service. Passing them does not establish target systemd wiring, actual power-loss durability, iBoot relay operation or correctness of any numerical research code.
These modules are a new parameterized derivative of the reviewed O3/RC1/C2 rules. They do not reuse any RecipeGram installation receipt as target authority. The complete release manifest, root-owned configuration and installed unit hashes must verify before observations can authorize an action. Target commissioning and the exact supported device/firmware profile remain separate release gates.
What health means
recovery.health(ctx, purpose='observe') returns healthy, result, reasons
and the component observations. result=PASS requires all applicable checks:
| Component | Required observation |
|---|---|
| Primary Ethernet | Exact configured interface, link UP, global non-link-local IPv4, default route on that interface, matching configured gateway and interface-bound successful gateway ping |
| Optional upstream probes | Up to three configured hosts, each reached using the primary interface; inability to reach them is distinguished from a failed local path |
| Application | Configured service running with successful Result/ExecMainStatus; configured HTTPS endpoint returns the exact expected status and, if specified, the case-sensitive body marker |
| Scientific sample | The workload adapter independently verifies current bound progress or validated completion with quiescence; PID existence is insufficient |
| USB heartbeat | Heartbeat module's exact fresh ACK and trusted current-process/unit evidence; configuration persistence is a separate device gate |
| Recovery state | No open recovery incident, manual recovery HOLD, controller INHIBIT, pending allowance-renewal transaction or blocked deployment |
| Source/configuration | Installed manifest, config and unit hashes still match the accepted installation |
Optional Wi-Fi observations are telemetry; they neither veto primary recovery nor establish external access. This implementation reports no tested inbound VPN, NAT or routing recovery from an interface address alone. A configured application without a body marker provides an HTTP-status check, not application transaction semantics. Configure a meaningful nonsecret health marker or a dedicated health endpoint before relying on backend readiness.
The endpoint uses curl with verified TLS, HTTPS-only protocol, no redirect
following, two-second connection and five-second total transfer limits; the
parent command has a seven-second deadline. Response content is not stored in
health evidence. Gateway/probe pings and ip commands have separate bounded
deadlines. No shell command is assembled from a configuration string.
purpose='renewal' additionally requires ARMED deployment. purpose='arm'
permits DISARMED, COMMISSIONING or MAINTENANCE and can accept a held workload solely after
an independent fresh workload.ready() proves the sealed run quiescent. The
deliberate power-test purpose applies the same science requirement in ARMED
mode. Ordinary health/renewal never treats an existing HOLD as normal health.
Recovery sequence
The manager runs from its own systemd service and survives the SSH session. It
does not repair anything in DISARMED or MAINTENANCE. Every real action requires
ARMED, allow_live_actions=true, source checks and the relevant action option.
- Record an incident and detection notice. Confirm the fault over the configured number of observations (default three). A single failed check is insufficient.
- For network faults, allow the configured infrastructure grace (default 900 seconds) from detection. This permits a router/DHCP server to return later than the host. It does not establish the cause of the interruption.
- If the local path is healthy but configured upstream probes fail, record a blocked upstream incident; restarting the host is not justified by this case.
- For local network failure, perform at most two explicitly enabled
NetworkManager connection reactivations:
nmcli --wait 20 connection up idwith the exact configured connection ID. This derivative does not perform RecipeGram's board-specific PCI remove/rescan operation. A host using another network manager must leave this action disabled until a reviewed adapter is supplied. It can still be configured for the separate governed warm-reboot fallback after the infrastructure grace. - An application-only fault can restart only its configured service, at most twice. It does not initiate a NIC action or host reboot.
- If local network recovery remains unsuccessful and host reboot is explicitly enabled, prepare the workload, wait for its bounded preparation result and independently verify READY. Integrity failure or uncertain preparation prevents a host reboot. Recheck Ethernet after preparation; if it returned, resume the sealed workload and verify progress without rebooting.
- Otherwise reserve one host reboot and request
systemctl --no-block reboot. Submission acceptance is not successful recovery. The next boot must be observed, the exact sealed workload resumed, and current progress/completion, heartbeat and network/application checks verified. Multiple healthy observations are required before the incident is closed.
Scientific or heartbeat failure vetoes ordinary network/application actuators. However, v0.2.0 does not implement the claimed loss-of-heartbeat scientific drain: the manager defers with a guardian-related reason, but the heartbeat module contains no workload preparation/HOLD path. The controller mutex does not supply that behavior. Persistent USB failure can therefore leave science writing until the external timer expires. A corrected release must implement and test a bounded containment policy before arming; abrupt AC loss and a locked kernel remain outside any graceful-drain guarantee. R24 is a separate budget and evidence chain.
Durable limits, interruption and reconciliation
Before each action, the manager stores a reservation and increments the action counter, then writes an immutable action-intent record. A crash or timeout cannot refund that counter. Successful command exit means submission was observed; it does not prove the fault recovered. Nonzero or lost results retain a pending reservation and block automatic replay. Reboot requests remain pending until a new boot is observed, with a same-boot waiting period rather than immediate resubmission. An unexpected boot is reported for review.
State is under the configured private root, principally
recovery/incident.json, recovery/actions/, recovery/closed/ and immutable
events. Do not remove a pending intent, edit a counter or delete an incident to
rearm a recovery path. A later healthy reconciliation retains all action counts
and receipts. An open incident prevents controller allowance renewal even when
the currently observed services are healthy.
Host/shell: target Ubuntu Bash. Privilege: sudo. Effect: observation only; no deliberate outage, reboot, service restart or email submission. Expected: a JSON health/status result. Failure: preserve the specific reasons and collect a report; do not change state files to manufacture PASS.
(
set -eu
sudo /usr/bin/python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json health
sudo /usr/bin/python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json status
)
REFERENCE_OBSERVATION_STATUS=$?
printf 'REFERENCE_OBSERVATION_STATUS=%s\n' "$REFERENCE_OBSERVATION_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Host/shell: target Ubuntu Bash. Privilege: sudo. Effect: explicit
operator review of a blocked incident; independently observes current health
and may close the incident while retaining spent counts/history. It performs no
recovery command and refunds no allowance. Expected: RECOVERY_VERIFIED or
NO_UNRESOLVED_INCIDENT. Failure: RECONCILIATION_NOT_HEALTHY; retain the
incident and repair the reported condition before another review.
(
set -eu
sudo /usr/bin/python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json reconcile
)
REFERENCE_RECONCILE_STATUS=$?
printf 'REFERENCE_RECONCILE_STATUS=%s\n' "$REFERENCE_RECONCILE_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Secure email setup and delivery
The main configuration contains SMTP host/port, mode starttls or smtps,
username, sender and an explicit recipient list. Password content is never
configuration data. The separate setup command obtains it without terminal
echo, writes the private credential file, and the mail service receives it
through systemd LoadCredential=smtp_password:.... The drainer reads only
$CREDENTIALS_DIRECTORY/smtp_password, with type, link, owner, mode and size
checks. It does not fall back to .env, shell variables or a password in a URL.
Both STARTTLS and implicit TLS use certificate verification and TLS 1.2 or newer. Plain SMTP is refused. SMTP attempts run under a real total deadline of 15 seconds, including DNS/login/protocol work, with four-second socket timeouts. A queue pass attempts at most three messages; the installed mail service's 180-second outer deadline also covers source and filesystem validation.
Host/shell: target Ubuntu Bash. Privilege: sudo. Effect: privately
provisions the SMTP credential only; does not send mail. Expected:
CREDENTIAL_WRITTEN_PRIVATELY. Failure: retain the refusal and correct the
configured private credential directory; never paste the password into logs.
(
set -eu
sudo /usr/bin/python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json credential-set --kind smtp
)
REFERENCE_SMTP_CREDENTIAL_STATUS=$?
printf 'REFERENCE_SMTP_CREDENTIAL_STATUS=%s\n' "$REFERENCE_SMTP_CREDENTIAL_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
The following deliberately submits a test alert to the recipients already selected in configuration. It requires email enabled and live effects enabled through the commissioning procedure. Do not use another person's address as an unrequested test target.
Host/shell: target Ubuntu Bash. Privilege: sudo. Effect: queues one
clearly labeled operator test, then starts the configured mail drainer using
systemd credentials; sends to configured recipients. Expected: QUEUED or
ALREADY_QUEUED, then a journal delivery outcome. SMTP_ACCEPTED is still not
inbox confirmation. Failure: retain queue/receipt; no manual resend after
ACCEPTANCE_UNKNOWN. Confirm inbox arrival independently.
(
set -eu
sudo /usr/bin/python3 -I -B /opt/iboot-reference/0.2.0/cli.py \
--config /etc/iboot-reference/config.json notice \
--kind detected --incident-id operator-mail-test-001 \
--message 'Operator-requested mail-path test; no failure or recovery is asserted.'
sudo systemctl start iboot-reference-mail.service
sudo journalctl --no-pager --unit=iboot-reference-mail.service --lines=40
)
REFERENCE_MAIL_TEST_STATUS=$?
printf 'REFERENCE_MAIL_TEST_STATUS=%s\n' "$REFERENCE_MAIL_TEST_STATUS"
printf 'RETURNED_TO_INTERACTIVE_SHELL=YES\n'
Queue IDs are derived from installation identity, event kind and incident ID.
Repeating the same notice does not generate another message. Delivery records
are bound to the immutable queued message hash. Before DATA transmission the
drainer durably records DATA_PENDING; interruption or timeout after this point
becomes ACCEPTANCE_UNKNOWN and is not automatically retried. This accepts a
possible missing alert instead of claiming certainty or duplicating an uncertain
submission. Known temporary pre-DATA failures retry with exponential backoff,
at most twelve attempts within 48 hours; permanent rejection, expiry and
exhaustion remain visible. A known negative DATA reply is distinguishable from
an unknown submission result. Successful SMTP 250 records SMTP_ACCEPTED with
inbox_receipt=NOT_CONFIRMED.
Notices distinguish detection, blocked recovery and verified recovery. They include timestamped observations and same-boot failed-check-to-recovery intervals where those exist. Exact electrical-outage cause/duration remains unknown without independent evidence. No interval is labeled exact Internet downtime simply because a host rebooted or the first probe failed.
Validation scope
The tests replace system commands, device evidence, SMTP transports and clocks; they exercise actual persistent records in private temporary directories. Recovery tests include failed stopped units, semantic health checks, grace, two bounded reactivations, integrity veto, late network return, durable ambiguous submission, host boot transition and verified scientific continuation. Mail tests cover deduplication, TLS-mode refusal, credential metadata, data-phase ambiguity, retries, exhaustion and acceptance versus receipt.
These tests do not assert physical NIC recovery, live SMTP acceptance, inbox delivery, actual systemd boot ordering, real relay timing or target compatibility. Those are explicitly separate commissioning results. The NetworkManager adapter is a portable policy choice; it is not evidence that PCIe-detached hardware can be recovered by connection activation on every machine.
Appendix D. Exact software validation boundary
INVALID_UNIT on the manager-timer query. The original fakes did not reveal that integration defect. The missing heartbeat-loss drain, commissioning-health interface and lifecycle gaps also remain release blockers despite these passes.210 offline tests passed under permission masks 0000, 0002, 0022 and 0077: 840 successful executions, no failures/errors/skips. Runtime: Python 3.12.14, Linux x86-64, UID 0. Exact machine-readable results are in validation/offline-tests.json.
Tests cover source/configuration trust, durable private state, installer interruption, exact USB acknowledgement handling, fragmented Telnet sessions, reset reservations, simulated 24-hour boundary/renewal interruption, source migration, sample corruption/prepare/resume, recovery incidents, email queue/TLS and device commissioning transitions. Joined tests use actual component code with injected OS/device/network transports. A harmless sample computes and validates integer squares in scratch directories.
Two additional regressions verify that an uncommissioned manager cannot dispatch and maintenance can return through commissioning with scientifically ready work.
Not performed: ordinary-user tests (the execution namespace maps only UID/GID 0; setpriv refused UID 65534), actual Ubuntu/systemd installation, physical USB/Telnet/relay commissioning, real SMTP delivery, real 24-hour renewal, real source upgrade/rollback or power-loss tests on another host. Changed umasks are not a substitute for ordinary-user tests. Run the same suite without sudo on the target before installation; any failure blocks installation.
Installer, controller and scientific migration functions are implemented and exercised offline, but failure injection is not exhaustive at every filesystem boundary. Interrupted source migration can intentionally retain a fence requiring diagnosis; this is containment, not a promise of automatic repair. No runtime action on RecipeGram or an iBoot was performed for this release.
Captured offline-suite output
The following is the actual captured summary from the development workspace; it is not a device screenshot or a physical commissioning result.
{"errors": 0, "failures": 0, "skipped": 0, "successful": true, "tests_run": 210, "uid": 0, "umask": "0000"}
{"errors": 0, "failures": 0, "skipped": 0, "successful": true, "tests_run": 210, "uid": 0, "umask": "0002"}
{"errors": 0, "failures": 0, "skipped": 0, "successful": true, "tests_run": 210, "uid": 0, "umask": "0022"}
{"errors": 0, "failures": 0, "skipped": 0, "successful": true, "tests_run": 210, "uid": 0, "umask": "0077"}
Appendix E. Implementation prompt for GPT-6 Astra Max
Copy the complete prompt below into a new conversation, attach this article, and identify the Ubuntu machine you want to equip. The prompt guides the implementation from source review and correction of the known release defects through target commissioning and operating instructions. Existing installations require a reviewed migration; the historical RecipeGram receipts do not authorize changes on another host.
The model name identifies your requested assistant. Actual execution depends on the tools and target access available in that conversation. Download verification, ordinary-user tests, fresh hardware evidence and workload validation remain required regardless of the assistant used.
PROMPT FOR GPT-6 ASTRA MAX
Implement this recovery design on the requesting user's own Ubuntu machine
ROLE AND DELIVERABLE
Act as the implementation engineer for the attached article by Dr. Julio C.
Spinelli, “Linux unattended crash recovery system for off-site management of
servers using iBoot-G2+.” Read the complete article, including its appendices,
before proposing target changes. Implement and validate a target-specific
deployment through evidence-based stages. Produce working configuration,
scripts, reports, commissioning records and an operating runbook. Do not stop
at a generic plan when the next safe implementation step can be completed.
Work only through tools actually available in this conversation. If you cannot
access the target, provide complete commands for the user to execute, review
the returned reports and continue from the observed result. Never imply that
a command ran, a setting changed or a physical test passed without evidence.
This prompt authorizes source review, isolated offline work and preparation of
a reviewable target implementation. Before applying changes, establish the
user's actual deployment scope and existing authorizations. Disruptive actions
require explicit authorization covering their target and effects. Do not ask
again for permission already granted for the same action and scope.
AUTHORITATIVE INPUTS AND VERSION BOUNDARY
Use the user's latest uploaded project files as the editing baseline. The
article's historical RecipeGram installation is a case study, not the target
configuration and not an installation authority for another machine.
Portable reference 0.2.0:
https://juliospinelli.com/wp-content/uploads/2026/09/iboot_ubuntu_reference_v0.2.0.zip
SHA-256:
82240026a56e5f856f68116d82d1941be804f5d0ffb9270cf7654e3ded07477f
Article review materials v0.2:
https://juliospinelli.com/wp-content/uploads/2026/09/iboot_article_review_materials_v0.2.zip
SHA-256:
fb091a5b8b54fa74f48d2299ae703bf1c91190c5f20c66be88f881f59358f06f
Download to a fresh review directory and verify both complete ZIPs before
extraction or execution. Inspect archive paths and refuse unsafe traversal,
symlink destinations or overwriting an existing review/deployment directory.
Read README.md, VALIDATION.md, INTERFACES.md, MANIFEST.json and all four manuals
under docs/ in the portable package. Use the review ZIP to trace claims and
remaining tests. It is not an installer; its article source predates later HTML
edits and need not reproduce the current article bytes.
These are integrity checksums, not digital signatures. Stop on a mismatch;
do not substitute a newly calculated value and call the download verified.
Record the digest of the exact current article separately. Do not embed its
own full-file digest in the bytes being hashed. For any changed software,
create a distinct version, manifest, test record and checksum; never silently
change the published 0.2.0 package or weaken its source-binding checks.
The reference is a release candidate. Its original 840 offline executions do
not prove ordinary-user operation, target systemd integration, actual relay
behavior, real SMTP delivery or a real 24-hour allowance renewal. Preserve each
unknown until newly collected evidence closes it. Confirm relevant current
vendor documentation when needed; use primary sources and the actual device's
model/firmware. Do not invent commands, settings, screenshots or support answers.
KNOWN LIVE-COMMISSIONING BLOCKERS IN THE UNCHANGED 0.2.0 ZIP
Treat published 0.2.0 as reference material pending newly versioned repairs,
not as a ready-to-arm release. This article review identified these defects:
1. Context.unit() accepts only .service names, but the device commissioning
gate queries the manager's .timer. The real path therefore fails INVALID_UNIT
in the deliberate power-test, acceptance and abort flows. Injected tests
did not establish that this joined implementation path works.
2. Existing physical-test intents prevent an ordinary repeat, while source or
configuration changes invalidate prior physical acceptance. The package
lacks a complete versioned recommissioning transition after an upgrade.
3. If arming consumes the controller-reset allowance, its trusted successful
recovery proof must be captured while the original heartbeat invocation is
still available. A deliberate host reboot before that proof can leave
renewal unable to establish the required recovery success.
4. The portable heartbeat implementation does not invoke workload preparation
or HOLD, although recovery reports that a guardian owns heartbeat-loss
quiescence. No implemented handoff establishes that protection. Persistent
USB failure can therefore let the external timer expire while an otherwise
live host is still writing scientific results. Implement and test a bounded
heartbeat-loss drain/containment policy, with an explicit time budget and
conservative failure outcome. Do not claim graceful preparation during a
kernel freeze or abrupt AC loss, when host code cannot execute.
5. The ordinary CLI health path expects an active manager, whereas commissioning
requires the manager inactive and needs its own health purpose. Provide a
correctly gated commissioning-health interface and prove actual postboot
ready/resume/progress through it; status 0 or a report alone is insufficient.
6. MAINTENANCE can be a dead end when the controller allowance is already spent:
arm requires an unused allowance, while renewal requires ARMED. Implement
and test a coherent maintenance/rearming transition that preserves all spent
history and does not silently grant a reset. Never discourage necessary
disarming to work around this defect, and do not treat waiting in maintenance
as sufficient to renew the allowance.
Before arming, physical testing or automatic activation, reproduce and repair
these defects in a new version, review state migration and conservative failure
behavior, and validate the actual Context/unit/device/heartbeat/recovery paths
together. Preserve test generations, all action intents and spent history;
do not fix repeatability by clearing records. Audit containment/abort with the
same real Context as normal execution. Do not bypass unit-name checks, fabricate
recovery proofs or extend accepted pins merely to make a gate pass. Deliver
the corrected package, new manifest/checksum and tests before requesting live
commissioning. This article-editing task did not modify either published ZIP.
PHASE 1 — IDENTIFY THE TARGET AND EXISTING PROTECTION
Begin with a read-only inventory and a short plain-language account of what is
known, what is missing and which next observation will resolve it. Ask only
questions that cannot be answered safely from available evidence. Establish:
* The exact machine, Ubuntu/kernel/systemd/Python/pyserial versions, architecture,
service account UID/GID, persistent storage, available privileges and tools.
* Whether this is a separate commissioning host or an existing production
installation; any current watchdog/provider, owned services, release pins,
receipts, HOLDs, open incidents, pending actions and spent reset allowances.
* Primary interface, driver/hardware identity, gateway and network manager.
Record the exact NetworkManager profile only if that adapter applies.
* Optional Wi-Fi, source routing and remote-access route. Local address/route
presence does not prove inbound VPN/NAT access. Preserve a tested recovery
path for the operator before any network change.
* Actual application unit and a safe HTTPS health endpoint with meaningful
expected status/content. A PID or generic HTTP 200 page is insufficient.
* Actual workload, durable output, validation rules, longest legitimate unit
of work, preparation bound, continuation environment and backup arrangement.
* Physical iBoot model/firmware, stable /dev/serial/by-id identity, serial
ownership, management address and protected LAN/VPN access.
* The traced Main load, voltage/current/cable ratings, other loads, BIOS/UEFI
AC-restore behavior, router/modem/switch power paths and local operator.
* All automation able to switch Main, including AutoPing, schedules, graceful
shutdown, expansion/external rules and manual management sessions.
Do not read, search, hash, collect or print .env files or credential contents.
Do not request secrets in chat or place them in command arguments, environment
dumps, shell history, screenshots or reports. Use protected local prompts and
the package's credential mechanism. Keep complete infrastructure inventories
private; redact shared evidence. No bulk collection of home directories.
If protection already exists, do not blindly run the fresh installer, start a
second serial writer, adopt old receipts or regenerate allowances. Prepare an
explicit coexistence/migration plan from verified current source and state.
Keep the existing provider running until the external watchdog is verifiably
disarmed or an accepted replacement owns the heartbeat. If compatibility is
unresolved, continue isolated analysis and hold the live transition.
PHASE 2 — AUDIT THE CODE AND RUN ORDINARY-USER OFFLINE TESTS
Review actual implementation against this target before sudo installation.
Verify the manifest and run the packaged offline suite as an ordinary user,
including its supported permission-mask cases. Record effective UID, runtime,
test count, failures and exact source digests. Root tests with varied umasks
are not a substitute. A failing security or permission check blocks installation;
fix the cause in a new reviewed release, then rerun the affected meaningful
tests. Do not chmod private state broadly to make a test pass.
Apply the known-blocker repairs above before relying on this package for live
commissioning. Tests must use the actual Context implementation for both
service and timer gates, verify the entire finite test/accept/abort route, and
exercise new commissioning generations after source migration. Include the
reset-success-proof-before-reboot sequence and reject forged or stale proof.
Exercise sustained USB failure while the host and workload remain alive. Prove
that the accepted guardian path takes ownership, prepares or contains the
workload within the available deadline, and records failure without inventing
READY. Also test the case where host execution is lost and only crash-consistent
committed results can be preserved. Verify commissioning-specific health with
the manager intentionally inactive, then normal health after accepted activation.
Add a regression that enters maintenance with a spent allowance and exercises
the corrected authorized exit/rearming path without refunding, deleting or
relabeling the old attempt. Check absent timer handling against the generated
unit set; this package has no manager timer.
Audit the source/config/unit bindings; filesystem ownership and link checks;
bounded serial/management/network calls; exclusive serial/management locking;
durable intent before action; handling of crashes between intent and result;
conservative handling of ambiguous outcomes; installation fencing; scientific
seals; and preservation of spent budgets across restart, upgrade and rollback.
Check the service dependencies and boot behavior on the actual systemd version.
Keep five mechanisms distinct: hardware outlet cycling, Ubuntu warm reboot,
iBoot-controller reboot, hardware rearming and software allowance renewal.
Never substitute an outlet-cycle command for the controller reboot command.
R24 can restore one future controller-reset allowance after verified success,
the full cooldown and health gates; it is not a scheduled daily reboot. An
unused allowance remains one. Uncertain or failed recovery earns no replacement.
PHASE 3 — ADAPT THE WORKLOAD AND PREPARE A DISARMED INSTALLATION
Build one target configuration using discovered values. Resolve all enabled
actions, paths, owners, units, credentials, network probes, timing and limits.
Keep live recovery disabled and services inactive at initial installation.
Do not use broad source find-and-replace for host adaptation.
The supplied sample-v1 adapter is a harmless integer-square workload. It is
not a Stage10c/R2 drop-in, not a Stage10d implementation and not evidence that
arbitrary scientific software can resume safely. Exercise the sample first.
For the user's real workload, implement and review its own adapter:
* Durable atomic output publication with validation of committed versus partial
work; exact run/input/parameter/code/environment identity.
* A protected HOLD that prevents new work admission, bounded drain of existing
work and an explicit refusal when preparation fails or times out.
* Independent READY checks for clean quiescence, inactive successful service,
no worker and available exclusive lock, plus validated sealed results.
* Resume intent retained across interruption; no blind replay after ambiguity.
* Current-invocation progress proved by two valid snapshots with growth and
unchanged committed prefix, or validated completion with quiescence.
* Restart after abrupt loss that rejects corruption/incompatible state and may
recompute uncommitted work without claiming zero loss of in-flight computation.
* Explicit rules for source migration and rollback that retain scientific
results, prior seals, incident history and spent/uncertain action records.
If a real workload adapter is not yet accepted, perform disruptive sample
commissioning on a separate test machine. A host reboot or Main outlet power
cut affects every process and storage writer on the connected host; configuration
cannot exempt an unaccepted workload. Before any disruptive test on a shared
machine, explicitly identify, stop and secure all other workloads with the
user's authorization. Do not enable automatic disruptive recovery while any
active workload on that host lacks an accepted recovery policy.
Produce an installation plan, changes/backups inventory, private-state layout,
reversal procedure and expected postchecks. Apply only the reviewed target
installation within the user's authorized scope. Do not stop at asking for
approval while this preparatory work remains unfinished. Retain a failed
installation's fence and report instead of repeatedly applying it.
PHASE 4 — VERIFY DISARMED HARDWARE AND HOST INTEGRATION
Start with an identified harmless load and a locally present operator. The
candidate recognizes the narrow firmware 1.62.117 profile: USB heartbeat,
controller frequency 60 seconds, fail count 30, Main, one maximum cycle,
30-second cycle interval, initial ON, Disable OFF enabled, and independent
switching disabled. Setup heartbeat action must be effectively None. Other
firmware or policy values require reviewed implementation support and tests;
do not spoof the firmware banner or alter an evidence file to satisfy the gate.
Read and reconcile genuine UI/CLI evidence for the actual target. Distinguish
accepted setters from independently effective readback. Device-disarm changes
the heartbeat action; it does not configure every other required field. Do not
assume a pending-restart change is effective or that an ACK proves saved action.
Maintain one management owner, close other sessions and account for the
observed possibility of management interfering with USB communication.
Enter credentials privately. Establish verified disarming before starting
the new provider and sample. Validate fresh exact ACKs, real systemd ownership,
source checks, sample prepare/ready/resume, current progress/completion and
network/application observations. The nominal threshold is 1,800 seconds;
measure the actual target timing and boot margin rather than treating that
product as a physical test result. Do not invent a separate heartbeat boot-grace
setting or use AutoPing restart delay as one.
PHASE 5 — AUTHORIZE AND PERFORM DISTINCT COMMISSIONING TESTS
Present the concrete test plan, affected load, anticipated SSH interruption,
expected evidence, containment/abort procedure and operator role before each
new class of disruptive action. Obtain explicit authorization for controller
configuration/arming, controller reboot, network/service interruption, host
reboot and deliberate Main power interruption as applicable. Prior permission
for one does not silently authorize all others. Configure SMTP and authorize
recipient-specific test delivery separately; never send an unsolicited alert.
Bind genuine captures, timestamps, configuration and physical load identity
to fresh target evidence. A lamp record cannot authorize a different host.
Keep the automatic manager service and renewal/mail timers inactive through the finite physical test.
Prove the workload READY before an authorized host interruption. Follow the
package's full commissioning and containment transitions, including boot-time
provider recovery; do not replace them with an ad hoc stop-and-wait script.
Test applicable cases with preserved receipts: fresh heartbeat operation;
transport recovery and authorized bounded controller recovery; normal host
restart; router returning after the host; local network versus upstream fault;
application-only failure; scientific integrity/readiness refusal; and the
explicitly authorized finite outlet watchdog test. For the latter, record
heartbeat-stop-to-actual-OFF, actual OFF duration, physical load observation,
changed boot, fresh ACKs, and validated postboot scientific continuation.
After the deliberate test, a prepared sample may correctly remain held. Run
the new version's independent postboot ready and controlled resume transition,
then verify fresh progress or validated completion; neither boot success nor
a READY record alone means scientific computation resumed. If arming spent a
controller reset, capture and verify its trusted recovery-success proof before
the deliberate host reboot while the original heartbeat invocation still
exists. Preserve the full cooldown and do not manufacture a replacement token.
Increasing trigger counters or successful command submission do not prove
relay operation or recovery. Arming acceptance is not physical-test acceptance.
If a test times out, an ACK fails or a mutation becomes uncertain, preserve the
record and enter the documented containment path. Do not delete an intent,
HOLD, fence or spent counter to obtain another attempt. Fresh status/report
collection is appropriate; repeating an uncertain mutation is not.
For email, use TLS and the protected credential setup. Distinguish queued,
SMTP_ACCEPTED, ACCEPTANCE_UNKNOWN and independently confirmed inbox receipt.
Do not resend an uncertain submission automatically. Outcome messages must
state observed Ethernet, optional Wi-Fi, application, heartbeat and scientific
state, with blockers and timestamped measurement definitions. Do not invent
electrical-outage duration or exact Internet downtime from boot/probe gaps.
PHASE 6 — ACCEPT, DOCUMENT AND MAINTAIN
Enable automatic manager/recovery only after the required target gates pass.
Before unattended use, deliver an acceptance table separating PASS, FAIL,
BLOCKED and NOT PERFORMED, each linked to exact evidence and software/config
digests. Label simulated versus real tests. Do not claim a real 24-hour renewal
until an eligible actual recovery, uninterrupted cooldown and renewal evidence
have occurred; no unnecessary outage is required merely to fill a checkbox.
Provide the installed inventory and manifest, redacted configuration, backups,
test transcripts, commissioning receipts, HTML/runbook, report archive and
SHA-256 sidecars. The final status must state what is actually installed,
running, armed, tested and still unknown, and the exact next action if blocked.
Document normal status/report collection, incident reconciliation, protected
maintenance, credential rotation, updates, rollback and uninstall. Maintenance
must independently establish effective disarming before provider removal or
service stopping that could let an armed watchdog expire. An unreachable
controller is not proof of disarming. Preserve the working provider and safe
physical arrangement until resolved. There is no automatic rearm timer to
assume. Require fresh readiness and commissioning evidence after a source-bound
change; never erase budgets or reuse historical commissioning as new authority.
COMMAND AND COMMUNICATION REQUIREMENTS
Give complete copyable blocks, one action group at a time. Label the target,
shell, privilege, effect, expected success, failure/abort response and report
path. Prefer bounded script-first/report-driven steps with explicit status
markers. Do not terminate the user's interactive SSH shell: put strict shell
settings inside a subshell and print procedure status afterward; do not append
exit to report blocks. Quote paths safely and do not rely on shell expansion of
untrusted input. Provide separate Windows PowerShell transfer instructions if
the user operates from Windows, using their verified SSH alias and real paths.
After each returned report, verify its checksums and explain the conclusion
and practical limits before the next dependent phase. Keep observations,
inferences, operator attestations and vendor statements distinct. Be candid
when evidence is missing. Continue all authorized, safe work; pause only for a
real missing input, unresolved gate or ungranted disruptive action.
START NOW
Read the supplied article and verify the two packages. Identify the target and
whether it already has active protection. Then provide the first bounded,
read-only inventory step and complete whatever isolated source review and
offline preparation are possible before requesting target changes. Reproduce
the known 0.2.0 blockers and prepare a newly versioned corrected implementation;
do not instruct the user to arm the unchanged published package.