v0.3.13-pre.011

crates/ksp-worker-raw-transaction-ingest-lib/README.md

@@ -1,5 +1,5 @@
 <!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
-<!-- version: 10 -->
+<!-- version: 11 -->
 
 # ksp-worker-raw-transaction-ingest-lib
 
@@ -165,7 +165,7 @@ Le Worker s'exécute sur le runtime Tokio courant du caller. Il ne crée pas de
 - utiliser la même source via `WorkerSnapshotSource` ;
 - attendre le terminal après drain et join des tâches possédées.
 
-Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source relaie le stop à toutes les sources et les rejoint avant de rendre son résultat au supervisor Worker ; les tâches source, hydration et persistence possédées sont ensuite drainées ou abort+join avant publication terminale. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
+Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source relaie le stop à toutes les sources et les rejoint avant de rendre son résultat au supervisor Worker ; les tâches source, hydration et persistence possédées sont ensuite drainées ou abort+join avant publication terminale. Si la deadline expire, l'abort du wrapper source détruit aussi son `JoinSet` interne et annule ses tâches imbriquées avant le terminal. Une faute déjà observée n'est pas remplacée par un stop concurrent, sauf le `drain_timeout` terminal lorsqu'une récupération bornée dépasse sa deadline. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
 
 ## Admission, coalescence et backpressure
 
@@ -183,7 +183,7 @@ source reference-bearing active => quota pending >= 1 et quota in-flight >= 1
 
 Pour garantir simultanément ces bornes et l'absence de starvation structurelle, le démarrage échoue avant spawn si le nombre de sources reference-bearing dépasse `admission_queue_capacity` ou `persistence_concurrency`. Un sémaphore global protège en plus l'ouverture effective des hydrations HTTP.
 
-Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu canonique identique ajoutent uniquement leur observation déterministe. Une divergence de slot, block time, format ou hash canonique devient un content conflict terminal ; aucune majorité, préférence provider ou overwrite n'est appliqué. Le Store conserve son guard durable final.
+Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. La publication du résultat partagé notifie les followers sous le verrou de registry avant de retirer la clé : une nouvelle génération de leader ne peut donc pas s'intercaler entre retrait et notification. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu canonique identique ajoutent uniquement leur observation déterministe. Une divergence de slot, block time, format ou hash canonique devient un content conflict terminal ; aucune majorité, préférence provider ou overwrite n'est appliqué. Le Store conserve son guard durable final.
 
 Les retries/reroutages HTTP appartiennent à `ksp-onchain-transport-lib`. Le Worker ne possède pas une seconde boucle de retry autour de `getTransaction`.
 
@@ -279,7 +279,7 @@ worker_raw_transaction_ingest.source_failed
 worker_raw_transaction_ingest.drain_timeout
 ```
 
-Les diagnostics et `Debug` ne recopient pas de payload RAW, signature, URL, credential, filtre provider, texte backend/provider arbitraire ou client inférieur.
+Les diagnostics et `Debug` ne recopient pas de payload RAW, signature, URL, credential, filtre provider, texte backend/provider arbitraire ou client inférieur. Les agrégations de compteurs de continuité multi-source utilisent une arithmétique vérifiée ; un overflow devient `counter_exhausted` au lieu d'être saturé silencieusement.
 
 ## Dépendances
 
@@ -302,8 +302,7 @@ La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Sto
 
 La verticale actuelle ne possède pas :
 
-- plusieurs sources productives simultanées dans `RawTransactionIngestRuntimeResources` ;
-- sélection Config interne au Worker ;
+- sélection Config interne, reconfiguration dynamique ou policy de failover entre sources ;
 - checkpoint persistent de processing frontier ;
 - campagne de réparation historique automatique ;
 - application Desk ou process autonome ;

crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md

@@ -1,5 +1,5 @@
 <!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
-<!-- version: 10 -->
+<!-- version: 11 -->
 
 # Utilisation de ksp-worker-raw-transaction-ingest-lib
 
@@ -333,7 +333,7 @@ let replay_attempts = snapshot.source_replay_attempt_total();
 let proven_gaps = snapshot.source_continuity_gap_total();
 ```
 
-`admission_queue_depth()`, `in_flight_persistence()`, `hydration_pending()`, `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` sont des gauges latest-value. Les compteurs cumulés ne wrapent jamais silencieusement ; l'épuisement est terminal avec `worker_raw_transaction_ingest.counter_exhausted`.
+`admission_queue_depth()`, `in_flight_persistence()`, `hydration_pending()`, `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` sont des gauges latest-value. Les compteurs cumulés et les agrégats multi-source ne wrapent ni ne saturent silencieusement ; l'épuisement est terminal avec `worker_raw_transaction_ingest.counter_exhausted`.
 
 ## Interpréter la processing frontier
 
@@ -386,7 +386,7 @@ if accepted {
 }
 ```
 
-`request_stop()` est idempotent. Le terminal n'est publié qu'après le drain borné et la récupération des tâches possédées.
+`request_stop()` est idempotent. Le terminal n'est publié qu'après le drain borné et la récupération des tâches possédées. Si la deadline de drain expire, toutes les tâches source/persistence encore possédées sont abortées puis jointes avant publication terminale ; une persistence libérée après ce terminal ne peut donc pas produire une complétion tardive. Une faute source déjà observée reste prioritaire face à un stop concurrent, sauf si le drain lui-même expire et devient le terminal `drain_timeout`.
 
 ## Interpréter les faults
 

crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs

@@ -1,11 +1,12 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
-// version: 25
+// version: 26
 
 #![warn(missing_docs)]
 #![deny(unreachable_pub)]
 #![forbid(unsafe_code)]
 
 //! Source-neutral runtime foundation for continuous KSP RAW transaction ingestion.
+//! Owned source, hydration and persistence tasks are bounded and joined before terminal publication.
 //!
 //! This tranche owns the concrete Worker family identity, validated technical settings
 //! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also

crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
-// version: 16
+// version: 17
 
 type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
 type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
@@ -343,13 +343,16 @@ fn source_completion(
     in_flight_persistence: usize,
     snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
 ) -> std::option::Option<ksp_core_lib::ErrorCode> {
-    match joined {
+    let source_code = match joined {
         std::result::Result::Ok(std::result::Result::Ok(())) => return std::option::Option::None,
-        std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => {},
-    }
+        std::result::Result::Ok(std::result::Result::Err(error)) if error.code() == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED => {
+            crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED
+        },
+        std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED,
+    };
     let published = snapshots.record_source_failure(state, admission_queue_depth, in_flight_persistence);
     return match published {
-        std::result::Result::Ok(()) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED),
+        std::result::Result::Ok(()) => std::option::Option::Some(source_code),
         std::result::Result::Err(error) => std::option::Option::Some(error.code()),
     };
 }

crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
-// version: 25
+// version: 26
 
 use sha2::Digest; // rust-rules: trait-import
 
@@ -156,14 +156,18 @@ impl RawTransactionIngestLiveSource {
                     } else {
                         crate::RawTransactionIngestSourceState::Failed
                     };
-                    inventory_publisher.publish(source_projection_with_state(latest, terminal_state));
+                    if let std::result::Result::Err(error) = inventory_publisher.publish(source_projection_with_state(latest, terminal_state)) {
+                        return std::result::Result::Err(error);
+                    }
                     return result;
                 }
                 changed = source_frontier_receiver.changed() => {
                     if changed.is_err() {
                         return std::result::Result::Err(crate::runtime_error("source.frontier_channel_closed"));
                     }
-                    inventory_publisher.publish(*source_frontier_receiver.borrow_and_update());
+                    if let std::result::Result::Err(error) = inventory_publisher.publish(*source_frontier_receiver.borrow_and_update()) {
+                        return std::result::Result::Err(error);
+                    }
                 }
             }
         }
@@ -171,7 +175,7 @@ impl RawTransactionIngestLiveSource {
 }
 
 impl RawTransactionIngestSourceInventory {
-    fn aggregate(&self) -> crate::RawTransactionIngestProcessingFrontierProjection {
+    fn aggregate(&self) -> ksp_core_lib::Result<crate::RawTransactionIngestProcessingFrontierProjection> {
         let mut hydration_pending = 0_usize;
         let mut oldest_pending_slot = std::option::Option::None;
         let mut processing_frontier_slot = std::option::Option::None;
@@ -189,7 +193,10 @@ impl RawTransactionIngestSourceInventory {
         let mut any_reconnecting = false;
         let mut all_closed = !self.source_projections.is_empty();
         for projection in &self.source_projections {
-            hydration_pending = hydration_pending.saturating_add(projection.hydration_pending());
+            hydration_pending = match hydration_pending.checked_add(projection.hydration_pending()) {
+                std::option::Option::Some(value) => value,
+                std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("hydration_pending")),
+            };
             oldest_pending_slot = minimum_optional_slot(oldest_pending_slot, projection.oldest_pending_slot());
             match projection.processing_frontier_slot() {
                 std::option::Option::Some(slot) => {
@@ -199,9 +206,18 @@ impl RawTransactionIngestSourceInventory {
                     all_frontiers_present = false;
                 },
             }
-            source_continuity_gap_total = source_continuity_gap_total.saturating_add(projection.source_continuity_gap_total());
-            source_reconnect_total = source_reconnect_total.saturating_add(projection.source_reconnect_total());
-            source_replay_attempt_total = source_replay_attempt_total.saturating_add(projection.source_replay_attempt_total());
+            source_continuity_gap_total = match source_continuity_gap_total.checked_add(projection.source_continuity_gap_total()) {
+                std::option::Option::Some(value) => value,
+                std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("source_continuity_gap_total")),
+            };
+            source_reconnect_total = match source_reconnect_total.checked_add(projection.source_reconnect_total()) {
+                std::option::Option::Some(value) => value,
+                std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("source_reconnect_total")),
+            };
+            source_replay_attempt_total = match source_replay_attempt_total.checked_add(projection.source_replay_attempt_total()) {
+                std::option::Option::Some(value) => value,
+                std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("source_replay_attempt_total")),
+            };
             match projection.source_state() {
                 std::option::Option::Some(crate::RawTransactionIngestSourceState::Active) => {
                     source_active += 1;
@@ -244,9 +260,11 @@ impl RawTransactionIngestSourceInventory {
         } else {
             std::option::Option::None
         };
-        return crate::RawTransactionIngestProcessingFrontierProjection::new(hydration_pending, processing_frontier_slot, oldest_pending_slot)
-            .with_source_continuity(source_state, source_reconnect_total, source_replay_attempt_total, source_continuity_gap_total)
-            .with_source_counts(source_total, source_active, source_reconnecting, source_failed);
+        return std::result::Result::Ok(
+            crate::RawTransactionIngestProcessingFrontierProjection::new(hydration_pending, processing_frontier_slot, oldest_pending_slot)
+                .with_source_continuity(source_state, source_reconnect_total, source_replay_attempt_total, source_continuity_gap_total)
+                .with_source_counts(source_total, source_active, source_reconnecting, source_failed),
+        );
     }
 
     fn new(source_keys: std::vec::Vec<[u8; 32]>) -> Self {
@@ -254,35 +272,45 @@ impl RawTransactionIngestSourceInventory {
         return Self { source_keys, source_projections };
     }
 
-    fn update(&mut self, entry_index: usize, source_key: [u8; 32], projection: crate::RawTransactionIngestProcessingFrontierProjection) {
+    fn update(
+        &mut self,
+        entry_index: usize,
+        source_key: [u8; 32],
+        projection: crate::RawTransactionIngestProcessingFrontierProjection,
+    ) -> ksp_core_lib::Result<()> {
         let expected_key = match self.source_keys.get(entry_index) {
             std::option::Option::Some(value) => value,
-            std::option::Option::None => return,
+            std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.inventory_entry_missing")),
         };
         if expected_key != &source_key {
-            return;
+            return std::result::Result::Err(crate::runtime_error("source.inventory_key_mismatch"));
         }
         let current = match self.source_projections.get_mut(entry_index) {
             std::option::Option::Some(value) => value,
-            std::option::Option::None => return,
+            std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.inventory_projection_missing")),
         };
         *current = projection;
-        return;
+        return std::result::Result::Ok(());
     }
 }
 
 impl RawTransactionIngestSourceInventoryPublisher {
-    fn publish(&self, projection: crate::RawTransactionIngestProcessingFrontierProjection) {
+    fn publish(&self, projection: crate::RawTransactionIngestProcessingFrontierProjection) -> ksp_core_lib::Result<()> {
         let aggregate = {
             let mut inventory = match self.inventory.lock() {
                 std::result::Result::Ok(value) => value,
                 std::result::Result::Err(poisoned) => poisoned.into_inner(),
             };
-            inventory.update(self.entry_index, self.source_key, projection);
-            inventory.aggregate()
+            if let std::result::Result::Err(error) = inventory.update(self.entry_index, self.source_key, projection) {
+                return std::result::Result::Err(error);
+            }
+            match inventory.aggregate() {
+                std::result::Result::Ok(value) => value,
+                std::result::Result::Err(error) => return std::result::Result::Err(error),
+            }
         };
         self.aggregate_sender.send_replace(aggregate);
-        return;
+        return std::result::Result::Ok(());
     }
 }
 
@@ -3502,15 +3530,19 @@ impl RawTransactionIngestGlobalHydrationRegistry {
     }
 
     fn publish_and_remove(&self, key: &RawTransactionIngestHydrationKey, result: RawTransactionIngestSharedHydrationResult) {
-        let sender = {
-            let mut pending = match self.pending.lock() {
-                std::result::Result::Ok(value) => value,
-                std::result::Result::Err(poisoned) => poisoned.into_inner(),
-            };
-            pending.remove(key)
+        let mut pending = match self.pending.lock() {
+            std::result::Result::Ok(value) => value,
+            std::result::Result::Err(poisoned) => poisoned.into_inner(),
         };
-        if let std::option::Option::Some(sender) = sender {
-            sender.send_replace(std::option::Option::Some(result));
+        let published = match pending.get(key) {
+            std::option::Option::Some(sender) => {
+                sender.send_replace(std::option::Option::Some(result));
+                true
+            },
+            std::option::Option::None => false,
+        };
+        if published {
+            let _removed = pending.remove(key);
         }
         return;
     }

crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
-// version: 27
+// version: 28
 
 //! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
 
@@ -663,3 +663,42 @@ fn v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backe
     }
     return;
 }
+
+#[test]
+fn v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades() {
+    let runtime = include_str!("../src/runtime.rs");
+    let resources = include_str!("../src/runtime_resources.rs");
+    for required in [
+        "tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
+        "persistence.abort_all()",
+        "children.abort_all()",
+        "while persistence.join_next().await.is_some() {}",
+        "while children.join_next().await.is_some() {}",
+        "ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
+    ] {
+        assert!(runtime.contains(required), "required pre.011 runtime shutdown guard missing: {required}");
+    }
+    for required in [
+        "RawTransactionIngestHydrationLeaderGuard",
+        "publish_and_remove",
+        "pending.remove(key)",
+        "source.inventory_key_mismatch",
+        "checked_add(projection.hydration_pending())",
+        "checked_add(projection.source_reconnect_total())",
+    ] {
+        assert!(resources.contains(required), "required pre.011 source-race guard missing: {required}");
+    }
+    for forbidden in [
+        "unbounded_channel",
+        "ksp_store_postgres_lib::",
+        "ksp_config_lib::",
+        "ksp_job_backfill_lib::",
+        "reqwest::",
+        "tokio_tungstenite::",
+        "tonic::",
+        "yellowstone_grpc_proto::",
+    ] {
+        assert!(!runtime.contains(forbidden) && !resources.contains(forbidden), "pre.011 crossed a Worker/facade boundary: {forbidden}");
+    }
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs

@@ -1,7 +1,7 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
-// version: 22
+// version: 23
 
-//! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
+//! External public, security, redaction and release-boundary hardening canaries through `pre.011`.
 
 fn network(value: &'static str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
     let result = ksp_store_lib::RawNetworkId::new(value);
@@ -854,3 +854,81 @@ fn v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted() {
     }
     return;
 }
+
+#[test]
+fn v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe() {
+    let runtime = include_str!("../src/runtime.rs");
+    let resources = include_str!("../src/runtime_resources.rs");
+    let supervisor = match runtime.split_once("async fn run_supervisor<Spawner>(") {
+        std::option::Option::Some((_, tail)) => match tail.split_once("fn spawn_persistence(") {
+            std::option::Option::Some((value, _)) => value,
+            std::option::Option::None => "",
+        },
+        std::option::Option::None => "",
+    };
+    let active_shutdown = match supervisor.split_once("let mut fault = supervise_until_stop(") {
+        std::option::Option::Some((_, value)) => value,
+        std::option::Option::None => "",
+    };
+    let stop_position = active_shutdown.find("source_stop_sender.send_replace(true);");
+    let stopping_position = active_shutdown.find("begin_stopping(");
+    let drain_position = active_shutdown.find("drain_owned_work(");
+    let terminal_position = active_shutdown.find("match fault {");
+    assert!(matches!((stop_position, stopping_position), (std::option::Option::Some(stop), std::option::Option::Some(stopping)) if stop < stopping));
+    assert!(matches!((stopping_position, drain_position), (std::option::Option::Some(stopping), std::option::Option::Some(drain)) if stopping < drain));
+    assert!(matches!((drain_position, terminal_position), (std::option::Option::Some(drain), std::option::Option::Some(terminal)) if drain < terminal));
+    for required in [
+        "tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
+        "persistence.abort_all()",
+        "children.abort_all()",
+        "while persistence.join_next().await.is_some() {}",
+        "while children.join_next().await.is_some() {}",
+    ] {
+        assert!(runtime.contains(required), "required pre.011 bounded-drain guard missing: {required}");
+    }
+    let registry_publish = match resources.split_once("fn publish_and_remove(&self, key: &RawTransactionIngestHydrationKey") {
+        std::option::Option::Some((_, tail)) => match tail.split_once("type RawTransactionIngestHydrationTasks") {
+            std::option::Option::Some((value, _)) => value,
+            std::option::Option::None => "",
+        },
+        std::option::Option::None => "",
+    };
+    let notify_position = registry_publish.find("sender.send_replace(std::option::Option::Some(result));");
+    let remove_position = registry_publish.find("pending.remove(key);");
+    assert!(matches!((notify_position, remove_position), (std::option::Option::Some(notify), std::option::Option::Some(remove)) if notify < remove));
+    let inventory = match resources.split_once("impl RawTransactionIngestSourceInventory {") {
+        std::option::Option::Some((_, tail)) => match tail.split_once("impl RawTransactionIngestSourceInventoryPublisher") {
+            std::option::Option::Some((value, _)) => value,
+            std::option::Option::None => "",
+        },
+        std::option::Option::None => "",
+    };
+    for required in [
+        "checked_add(projection.hydration_pending())",
+        "checked_add(projection.source_continuity_gap_total())",
+        "checked_add(projection.source_reconnect_total())",
+        "checked_add(projection.source_replay_attempt_total())",
+        "source.inventory_entry_missing",
+        "source.inventory_key_mismatch",
+        "source.inventory_projection_missing",
+    ] {
+        assert!(inventory.contains(required), "required pre.011 inventory guard missing: {required}");
+    }
+    assert!(!inventory.contains("saturating_add"), "pre.011 source inventory must not silently saturate aggregate counters");
+    for source_marker in [
+        "impl crate::RawTransactionIngestYellowstoneSource",
+        "impl crate::RawTransactionIngestHeliusTransactionSource",
+        "impl crate::RawTransactionIngestHttpBlockPollingSource",
+        "impl crate::RawTransactionIngestStandardBlockSource",
+        "impl crate::RawTransactionIngestStandardLogsSource",
+    ] {
+        let tail = match resources.split_once(source_marker) {
+            std::option::Option::Some((_, value)) => value,
+            std::option::Option::None => "",
+        };
+        assert!(tail.contains("stop_receiver.changed()"), "pre.011 source lacks stop-preemptible wait: {source_marker}");
+    }
+    assert!(!runtime.contains("unbounded_channel"));
+    assert!(!resources.contains("unbounded_channel"));
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
-// version: 20
+// version: 21
 
 //! External public-surface proofs for the RAW transaction ingest Worker foundation.
 
@@ -385,3 +385,27 @@ fn v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral()
     }
     return;
 }
+
+#[test]
+fn v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface() {
+    let root = include_str!("../src/lib.rs");
+    for forbidden in [
+        "pub use self::runtime_resources::RawTransactionIngestSourceInventory",
+        "pub use self::runtime_resources::RawTransactionIngestSourceInventoryPublisher",
+        "pub use self::runtime_resources::RawTransactionIngestGlobalHydrationRegistry",
+        "pub use self::runtime_resources::RawTransactionIngestHydrationLeaderGuard",
+        "publish_and_remove",
+        "source.inventory_key_mismatch",
+    ] {
+        assert!(!root.contains(forbidden), "pre.011 private race-hardening implementation leaked publicly: {forbidden}");
+    }
+    assert_eq!(
+        ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED,
+        ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "counter_exhausted")
+    );
+    assert_eq!(
+        ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT,
+        ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "drain_timeout")
+    );
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs

@@ -1,7 +1,7 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
-// version: 18
+// version: 19
 
-//! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
+//! Release-completeness canaries through the `pre.011` races/shutdown hardening tranche.
 
 #[test]
 fn pre_010_production_module_inventory_is_exact() -> std::io::Result<()> {
@@ -131,6 +131,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
         "v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private",
         "v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit",
         "v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted",
+        "v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe",
     ] {
         assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
     }
@@ -152,6 +153,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
     assert!(dependency_boundary.contains("v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_only"));
     assert!(dependency_boundary.contains("v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades"));
     assert!(dependency_boundary.contains("v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backend_free"));
+    assert!(dependency_boundary.contains("v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades"));
     let public_api = include_str!("public_api.rs");
     assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
     assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
@@ -166,6 +168,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
     assert!(public_api.contains("v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private"));
     assert!(public_api.contains("v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private"));
     assert!(public_api.contains("v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral"));
+    assert!(public_api.contains("v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface"));
     assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
     assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
     return;

crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
-// version: 10
+// version: 11
 
 struct ActiveTaskGuard {
     active: std::sync::Arc<std::sync::atomic::AtomicUsize>,
@@ -899,3 +899,98 @@ async fn pre_009_drain_timeout_aborts_and_joins_all_owned_source_and_persistence
     assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
     return;
 }
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_011_source_counter_exhaustion_stays_terminal_and_is_not_collapsed_to_source_failed() {
+    let settings = match settings("mainnet") {
+        std::option::Option::Some(value) => value,
+        std::option::Option::None => return,
+    };
+    let handle = match super::start_foundation_with_source_spawner(
+        settings,
+        tokio::runtime::Handle::current(),
+        std::option::Option::None,
+        move |children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>, _stop_receiver, _admission_sender| {
+            let _abort_handle = children.spawn(async move {
+                return std::result::Result::Err(crate::counter_exhausted_error("source_reconnect_total"));
+            });
+        },
+    ) {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    let source = handle.snapshot_source();
+    let terminal = match handle.wait_terminal().await {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
+    let snapshot = source.current();
+    assert_eq!(snapshot.source_failure_total(), 1);
+    assert_eq!(snapshot.worker_snapshot().state(), terminal);
+    assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
+    let terminal_sequence = snapshot.worker_snapshot().sequence();
+    assert!(!handle.request_stop());
+    let retained = source.current();
+    assert_eq!(retained.worker_snapshot().state(), terminal);
+    assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
+    return;
+}
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_011_drain_timeout_prevents_late_persistence_completion_after_terminal() {
+    let settings = match settings_with_runtime_limits(1, 1, crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT) {
+        std::option::Option::Some(value) => value,
+        std::option::Option::None => return,
+    };
+    let network = settings.network().clone();
+    let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
+    let runtime_port: super::PersistencePort = port.clone();
+    let source_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
+    let source_active_for_task = std::sync::Arc::clone(&source_active);
+    let handle = match super::start_foundation_with_port_and_source_spawner(
+        settings,
+        tokio::runtime::Handle::current(),
+        std::option::Option::Some(runtime_port),
+        move |children, _stop_receiver, admission_sender| {
+            let _abort_handle = children.spawn(async move {
+                let ingress = match runtime_ingress(&network, 81) {
+                    std::option::Option::Some(value) => value,
+                    std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
+                };
+                if admission_sender.send(ingress).await.is_err() {
+                    return std::result::Result::Err(crate::runtime_error("test.source_failed"));
+                }
+                let _guard = ActiveTaskGuard::new(source_active_for_task);
+                return std::future::pending::<ksp_core_lib::Result<()>>().await;
+            });
+        },
+    ) {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    assert!(wait_for_max_active(&port, 1).await);
+    assert!(wait_for_active_count(&source_active, 1).await);
+    assert!(handle.request_stop());
+    let terminal = match handle.wait_terminal().await {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT));
+    let snapshot = handle.snapshot_source().current();
+    let terminal_sequence = snapshot.worker_snapshot().sequence();
+    assert_eq!(snapshot.in_flight_persistence(), 0);
+    assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
+    assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
+    assert_eq!(source_active.load(std::sync::atomic::Ordering::Acquire), 0);
+    port.release();
+    for _ in 0..64 {
+        tokio::task::yield_now().await;
+    }
+    assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
+    let retained = handle.snapshot_source().current();
+    assert_eq!(retained.worker_snapshot().state(), terminal);
+    assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
+    assert_eq!(retained.in_flight_persistence(), 0);
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
-// version: 22
+// version: 23
 
 fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
     return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
@@ -971,13 +971,13 @@ fn v0_3_13_pre_007_source_inventory_aggregates_frontier_and_lifecycle_conservati
         source_key: [1_u8; 32],
     };
     let second = super::RawTransactionIngestSourceInventoryPublisher { aggregate_sender, entry_index: 1, inventory, source_key: [2_u8; 32] };
-    first.publish(
-        crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(50), std::option::Option::Some(45)).with_source_continuity(
-            std::option::Option::Some(crate::RawTransactionIngestSourceState::Active),
-            1,
-            2,
-            0,
-        ),
+    assert!(
+        first
+            .publish(
+                crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(50), std::option::Option::Some(45))
+                    .with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 2, 0),
+            )
+            .is_ok()
     );
     let first_aggregate = *aggregate_receiver.borrow();
     assert_eq!(first_aggregate.hydration_pending(), 1);
@@ -988,13 +988,13 @@ fn v0_3_13_pre_007_source_inventory_aggregates_frontier_and_lifecycle_conservati
     assert_eq!(first_aggregate.source_active(), 1);
     assert_eq!(first_aggregate.source_reconnecting(), 0);
     assert_eq!(first_aggregate.source_failed(), 0);
-    second.publish(
-        crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(42), std::option::Option::Some(40)).with_source_continuity(
-            std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting),
-            3,
-            4,
-            1,
-        ),
+    assert!(
+        second
+            .publish(
+                crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(42), std::option::Option::Some(40))
+                    .with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting), 3, 4, 1),
+            )
+            .is_ok()
     );
     let aggregate = *aggregate_receiver.borrow();
     assert_eq!(aggregate.hydration_pending(), 3);
@@ -3146,3 +3146,204 @@ async fn v0_3_13_pre_009_global_hydration_permit_and_leader_cleanup_are_bounded_
     assert!(matches!(resubscribed, std::result::Result::Ok((true, _))));
     return;
 }
+
+#[test]
+fn v0_3_13_pre_011_source_inventory_rejects_stale_identity_and_counter_exhaustion() {
+    let mut identity_inventory = super::RawTransactionIngestSourceInventory::new(std::vec![[1_u8; 32]]);
+    let mismatch = identity_inventory.update(0, [2_u8; 32], crate::RawTransactionIngestProcessingFrontierProjection::empty());
+    let mismatch = match mismatch {
+        std::result::Result::Ok(()) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(mismatch.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
+    assert_eq!(mismatch.context()[0].value(), "source.inventory_key_mismatch");
+    let missing = identity_inventory.update(1, [1_u8; 32], crate::RawTransactionIngestProcessingFrontierProjection::empty());
+    let missing = match missing {
+        std::result::Result::Ok(()) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(missing.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
+    assert_eq!(missing.context()[0].value(), "source.inventory_entry_missing");
+    let mut pending_inventory = super::RawTransactionIngestSourceInventory::new(std::vec![[3_u8; 32], [4_u8; 32]]);
+    assert!(
+        pending_inventory
+            .update(
+                0,
+                [3_u8; 32],
+                crate::RawTransactionIngestProcessingFrontierProjection::new(usize::MAX, std::option::Option::Some(10), std::option::Option::None),
+            )
+            .is_ok()
+    );
+    assert!(
+        pending_inventory
+            .update(1, [4_u8; 32], crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(11), std::option::Option::None),)
+            .is_ok()
+    );
+    let pending_error = match pending_inventory.aggregate() {
+        std::result::Result::Ok(_) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(pending_error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED);
+    assert_eq!(pending_error.context()[0].value(), "hydration_pending");
+    let mut continuity_inventory = super::RawTransactionIngestSourceInventory::new(std::vec![[5_u8; 32], [6_u8; 32]]);
+    assert!(
+        continuity_inventory
+            .update(
+                0,
+                [5_u8; 32],
+                crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(20), std::option::Option::None)
+                    .with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), u64::MAX, 0, 0),
+            )
+            .is_ok()
+    );
+    assert!(
+        continuity_inventory
+            .update(
+                1,
+                [6_u8; 32],
+                crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(21), std::option::Option::None)
+                    .with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 0, 0),
+            )
+            .is_ok()
+    );
+    let continuity_error = match continuity_inventory.aggregate() {
+        std::result::Result::Ok(_) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(continuity_error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED);
+    assert_eq!(continuity_error.context()[0].value(), "source_reconnect_total");
+    return;
+}
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_011_aborted_hydration_leader_notifies_followers_and_releases_registry_key() {
+    let network = match ksp_store_lib::RawNetworkId::new("devnet") {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    let registry = std::sync::Arc::new(super::RawTransactionIngestGlobalHydrationRegistry::new(2, 1));
+    let key = super::RawTransactionIngestHydrationKey {
+        commitment: ksp_onchain_transport_lib::SolanaCommitment::Confirmed.as_str(),
+        network,
+        signature: ksp_store_lib::RawTransactionSignature::new([91_u8; 64]),
+    };
+    let subscribed = registry.subscribe_or_lead(&key);
+    let (leader, follower) = match subscribed {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    assert!(leader);
+    let task_registry = std::sync::Arc::clone(&registry);
+    let task_key = key.clone();
+    let leader_task = tokio::spawn(async move {
+        let _guard = super::RawTransactionIngestHydrationLeaderGuard::new(task_registry, task_key);
+        return std::future::pending::<()>().await;
+    });
+    tokio::task::yield_now().await;
+    leader_task.abort();
+    let _joined = leader_task.await;
+    let published = follower.borrow().clone();
+    match published {
+        std::option::Option::Some(super::RawTransactionIngestSharedHydrationResult::Failed(code)) => {
+            assert_eq!(code, crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED);
+        },
+        _ => return,
+    }
+    assert!(matches!(registry.subscribe_or_lead(&key), std::result::Result::Ok((true, _))));
+    return;
+}
+
+struct Pre011SourceActiveGuard {
+    active: std::sync::Arc<std::sync::atomic::AtomicUsize>,
+}
+
+impl Pre011SourceActiveGuard {
+    fn new(active: std::sync::Arc<std::sync::atomic::AtomicUsize>) -> Self {
+        active.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
+        return Self { active };
+    }
+}
+
+impl std::ops::Drop for Pre011SourceActiveGuard {
+    fn drop(&mut self) {
+        self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
+        return;
+    }
+}
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_011_aborting_outer_source_supervisor_aborts_nested_source_tasks() {
+    let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
+    let (_stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
+    let (source_stop_sender, _source_stop_receiver) = tokio::sync::watch::channel(false);
+    let mut children = tokio::task::JoinSet::new();
+    let task_active = std::sync::Arc::clone(&active);
+    let _abort_handle = children.spawn(async move {
+        let _guard = Pre011SourceActiveGuard::new(task_active);
+        return std::future::pending::<ksp_core_lib::Result<()>>().await;
+    });
+    let supervisor = tokio::spawn(super::supervise_live_source_tasks(stop_receiver, source_stop_sender, children));
+    for _ in 0..64 {
+        if active.load(std::sync::atomic::Ordering::Acquire) == 1 {
+            break;
+        }
+        tokio::task::yield_now().await;
+    }
+    assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 1);
+    supervisor.abort();
+    let _joined = supervisor.await;
+    for _ in 0..64 {
+        if active.load(std::sync::atomic::Ordering::Acquire) == 0 {
+            break;
+        }
+        tokio::task::yield_now().await;
+    }
+    assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
+    return;
+}
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_011_stop_racing_ready_source_fault_preserves_fault_and_joins_sibling() {
+    let sibling_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
+    let fault_gate = std::sync::Arc::new(tokio::sync::Notify::new());
+    let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
+    let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
+    let mut children = tokio::task::JoinSet::new();
+    let task_fault_gate = std::sync::Arc::clone(&fault_gate);
+    let _fault_abort_handle = children.spawn(async move {
+        task_fault_gate.notified().await;
+        return std::result::Result::Err(crate::runtime_error("test.pre_011_ready_source_fault"));
+    });
+    let sibling_active_for_task = std::sync::Arc::clone(&sibling_active);
+    let mut sibling_stop_receiver = source_stop_receiver.clone();
+    let _sibling_abort_handle = children.spawn(async move {
+        let _guard = Pre011SourceActiveGuard::new(sibling_active_for_task);
+        loop {
+            let changed = sibling_stop_receiver.changed().await;
+            if changed.is_err() || *sibling_stop_receiver.borrow() {
+                return std::result::Result::Ok(());
+            }
+        }
+    });
+    let supervisor = tokio::spawn(super::supervise_live_source_tasks(stop_receiver, source_stop_sender, children));
+    for _ in 0..64 {
+        if sibling_active.load(std::sync::atomic::Ordering::Acquire) == 1 {
+            break;
+        }
+        tokio::task::yield_now().await;
+    }
+    assert_eq!(sibling_active.load(std::sync::atomic::Ordering::Acquire), 1);
+    stop_sender.send_replace(true);
+    fault_gate.notify_one();
+    let result = match supervisor.await {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    let error = match result {
+        std::result::Result::Ok(()) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(error.context()[0].value(), "test.pre_011_ready_source_fault");
+    assert_eq!(sibling_active.load(std::sync::atomic::Ordering::Acquire), 0);
+    return;
+}