v0.3.13-pre.007

crates/ksp-worker-raw-transaction-ingest-lib/README.md

@@ -1,5 +1,5 @@
 <!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
-<!-- version: 6 -->
+<!-- version: 7 -->
 
 # ksp-worker-raw-transaction-ingest-lib
 
@@ -12,8 +12,8 @@ RawTransactionIngestWorker::start
     -> fondation source-neutral, sans source productive
 
 RawTransactionIngestWorker::start_with_runtime_resources
-    -> même runtime + une source productive supervisée
-       Yellowstone, WS standard logsSubscribe, WS standard blockSubscribe, Helius transactionSubscribe ou HTTP block polling
+    -> même runtime + 1..32 sources productives supervisées simultanément
+       Yellowstone, WS standard logsSubscribe, WS standard blockSubscribe, Helius transactionSubscribe et/ou HTTP block polling
        + hydration HTTP getTransaction lorsque la source produit une référence
 ```
 
@@ -82,7 +82,11 @@ La construction est sans I/O et refuse notamment :
 - une provenance provider/endpoint non représentable ;
 - l'absence d'une route HTTP compatible pour `getTransaction` sur le même réseau.
 
-Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques. `pre.006` sait exécuter une source unique Yellowstone, Standard Logs, Standard Block, Helius Transaction ou HTTP Block Polling ; une activation simultanée de plusieurs sources reste rejetée avant spawn jusqu'au supervisor dédié. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés.
+Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques et les démarre simultanément sous un supervisor privé. La collection est validée entièrement avant spawn ; aucun sous-ensemble silencieux, source primaire implicite ou standby n'est choisi. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés.
+
+Le supervisor possède toutes les tâches source. Une source qui échoue est terminale pour le Worker et déclenche l'arrêt coopératif puis le join des autres sources, car cette release ne suppose aucune équivalence de coverage. Une fermeture propre d'une source alors que le Worker n'est pas en arrêt est également traitée comme une perte de source configurée et devient terminale.
+
+Chaque source conserve provisoirement ses mécanismes de production/hydration existants. Un inventaire privé `source_key -> latest processing/source state`, borné à 32 entrées, agrège la projection run-local. La frontier agrégée reste conservative : elle n'expose un `processing_frontier_slot` que lorsque toutes les sources en possèdent un, choisit le minimum des frontiers connus et le plus ancien pending. La convergence/coalescence cross-source des mêmes transactions reste une tranche séparée.
 
 ## Contrat de source Standard Logs + HTTP
 
@@ -161,7 +165,7 @@ Le Worker s'exécute sur le runtime Tokio courant du caller. Il ne crée pas de
 - utiliser la même source via `WorkerSnapshotSource` ;
 - attendre le terminal après drain et join des tâches possédées.
 
-Le shutdown est borné par `shutdown_drain_timeout`. Les tâches source, hydration et persistence possédées sont drainées ou abort+join avant publication terminale. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
+Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source relaie le stop à toutes les sources et les rejoint avant de rendre son résultat au supervisor Worker ; les tâches source, hydration et persistence possédées sont ensuite drainées ou abort+join avant publication terminale. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
 
 ## Admission, coalescence et backpressure
 

crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md

@@ -1,5 +1,5 @@
 <!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
-<!-- version: 6 -->
+<!-- version: 7 -->
 
 # Utilisation de ksp-worker-raw-transaction-ingest-lib
 
@@ -101,7 +101,23 @@ let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWo
 };
 ```
 
-Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. Une exécution productive accepte une source unique Yellowstone, Standard Logs, Standard Block, Helius Transaction ou HTTP Block Polling ; une collection de plusieurs sources est validable à la composition mais reste rejetée avant spawn tant que la supervision simultanée n'est pas disponible.
+Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. L'agrégat accepte 1 à 32 sources logiques et les lance simultanément ; les doublons d'identité et les mélanges de réseaux sont refusés avant spawn. Il n'existe pas de source primaire, standby ou fallback implicite : toute source configurée fait partie du run.
+
+### Composer plusieurs sources simultanées
+
+Une fois un premier `RawTransactionIngestRuntimeResources` construit, le caller ajoute les autres sources avec les méthodes `try_push_*` correspondant à leur capability. Exemple conceptuel :
+
+```rust
+let mut resources = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::new(yellowstone_source);
+if let std::result::Result::Err(error) = resources.try_push_standard_logs_source(standard_logs_source) {
+    return std::result::Result::Err(error);
+}
+if let std::result::Result::Err(error) = resources.try_push_http_block_polling_source(http_polling_source) {
+    return std::result::Result::Err(error);
+}
+```
+
+La validation est transactionnelle à chaque ajout : limite globale 32, réseau unique et `source_key` logique unique. Au démarrage, toutes les sources présentes sont supervisées ensemble. La défaillance d'une source est terminale pour le Worker ; les autres sources sont arrêtées et jointes, car le Worker ne suppose pas qu'elles couvrent les mêmes filtres ou le même univers de transactions.
 
 ### Source Standard Logs productive
 

crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
-// version: 13
+// version: 14
 
 type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
 type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
@@ -99,8 +99,8 @@ impl crate::RawTransactionIngestWorker {
 
     /// Starts one Worker with caller-composed runtime resources.
     ///
-    /// `pre.003` activates either the existing Yellowstone source or one standard Solana `logsSubscribe` source through the same bounded Worker foundation.
-    /// Aggregates with more than one source still fail closed before any source task is spawned until the dedicated simultaneous supervisor tranche.
+    /// Every validated source in the bounded runtime-resource aggregate is started concurrently under one private source supervisor. Any configured source
+    /// failure remains terminal for the Worker because this release does not infer equivalent coverage or failover between source families.
     pub fn start_with_runtime_resources(
         settings: crate::RawTransactionIngestSettings,
         store: std::sync::Arc<ksp_store_lib::Store>,
@@ -117,9 +117,6 @@ impl crate::RawTransactionIngestWorker {
         if let std::result::Result::Err(error) = runtime_resources.validate_network(settings.network()) {
             return std::result::Result::Err(error);
         }
-        if let std::result::Result::Err(error) = runtime_resources.validate_single_source_activation() {
-            return std::result::Result::Err(error);
-        }
         let source_settings = settings.clone();
         let (processing_frontier_sender, processing_frontier_receiver) =
             tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
@@ -131,7 +128,7 @@ impl crate::RawTransactionIngestWorker {
             std::option::Option::Some(processing_frontier_receiver),
             move |children, stop_receiver, admission_sender| {
                 let _abort_handle = children.spawn(async move {
-                    return runtime_resources.run_single_live_source(source_settings, stop_receiver, admission_sender, processing_frontier_sender).await;
+                    return runtime_resources.run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender).await;
                 });
             },
         );

crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
-// version: 20
+// version: 21
 
 use sha2::Digest; // rust-rules: trait-import
 
@@ -51,6 +51,19 @@ enum RawTransactionIngestLiveSource {
     Yellowstone(crate::RawTransactionIngestYellowstoneSource),
 }
 
+struct RawTransactionIngestSourceInventory {
+    source_keys: std::vec::Vec<[u8; 32]>,
+    source_projections: std::vec::Vec<crate::RawTransactionIngestProcessingFrontierProjection>,
+}
+
+#[derive(Clone)]
+struct RawTransactionIngestSourceInventoryPublisher {
+    aggregate_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
+    entry_index: usize,
+    inventory: std::sync::Arc<std::sync::Mutex<RawTransactionIngestSourceInventory>>,
+    source_key: [u8; 32],
+}
+
 impl RawTransactionIngestLiveSource {
     fn network(&self) -> &ksp_store_lib::RawNetworkId {
         return match self {
@@ -77,15 +90,148 @@ impl RawTransactionIngestLiveSource {
         settings: crate::RawTransactionIngestSettings,
         stop_receiver: tokio::sync::watch::Receiver<bool>,
         admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
-        processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
+        inventory_publisher: RawTransactionIngestSourceInventoryPublisher,
     ) -> ksp_core_lib::Result<()> {
-        return match self {
-            Self::HeliusTransaction(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
-            Self::HttpBlockPolling(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
-            Self::StandardBlock(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
-            Self::StandardLogs(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
-            Self::Yellowstone(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
+        let (source_frontier_sender, mut source_frontier_receiver) =
+            tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
+        let mut source_future = std::boxed::Box::pin(async move {
+            return match self {
+                Self::HeliusTransaction(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender).await,
+                Self::HttpBlockPolling(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender).await,
+                Self::StandardBlock(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender).await,
+                Self::StandardLogs(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender).await,
+                Self::Yellowstone(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender).await,
+            };
+        });
+        loop {
+            tokio::select! {
+                biased;
+                result = &mut source_future => {
+                    let latest = *source_frontier_receiver.borrow_and_update();
+                    let terminal_state = if result.is_ok() {
+                        crate::RawTransactionIngestSourceState::Closed
+                    } else {
+                        crate::RawTransactionIngestSourceState::Failed
+                    };
+                    inventory_publisher.publish(source_projection_with_state(latest, terminal_state));
+                    return result;
+                }
+                changed = source_frontier_receiver.changed() => {
+                    if changed.is_err() {
+                        return std::result::Result::Err(crate::runtime_error("source.frontier_channel_closed"));
+                    }
+                    inventory_publisher.publish(*source_frontier_receiver.borrow_and_update());
+                }
+            }
+        }
+    }
+}
+
+impl RawTransactionIngestSourceInventory {
+    fn aggregate(&self) -> crate::RawTransactionIngestProcessingFrontierProjection {
+        let mut hydration_pending = 0_usize;
+        let mut oldest_pending_slot = std::option::Option::None;
+        let mut processing_frontier_slot = std::option::Option::None;
+        let mut all_frontiers_present = !self.source_projections.is_empty();
+        let mut source_continuity_gap_total = 0_u64;
+        let mut source_reconnect_total = 0_u64;
+        let mut source_replay_attempt_total = 0_u64;
+        let mut any_active = false;
+        let mut any_closing = false;
+        let mut any_failed = false;
+        let mut any_reconnecting = false;
+        let mut all_closed = !self.source_projections.is_empty();
+        for projection in &self.source_projections {
+            hydration_pending = hydration_pending.saturating_add(projection.hydration_pending());
+            oldest_pending_slot = minimum_optional_slot(oldest_pending_slot, projection.oldest_pending_slot());
+            match projection.processing_frontier_slot() {
+                std::option::Option::Some(slot) => {
+                    processing_frontier_slot = minimum_optional_slot(processing_frontier_slot, std::option::Option::Some(slot));
+                },
+                std::option::Option::None => {
+                    all_frontiers_present = false;
+                },
+            }
+            source_continuity_gap_total = source_continuity_gap_total.saturating_add(projection.source_continuity_gap_total());
+            source_reconnect_total = source_reconnect_total.saturating_add(projection.source_reconnect_total());
+            source_replay_attempt_total = source_replay_attempt_total.saturating_add(projection.source_replay_attempt_total());
+            match projection.source_state() {
+                std::option::Option::Some(crate::RawTransactionIngestSourceState::Active) => {
+                    any_active = true;
+                    all_closed = false;
+                },
+                std::option::Option::Some(crate::RawTransactionIngestSourceState::Closing) => {
+                    any_closing = true;
+                    all_closed = false;
+                },
+                std::option::Option::Some(crate::RawTransactionIngestSourceState::Closed) => {},
+                std::option::Option::Some(crate::RawTransactionIngestSourceState::Failed) => {
+                    any_failed = true;
+                    all_closed = false;
+                },
+                std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting) => {
+                    any_reconnecting = true;
+                    all_closed = false;
+                },
+                std::option::Option::None => {
+                    all_closed = false;
+                },
+            }
+        }
+        if !all_frontiers_present {
+            processing_frontier_slot = std::option::Option::None;
+        }
+        let source_state = if any_failed {
+            std::option::Option::Some(crate::RawTransactionIngestSourceState::Failed)
+        } else if any_reconnecting {
+            std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting)
+        } else if any_closing {
+            std::option::Option::Some(crate::RawTransactionIngestSourceState::Closing)
+        } else if any_active {
+            std::option::Option::Some(crate::RawTransactionIngestSourceState::Active)
+        } else if all_closed {
+            std::option::Option::Some(crate::RawTransactionIngestSourceState::Closed)
+        } else {
+            std::option::Option::None
         };
+        return crate::RawTransactionIngestProcessingFrontierProjection::new(hydration_pending, processing_frontier_slot, oldest_pending_slot)
+            .with_source_continuity(source_state, source_reconnect_total, source_replay_attempt_total, source_continuity_gap_total);
+    }
+
+    fn new(source_keys: std::vec::Vec<[u8; 32]>) -> Self {
+        let source_projections = source_keys.iter().map(|_source_key| return crate::RawTransactionIngestProcessingFrontierProjection::empty()).collect();
+        return Self { source_keys, source_projections };
+    }
+
+    fn update(&mut self, entry_index: usize, source_key: [u8; 32], projection: crate::RawTransactionIngestProcessingFrontierProjection) {
+        let expected_key = match self.source_keys.get(entry_index) {
+            std::option::Option::Some(value) => value,
+            std::option::Option::None => return,
+        };
+        if expected_key != &source_key {
+            return;
+        }
+        let current = match self.source_projections.get_mut(entry_index) {
+            std::option::Option::Some(value) => value,
+            std::option::Option::None => return,
+        };
+        *current = projection;
+        return;
+    }
+}
+
+impl RawTransactionIngestSourceInventoryPublisher {
+    fn publish(&self, projection: crate::RawTransactionIngestProcessingFrontierProjection) {
+        let aggregate = {
+            let mut inventory = match self.inventory.lock() {
+                std::result::Result::Ok(value) => value,
+                std::result::Result::Err(poisoned) => poisoned.into_inner(),
+            };
+            inventory.update(self.entry_index, self.source_key, projection);
+            inventory.aggregate()
+        };
+        self.aggregate_sender.send_replace(aggregate);
+        return;
     }
 }
 
@@ -1610,9 +1756,8 @@ impl std::fmt::Debug for crate::RawTransactionIngestYellowstoneSource {
 
 /// Caller-composed runtime resources accepted by the continuous RAW transaction ingest Worker.
 ///
-/// The aggregate owns a bounded collection of capability-specific live-source contracts. `pre.006` supports Yellowstone, standard Solana logs/block, Helius
-/// transaction and HTTP live block polling sources while deliberately keeping simultaneous multi-source supervision gated until the dedicated supervisor
-/// tranche.
+/// The aggregate owns a bounded collection of capability-specific live-source contracts. All validated sources are started together by one private supervisor;
+/// source identities and per-source lifecycle remain private while the existing Worker snapshot receives only a conservative source-neutral aggregate.
 pub struct RawTransactionIngestRuntimeResources {
     sources: std::vec::Vec<RawTransactionIngestLiveSource>,
 }
@@ -1776,34 +1921,37 @@ impl crate::RawTransactionIngestRuntimeResources {
         return std::result::Result::Ok(());
     }
 
-    /// Validates that the current supervisor tranche can activate the aggregate without silently dropping configured sources.
-    pub(crate) fn validate_single_source_activation(&self) -> ksp_core_lib::Result<()> {
-        if self.sources.is_empty() {
-            return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty"));
-        }
-        if self.sources.len() != 1 {
-            return std::result::Result::Err(crate::runtime_error("runtime_resources.multi_source_activation_pending"));
-        }
-        return std::result::Result::Ok(());
-    }
-
-    /// Runs the already validated single live source through the common Worker source-task contract.
-    pub(crate) async fn run_single_live_source(
+    /// Runs every validated source concurrently under one private bounded source supervisor.
+    pub(crate) async fn run_live_sources(
         self,
         settings: crate::RawTransactionIngestSettings,
         stop_receiver: tokio::sync::watch::Receiver<bool>,
         admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
         processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
     ) -> ksp_core_lib::Result<()> {
-        let mut sources = self.sources.into_iter();
-        let source = match sources.next() {
-            std::option::Option::Some(value) => value,
-            std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty")),
-        };
-        if sources.next().is_some() {
-            return std::result::Result::Err(crate::runtime_error("runtime_resources.multi_source_activation_pending"));
+        if self.sources.is_empty() || self.sources.len() > crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES {
+            return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_invalid"));
         }
-        return source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await;
+        let source_keys = self.sources.iter().map(RawTransactionIngestLiveSource::source_key).collect::<std::vec::Vec<_>>();
+        let inventory = std::sync::Arc::new(std::sync::Mutex::new(RawTransactionIngestSourceInventory::new(source_keys)));
+        let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
+        let mut children = tokio::task::JoinSet::new();
+        for (entry_index, source) in self.sources.into_iter().enumerate() {
+            let publisher = RawTransactionIngestSourceInventoryPublisher {
+                aggregate_sender: processing_frontier_sender.clone(),
+                entry_index,
+                inventory: std::sync::Arc::clone(&inventory),
+                source_key: source.source_key(),
+            };
+            let source_admission_sender = admission_sender.clone();
+            let source_settings = settings.clone();
+            let source_stop_receiver = source_stop_receiver.clone();
+            let _abort_handle = children.spawn(async move {
+                return source.run(source_settings, source_stop_receiver, source_admission_sender, publisher).await;
+            });
+        }
+        std::mem::drop(admission_sender);
+        return supervise_live_source_tasks(stop_receiver, source_stop_sender, children).await;
     }
 }
 
@@ -1828,6 +1976,87 @@ impl std::hash::Hasher for RawTransactionIngestSourceKeyHashWriter<'_> {
     }
 }
 
+fn minimum_optional_slot(first: std::option::Option<u64>, second: std::option::Option<u64>) -> std::option::Option<u64> {
+    return match (first, second) {
+        (std::option::Option::Some(left), std::option::Option::Some(right)) => std::option::Option::Some(left.min(right)),
+        (std::option::Option::Some(value), std::option::Option::None) | (std::option::Option::None, std::option::Option::Some(value)) => {
+            std::option::Option::Some(value)
+        },
+        (std::option::Option::None, std::option::Option::None) => std::option::Option::None,
+    };
+}
+
+fn source_projection_with_state(
+    projection: crate::RawTransactionIngestProcessingFrontierProjection,
+    state: crate::RawTransactionIngestSourceState,
+) -> crate::RawTransactionIngestProcessingFrontierProjection {
+    return crate::RawTransactionIngestProcessingFrontierProjection::new(
+        projection.hydration_pending(),
+        projection.processing_frontier_slot(),
+        projection.oldest_pending_slot(),
+    )
+    .with_source_continuity(
+        std::option::Option::Some(state),
+        projection.source_reconnect_total(),
+        projection.source_replay_attempt_total(),
+        projection.source_continuity_gap_total(),
+    );
+}
+
+async fn supervise_live_source_tasks(
+    mut stop_receiver: tokio::sync::watch::Receiver<bool>,
+    source_stop_sender: tokio::sync::watch::Sender<bool>,
+    mut children: tokio::task::JoinSet<ksp_core_lib::Result<()>>,
+) -> ksp_core_lib::Result<()> {
+    loop {
+        if *stop_receiver.borrow() {
+            source_stop_sender.send_replace(true);
+            return drain_live_source_tasks(&mut children, std::option::Option::None).await;
+        }
+        let joined = tokio::select! {
+            biased;
+            changed = stop_receiver.changed() => {
+                if changed.is_err() || *stop_receiver.borrow() {
+                    source_stop_sender.send_replace(true);
+                    return drain_live_source_tasks(&mut children, std::option::Option::None).await;
+                }
+                continue;
+            }
+            value = children.join_next(), if !children.is_empty() => value,
+        };
+        let first_fault = match joined {
+            std::option::Option::Some(std::result::Result::Ok(std::result::Result::Ok(()))) => {
+                std::option::Option::Some(crate::runtime_error("source.configured_source_closed"))
+            },
+            std::option::Option::Some(std::result::Result::Ok(std::result::Result::Err(error))) => std::option::Option::Some(error),
+            std::option::Option::Some(std::result::Result::Err(_)) => std::option::Option::Some(crate::runtime_error("source.task_join_failed")),
+            std::option::Option::None => std::option::Option::Some(crate::runtime_error("source.task_set_empty")),
+        };
+        source_stop_sender.send_replace(true);
+        return drain_live_source_tasks(&mut children, first_fault).await;
+    }
+}
+
+async fn drain_live_source_tasks(
+    children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>,
+    mut first_fault: std::option::Option<ksp_core_lib::Error>,
+) -> ksp_core_lib::Result<()> {
+    while let std::option::Option::Some(joined) = children.join_next().await {
+        if first_fault.is_some() {
+            continue;
+        }
+        first_fault = match joined {
+            std::result::Result::Ok(std::result::Result::Ok(())) => std::option::Option::None,
+            std::result::Result::Ok(std::result::Result::Err(error)) => std::option::Option::Some(error),
+            std::result::Result::Err(_) => std::option::Option::Some(crate::runtime_error("source.task_join_failed")),
+        };
+    }
+    return match first_fault {
+        std::option::Option::Some(error) => std::result::Result::Err(error),
+        std::option::Option::None => std::result::Result::Ok(()),
+    };
+}
+
 fn http_block_polling_live_source_key(
     network: &ksp_store_lib::RawNetworkId,
     polling_role: &ksp_onchain_transport_lib::HttpRoleName,

crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
-// version: 22
+// version: 23
 
 //! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
 
@@ -158,11 +158,7 @@ fn v0_3_12_pre_005_block_and_continuity_adapters_remain_private_and_transport_fa
 fn v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission() {
     let runtime = include_str!("../src/runtime.rs");
     let resources = include_str!("../src/runtime_resources.rs");
-    for required in [
-        "runtime_resources.validate_single_source_activation()",
-        "children.spawn",
-        ".run_single_live_source(source_settings, stop_receiver, admission_sender, processing_frontier_sender)",
-    ] {
+    for required in ["children.spawn", ".run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"] {
         assert!(runtime.contains(required), "required pre.006 supervisor wiring missing: {required}");
     }
     for required in [
@@ -260,7 +256,7 @@ fn v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydr
     ] {
         assert!(resources.contains(required), "required pre.003 standard logs contract missing: {required}");
     }
-    assert!(runtime.contains("run_single_live_source"));
+    assert!(runtime.contains("run_live_sources"));
     for forbidden in [
         "ksp_config_lib::",
         "ksp_job_backfill_lib::",
@@ -468,6 +464,46 @@ fn manifest_dependency_names(section: &str) -> std::vec::Vec<&str> {
     return names;
 }
 
+#[test]
+fn v0_3_13_pre_007_multi_source_supervisor_and_inventory_remain_private_bounded_and_source_neutral() {
+    let root = include_str!("../src/lib.rs");
+    let runtime = include_str!("../src/runtime.rs");
+    let resources = include_str!("../src/runtime_resources.rs");
+    for required in [
+        "RawTransactionIngestSourceInventory",
+        "RawTransactionIngestSourceInventoryPublisher",
+        "for (entry_index, source) in self.sources.into_iter().enumerate()",
+        "tokio::task::JoinSet::new()",
+        "supervise_live_source_tasks",
+        "drain_live_source_tasks",
+        "source_stop_sender.send_replace(true)",
+        "processing_frontier_sender.clone()",
+        "source.configured_source_closed",
+    ] {
+        assert!(resources.contains(required), "required pre.007 multi-source supervisor contract missing: {required}");
+    }
+    assert!(runtime.contains("run_live_sources"));
+    assert!(!runtime.contains("validate_single_source_activation"));
+    assert!(!resources.contains("multi_source_activation_pending"));
+    for forbidden in [
+        "pub struct RawTransactionIngestSourceInventory",
+        "pub struct RawTransactionIngestSourceInventoryPublisher",
+        "unbounded_channel",
+        "ksp_config_lib::",
+        "ksp_job_backfill_lib::",
+        "ksp_store_postgres_lib::",
+        "reqwest::",
+        "tonic::",
+        "yellowstone_grpc_proto::",
+    ] {
+        assert!(
+            !root.contains(forbidden) && !runtime.contains(forbidden) && !resources.contains(forbidden),
+            "pre.007 crossed a private/bounded boundary: {forbidden}"
+        );
+    }
+    return;
+}
+
 #[test]
 fn v0_3_12_pre_007_processing_frontier_remains_run_local_and_backend_neutral_after_continuity_extension() {
     let runtime = include_str!("../src/runtime.rs");

crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
-// version: 16
+// version: 17
 
 //! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
 
@@ -397,7 +397,7 @@ fn v0_3_12_pre_006_runtime_resource_contract_opens_one_supervised_transport_sour
         assert!(resources.contains(required), "productive runtime-resource source behavior missing: {required}");
     }
     assert!(runtime.contains("start_with_runtime_resources"));
-    assert!(runtime.contains("run_single_live_source(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"));
+    assert!(runtime.contains("run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"));
     for forbidden in ["ksp_config_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
         assert!(!resources.contains(forbidden) && !runtime.contains(forbidden), "pre.006 runtime source crossed a forbidden boundary: {forbidden}");
     }
@@ -718,3 +718,36 @@ fn v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit(
     assert!(!resources.contains("unbounded_channel"), "Worker introduced an unbounded channel");
     return;
 }
+
+#[test]
+fn v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity() {
+    let root = include_str!("../src/lib.rs");
+    let resources = include_str!("../src/runtime_resources.rs");
+    for required in [
+        "MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES",
+        "RawTransactionIngestSourceInventory",
+        "std::sync::Mutex",
+        "tokio::task::JoinSet",
+        "source.configured_source_closed",
+        "source.task_join_failed",
+        "source_stop_sender.send_replace(true)",
+        "drain_live_source_tasks",
+        "processing_frontier_slot = std::option::Option::None",
+        "saturating_add",
+    ] {
+        assert!(resources.contains(required), "required pre.007 supervisor hardening guard missing: {required}");
+    }
+    for forbidden in [
+        "pub struct RawTransactionIngestSourceInventory",
+        "pub struct RawTransactionIngestSourceInventoryPublisher",
+        "pub fn source_key(",
+        "source_keys:",
+        "unbounded_channel",
+        "primary_source",
+        "standby_source",
+        "first_provider_wins",
+    ] {
+        assert!(!root.contains(forbidden), "private source identity/scheduling policy leaked into public root: {forbidden}");
+    }
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
-// version: 16
+// version: 17
 
 //! External public-surface proofs for the RAW transaction ingest Worker foundation.
 
@@ -327,3 +327,12 @@ fn v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processin
     }
     return;
 }
+
+#[test]
+fn v0_3_13_pre_007_source_inventory_and_logical_keys_remain_private() {
+    let root = include_str!("../src/lib.rs");
+    for forbidden in ["RawTransactionIngestSourceInventory", "RawTransactionIngestSourceInventoryPublisher", "RawTransactionIngestLiveSource", "source_key"] {
+        assert!(!root.contains(forbidden), "pre.007 private multi-source implementation leaked through crate root: {forbidden}");
+    }
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
-// version: 14
+// version: 15
 
 //! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
 
@@ -127,6 +127,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
         "v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit",
         "v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit",
         "v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted",
+        "v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity",
     ] {
         assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
     }
@@ -144,6 +145,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
     assert!(dependency_boundary.contains("v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only"));
     assert!(dependency_boundary.contains("v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only"));
     assert!(dependency_boundary.contains("v0_3_13_pre_006_http_block_polling_reuses_transport_facades_without_becoming_backfill"));
+    assert!(dependency_boundary.contains("v0_3_13_pre_007_multi_source_supervisor_and_inventory_remain_private_bounded_and_source_neutral"));
     let public_api = include_str!("public_api.rs");
     assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
     assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
@@ -154,6 +156,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
     assert!(public_api.contains("v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport_owned"));
     assert!(public_api.contains("v0_3_13_pre_004_standard_block_runtime_resource_surface_is_typed_and_transport_owned"));
     assert!(public_api.contains("v0_3_13_pre_006_http_block_polling_runtime_resource_surface_is_bounded_and_transport_owned"));
+    assert!(public_api.contains("v0_3_13_pre_007_source_inventory_and_logical_keys_remain_private"));
     assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
     assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
     return;

crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
-// version: 18
+// version: 19
 
 fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
     return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
@@ -959,24 +959,118 @@ async fn v0_3_13_pre_002_runtime_resources_reject_duplicate_and_cross_network_so
     return;
 }
 
+#[test]
+fn v0_3_13_pre_007_source_inventory_aggregates_frontier_and_lifecycle_conservatively() {
+    let source_keys = std::vec![[1_u8; 32], [2_u8; 32]];
+    let inventory = std::sync::Arc::new(std::sync::Mutex::new(super::RawTransactionIngestSourceInventory::new(source_keys)));
+    let (aggregate_sender, aggregate_receiver) = tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
+    let first = super::RawTransactionIngestSourceInventoryPublisher {
+        aggregate_sender: aggregate_sender.clone(),
+        entry_index: 0,
+        inventory: std::sync::Arc::clone(&inventory),
+        source_key: [1_u8; 32],
+    };
+    let second = super::RawTransactionIngestSourceInventoryPublisher { aggregate_sender, entry_index: 1, inventory, source_key: [2_u8; 32] };
+    first.publish(
+        crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(50), std::option::Option::Some(45)).with_source_continuity(
+            std::option::Option::Some(crate::RawTransactionIngestSourceState::Active),
+            1,
+            2,
+            0,
+        ),
+    );
+    let first_aggregate = *aggregate_receiver.borrow();
+    assert_eq!(first_aggregate.hydration_pending(), 1);
+    assert_eq!(first_aggregate.processing_frontier_slot(), std::option::Option::None);
+    assert_eq!(first_aggregate.oldest_pending_slot(), std::option::Option::Some(45));
+    assert_eq!(first_aggregate.source_state(), std::option::Option::Some(crate::RawTransactionIngestSourceState::Active));
+    second.publish(
+        crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(42), std::option::Option::Some(40)).with_source_continuity(
+            std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting),
+            3,
+            4,
+            1,
+        ),
+    );
+    let aggregate = *aggregate_receiver.borrow();
+    assert_eq!(aggregate.hydration_pending(), 3);
+    assert_eq!(aggregate.processing_frontier_slot(), std::option::Option::Some(42));
+    assert_eq!(aggregate.oldest_pending_slot(), std::option::Option::Some(40));
+    assert_eq!(aggregate.source_state(), std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting));
+    assert_eq!(aggregate.source_reconnect_total(), 4);
+    assert_eq!(aggregate.source_replay_attempt_total(), 6);
+    assert_eq!(aggregate.source_continuity_gap_total(), 1);
+    return;
+}
+
 #[tokio::test(flavor = "current_thread")]
-async fn v0_3_13_pre_002_multi_source_activation_fails_closed_until_supervisor_tranche() {
-    let first = match source_with_identity("devnet", "source-a", "tx", std::option::Option::None) {
-        std::option::Option::Some(value) => value,
-        std::option::Option::None => return,
+async fn v0_3_13_pre_007_source_supervisor_joins_all_children_on_stop() {
+    let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
+    let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
+    let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
+    let mut children = tokio::task::JoinSet::new();
+    for _ in 0..3 {
+        let active = std::sync::Arc::clone(&active);
+        let mut source_stop_receiver = source_stop_receiver.clone();
+        let _abort_handle = children.spawn(async move {
+            active.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
+            loop {
+                let changed = source_stop_receiver.changed().await;
+                if changed.is_err() || *source_stop_receiver.borrow() {
+                    active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
+                    return std::result::Result::Ok(());
+                }
+            }
+        });
+    }
+    let supervisor = tokio::spawn(super::supervise_live_source_tasks(stop_receiver, source_stop_sender, children));
+    for _ in 0..64 {
+        if active.load(std::sync::atomic::Ordering::Acquire) == 3 {
+            break;
+        }
+        tokio::task::yield_now().await;
+    }
+    assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 3);
+    stop_sender.send_replace(true);
+    let joined = supervisor.await;
+    let result = match joined {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
     };
-    let second = match source_with_identity("devnet", "source-b", "tx", std::option::Option::None) {
-        std::option::Option::Some(value) => value,
-        std::option::Option::None => return,
-    };
-    let mut resources = crate::RawTransactionIngestRuntimeResources::new(first);
-    assert!(resources.try_push_yellowstone_source(second).is_ok());
-    let error = match resources.validate_single_source_activation() {
-        std::result::Result::Ok(()) => return,
-        std::result::Result::Err(error) => error,
-    };
-    assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
-    assert!(error.context().iter().any(|context| return context.value() == "runtime_resources.multi_source_activation_pending"));
+    assert!(result.is_ok());
+    assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
+    return;
+}
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_007_source_failure_stops_and_joins_sibling_sources() {
+    let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(2));
+    let sibling_active = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
+    let (_stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
+    let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
+    let mut children = tokio::task::JoinSet::new();
+    let failing_barrier = std::sync::Arc::clone(&barrier);
+    let _failing_abort_handle = children.spawn(async move {
+        let _barrier_wait = failing_barrier.wait().await;
+        return std::result::Result::Err(crate::runtime_error("test.pre_007_source_failed"));
+    });
+    let sibling_barrier = std::sync::Arc::clone(&barrier);
+    let sibling_active_for_task = std::sync::Arc::clone(&sibling_active);
+    let mut sibling_stop_receiver = source_stop_receiver.clone();
+    let _sibling_abort_handle = children.spawn(async move {
+        sibling_active_for_task.store(true, std::sync::atomic::Ordering::Release);
+        let _barrier_wait = sibling_barrier.wait().await;
+        loop {
+            let changed = sibling_stop_receiver.changed().await;
+            if changed.is_err() || *sibling_stop_receiver.borrow() {
+                sibling_active_for_task.store(false, std::sync::atomic::Ordering::Release);
+                return std::result::Result::Ok(());
+            }
+        }
+    });
+    let result = super::supervise_live_source_tasks(stop_receiver, source_stop_sender, children).await;
+    assert!(result.is_err());
+    assert!(!sibling_active.load(std::sync::atomic::Ordering::Acquire));
     return;
 }