v0.3.13-pre.009

crates/ksp-worker-raw-transaction-ingest-lib/README.md

@@ -1,5 +1,5 @@
 <!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
-<!-- version: 8 -->
+<!-- version: 9 -->
 
 # ksp-worker-raw-transaction-ingest-lib
 
@@ -171,14 +171,19 @@ Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source
 
 La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes subissent la backpressure ; aucune queue non bornée ni silent drop n'est autorisé.
 
-Le Worker possède des coordinateurs source-neutral Yellowstone, Standard Logs et Helius Transaction branchés sur un registre global d'hydration partagé. Standard Block et HTTP Block Polling n'entrent pas dans ce registre lorsqu'une transaction est direct-qualified :
+Le Worker possède des coordinateurs source-neutral Yellowstone, Standard Logs et Helius Transaction branchés sur un registre global d'hydration partagé. Standard Block et HTTP Block Polling n'entrent pas dans ce registre lorsqu'une transaction est direct-qualified.
+
+Les sources reference-bearing reçoivent des quotas déterministes dont la somme reste exactement dans les bornes techniques configurées :
 
 ```text
-in-flight hydration <= persistence_concurrency
-pending source signals <= admission_queue_capacity
+somme pending source signals <= admission_queue_capacity
+somme hydration tasks in flight <= persistence_concurrency
+source reference-bearing active => quota pending >= 1 et quota in-flight >= 1
 ```
 
-Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu identique ajoutent uniquement leur observation déterministe. Un hash canonique divergent devient un content conflict terminal.
+Pour garantir simultanément ces bornes et l'absence de starvation structurelle, le démarrage échoue avant spawn si le nombre de sources reference-bearing dépasse `admission_queue_capacity` ou `persistence_concurrency`. Un sémaphore global protège en plus l'ouverture effective des hydrations HTTP.
+
+Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu canonique identique ajoutent uniquement leur observation déterministe. Une divergence de slot, block time, format ou hash canonique devient un content conflict terminal ; aucune majorité, préférence provider ou overwrite n'est appliqué. Le Store conserve son guard durable final.
 
 Les retries/reroutages HTTP appartiennent à `ksp-onchain-transport-lib`. Le Worker ne possède pas une seconde boucle de retry autour de `getTransaction`.
 

crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md

@@ -1,5 +1,5 @@
 <!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
-<!-- version: 8 -->
+<!-- version: 9 -->
 
 # Utilisation de ksp-worker-raw-transaction-ingest-lib
 
@@ -255,6 +255,19 @@ Yellowstone Account/Ping/Pong/Entry -> sans RAW Transaction dans cette verticale
 
 Les signaux de même `(network, signature, commitment)` sont coalescés globalement avant l'hydration HTTP. Si plusieurs sources produisent ensuite la même transaction canonique, le Worker conserve une seule entité RAW et enregistre séparément les observations déterministes propres à chaque source. Le Worker ne possède pas une boucle de retry HTTP : reroutage/retry/backoff restent dans `ksp-onchain-transport-lib`.
 
+Les sources qui nécessitent `getTransaction` partagent des quotas bornés et déterministes. Pour une composition valide :
+
+```text
+nombre de sources avec hydration <= admission_queue_capacity
+nombre de sources avec hydration <= persistence_concurrency
+somme de leurs pending <= admission_queue_capacity
+somme de leurs tâches d'hydration actives <= persistence_concurrency
+```
+
+Ces contraintes garantissent au moins une part à chaque source reference-bearing sans introduire de scheduler pondéré. Si les settings ne permettent pas cette répartition, le démarrage échoue avant spawn avec `runtime_invalid`; le caller doit augmenter la capacité concernée ou réduire le nombre de sources nécessitant une hydration.
+
+Pour une même identité `(network, signature)`, une divergence canonique de slot, block time, format ou hash est un `content_conflict`. Le Worker ne choisit ni majorité ni provider préféré et n'écrase pas un contenu divergent ; le Store reste l'autorité durable finale du conflit.
+
 ## Observer le snapshot concret
 
 ```rust

crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
-// version: 3
+// version: 4
 
 /// Canonical entity disposition produced by one Worker Store persistence attempt.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
@@ -92,11 +92,23 @@ struct RawTransactionIngestPersistenceKey {
     signature: ksp_store_lib::RawTransactionSignature,
 }
 
+#[derive(Clone, Eq, PartialEq)]
+struct RawTransactionIngestCanonicalState {
+    block_time: std::option::Option<ksp_store_lib::RawTimestamp>,
+    content_hash: ksp_store_lib::RawContentHash,
+    format_id: ksp_store_lib::RawFormatId,
+    format_version: u32,
+    slot: u64,
+}
+
 /// Private bounded run-local cache serializing repeated canonical identities before Store writes.
 pub(crate) struct RawTransactionIngestPersistenceConvergence {
     max_entries: usize,
     entries: std::sync::Mutex<
-        std::collections::BTreeMap<RawTransactionIngestPersistenceKey, std::sync::Arc<tokio::sync::Mutex<std::option::Option<ksp_store_lib::RawContentHash>>>>,
+        std::collections::BTreeMap<
+            RawTransactionIngestPersistenceKey,
+            std::sync::Arc<tokio::sync::Mutex<std::option::Option<RawTransactionIngestCanonicalState>>>,
+        >,
     >,
 }
 
@@ -110,7 +122,7 @@ impl crate::RawTransactionIngestPersistenceConvergence {
     fn entry(
         &self,
         key: RawTransactionIngestPersistenceKey,
-    ) -> std::option::Option<std::sync::Arc<tokio::sync::Mutex<std::option::Option<ksp_store_lib::RawContentHash>>>> {
+    ) -> std::option::Option<std::sync::Arc<tokio::sync::Mutex<std::option::Option<RawTransactionIngestCanonicalState>>>> {
         let mut entries = match self.entries.lock() {
             std::result::Result::Ok(value) => value,
             std::result::Result::Err(poisoned) => poisoned.into_inner(),
@@ -177,16 +189,16 @@ where
     if acquisition.observation().transaction() != &reference {
         return std::result::Result::Err(crate::runtime_error("persistence.acquisition_reference_mismatch"));
     }
-    let content_hash = acquisition.transaction().payload().content_hash();
+    let canonical_state = canonical_state(acquisition.transaction());
     let key = RawTransactionIngestPersistenceKey { network: reference.network().clone(), signature: reference.signature() };
     let entry = convergence.entry(key);
     let entry = match entry {
         std::option::Option::Some(value) => value,
         std::option::Option::None => return crate::persist_raw_transaction_ingest_acquisition(port, acquisition).await,
     };
-    let mut known_hash = entry.lock().await;
-    if let std::option::Option::Some(known_hash_value) = *known_hash {
-        if known_hash_value != content_hash {
+    let mut known_state = entry.lock().await;
+    if let std::option::Option::Some(known_state_value) = known_state.as_ref() {
+        if known_state_value != &canonical_state {
             return std::result::Result::Err(crate::content_conflict_error());
         }
         let (_transaction, observation) = acquisition.into_parts();
@@ -202,11 +214,21 @@ where
         std::result::Result::Err(error) => return std::result::Result::Err(error),
     };
     if outcome.entity() != crate::RawTransactionIngestEntityPersistence::SkippedPurged {
-        *known_hash = std::option::Option::Some(content_hash);
+        *known_state = std::option::Option::Some(canonical_state);
     }
     return std::result::Result::Ok(outcome);
 }
 
+fn canonical_state(transaction: &ksp_store_lib::RawTransaction) -> RawTransactionIngestCanonicalState {
+    return RawTransactionIngestCanonicalState {
+        block_time: transaction.block_time(),
+        content_hash: transaction.payload().content_hash(),
+        format_id: transaction.payload().format_id().clone(),
+        format_version: transaction.payload().format_version(),
+        slot: transaction.slot(),
+    };
+}
+
 fn map_store_error(error: ksp_core_lib::Error) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
     if error.code() == ksp_store_lib::ERROR_CODE_RAW_CONFLICT {
         return std::result::Result::Err(crate::content_conflict_error());

crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
-// version: 23
+// version: 24
 
 use sha2::Digest; // rust-rules: trait-import
 
@@ -85,12 +85,18 @@ impl RawTransactionIngestLiveSource {
         };
     }
 
+    fn uses_hydration(&self) -> bool {
+        return matches!(self, Self::HeliusTransaction(_) | Self::StandardLogs(_) | Self::Yellowstone(_));
+    }
+
     async fn run(
         self,
         settings: crate::RawTransactionIngestSettings,
         stop_receiver: tokio::sync::watch::Receiver<bool>,
         admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
         global_hydration_registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>,
+        hydration_pending_limit: usize,
+        hydration_in_flight_limit: usize,
         inventory_publisher: RawTransactionIngestSourceInventoryPublisher,
     ) -> ksp_core_lib::Result<()> {
         let (source_frontier_sender, mut source_frontier_receiver) =
@@ -98,12 +104,46 @@ impl RawTransactionIngestLiveSource {
         let mut source_future = std::boxed::Box::pin(async move {
             return match self {
                 Self::HeliusTransaction(source) => {
-                    source.run(settings, stop_receiver, admission_sender, source_frontier_sender, global_hydration_registry).await
+                    source
+                        .run(
+                            settings,
+                            stop_receiver,
+                            admission_sender,
+                            source_frontier_sender,
+                            global_hydration_registry,
+                            hydration_pending_limit,
+                            hydration_in_flight_limit,
+                        )
+                        .await
                 },
                 Self::HttpBlockPolling(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender).await,
                 Self::StandardBlock(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender).await,
-                Self::StandardLogs(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender, global_hydration_registry).await,
-                Self::Yellowstone(source) => source.run(settings, stop_receiver, admission_sender, source_frontier_sender, global_hydration_registry).await,
+                Self::StandardLogs(source) => {
+                    source
+                        .run(
+                            settings,
+                            stop_receiver,
+                            admission_sender,
+                            source_frontier_sender,
+                            global_hydration_registry,
+                            hydration_pending_limit,
+                            hydration_in_flight_limit,
+                        )
+                        .await
+                },
+                Self::Yellowstone(source) => {
+                    source
+                        .run(
+                            settings,
+                            stop_receiver,
+                            admission_sender,
+                            source_frontier_sender,
+                            global_hydration_registry,
+                            hydration_pending_limit,
+                            hydration_in_flight_limit,
+                        )
+                        .await
+                },
             };
         });
         loop {
@@ -628,6 +668,8 @@ impl crate::RawTransactionIngestYellowstoneSource {
         admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
         processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
         global_hydration_registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>,
+        hydration_pending_limit: usize,
+        hydration_in_flight_limit: usize,
     ) -> ksp_core_lib::Result<()> {
         let opened = tokio::select! {
             biased;
@@ -641,7 +683,8 @@ impl crate::RawTransactionIngestYellowstoneSource {
             std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())),
         };
         let hydration = self.hydration_context();
-        let mut coordinator = RawTransactionIngestHydrationCoordinator::with_global_registry(&settings, global_hydration_registry);
+        let mut coordinator =
+            RawTransactionIngestHydrationCoordinator::with_global_registry(global_hydration_registry, hydration_pending_limit, hydration_in_flight_limit);
         let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
         let snapshot_source = session.snapshot_source();
         if let std::result::Result::Err(error) = processing_frontier.observe_session_snapshot(snapshot_source.current()) {
@@ -865,6 +908,8 @@ impl crate::RawTransactionIngestHeliusTransactionSource {
         admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
         processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
         global_hydration_registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>,
+        hydration_pending_limit: usize,
+        hydration_in_flight_limit: usize,
     ) -> ksp_core_lib::Result<()> {
         let connected = tokio::select! {
             biased;
@@ -894,7 +939,8 @@ impl crate::RawTransactionIngestHeliusTransactionSource {
             },
         };
         let hydration = self.hydration_context();
-        let mut coordinator = RawTransactionIngestHydrationCoordinator::with_global_registry(&settings, global_hydration_registry);
+        let mut coordinator =
+            RawTransactionIngestHydrationCoordinator::with_global_registry(global_hydration_registry, hydration_pending_limit, hydration_in_flight_limit);
         let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
         processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Active);
         let mut fault = std::option::Option::None;
@@ -1584,6 +1630,8 @@ impl crate::RawTransactionIngestStandardLogsSource {
         admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
         processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
         global_hydration_registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>,
+        hydration_pending_limit: usize,
+        hydration_in_flight_limit: usize,
     ) -> ksp_core_lib::Result<()> {
         let connected = tokio::select! {
             biased;
@@ -1613,7 +1661,8 @@ impl crate::RawTransactionIngestStandardLogsSource {
             },
         };
         let hydration = self.hydration_context();
-        let mut coordinator = RawTransactionIngestHydrationCoordinator::with_global_registry(&settings, global_hydration_registry);
+        let mut coordinator =
+            RawTransactionIngestHydrationCoordinator::with_global_registry(global_hydration_registry, hydration_pending_limit, hydration_in_flight_limit);
         let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
         processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Active);
         let mut fault = std::option::Option::None;
@@ -1939,11 +1988,29 @@ impl crate::RawTransactionIngestRuntimeResources {
             return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_invalid"));
         }
         let source_keys = self.sources.iter().map(RawTransactionIngestLiveSource::source_key).collect::<std::vec::Vec<_>>();
+        let hydration_source_count = self.sources.iter().filter(|source| return source.uses_hydration()).count();
+        if let std::result::Result::Err(error) =
+            validate_hydration_fairness_capacity(settings.admission_queue_capacity(), settings.persistence_concurrency(), hydration_source_count)
+        {
+            return std::result::Result::Err(error);
+        }
+        let hydration_pending_budget = settings.admission_queue_capacity();
+        let hydration_in_flight_budget = settings.persistence_concurrency();
         let inventory = std::sync::Arc::new(std::sync::Mutex::new(RawTransactionIngestSourceInventory::new(source_keys)));
-        let global_hydration_registry = std::sync::Arc::new(RawTransactionIngestGlobalHydrationRegistry::new(&settings));
+        let global_hydration_registry =
+            std::sync::Arc::new(RawTransactionIngestGlobalHydrationRegistry::new(hydration_pending_budget, hydration_in_flight_budget));
         let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
         let mut children = tokio::task::JoinSet::new();
+        let mut hydration_entry_index = 0_usize;
         for (entry_index, source) in self.sources.into_iter().enumerate() {
+            let (hydration_pending_limit, hydration_in_flight_limit) = if source.uses_hydration() {
+                let pending_limit = hydration_pending_limit(hydration_pending_budget, hydration_source_count, hydration_entry_index);
+                let in_flight_limit = hydration_in_flight_limit(hydration_in_flight_budget, hydration_source_count, hydration_entry_index);
+                hydration_entry_index += 1;
+                (pending_limit, in_flight_limit)
+            } else {
+                (0, 0)
+            };
             let publisher = RawTransactionIngestSourceInventoryPublisher {
                 aggregate_sender: processing_frontier_sender.clone(),
                 entry_index,
@@ -1955,7 +2022,17 @@ impl crate::RawTransactionIngestRuntimeResources {
             let source_stop_receiver = source_stop_receiver.clone();
             let source_global_hydration_registry = std::sync::Arc::clone(&global_hydration_registry);
             let _abort_handle = children.spawn(async move {
-                return source.run(source_settings, source_stop_receiver, source_admission_sender, source_global_hydration_registry, publisher).await;
+                return source
+                    .run(
+                        source_settings,
+                        source_stop_receiver,
+                        source_admission_sender,
+                        source_global_hydration_registry,
+                        hydration_pending_limit,
+                        hydration_in_flight_limit,
+                        publisher,
+                    )
+                    .await;
             });
         }
         std::mem::drop(admission_sender);
@@ -1963,6 +2040,33 @@ impl crate::RawTransactionIngestRuntimeResources {
     }
 }
 
+fn validate_hydration_fairness_capacity(
+    admission_queue_capacity: usize,
+    persistence_concurrency: usize,
+    hydration_source_count: usize,
+) -> ksp_core_lib::Result<()> {
+    if hydration_source_count > admission_queue_capacity {
+        return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_pending_capacity_insufficient"));
+    }
+    if hydration_source_count > persistence_concurrency {
+        return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_concurrency_insufficient"));
+    }
+    return std::result::Result::Ok(());
+}
+
+fn hydration_pending_limit(total_budget: usize, hydration_source_count: usize, hydration_entry_index: usize) -> usize {
+    if hydration_source_count == 0 || hydration_entry_index >= hydration_source_count {
+        return 0;
+    }
+    let base = total_budget / hydration_source_count;
+    let remainder = total_budget % hydration_source_count;
+    return base + usize::from(hydration_entry_index < remainder);
+}
+
+fn hydration_in_flight_limit(total_budget: usize, hydration_source_count: usize, hydration_entry_index: usize) -> usize {
+    return hydration_pending_limit(total_budget, hydration_source_count, hydration_entry_index);
+}
+
 impl std::fmt::Debug for crate::RawTransactionIngestRuntimeResources {
     fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         return formatter.debug_struct("RawTransactionIngestRuntimeResources").field("source_count", &self.sources.len()).finish();
@@ -3351,6 +3455,7 @@ enum RawTransactionIngestSharedHydrationResult {
 }
 
 struct RawTransactionIngestGlobalHydrationRegistry {
+    hydration_permits: std::sync::Arc<tokio::sync::Semaphore>,
     max_pending: usize,
     pending: std::sync::Mutex<
         std::collections::BTreeMap<
@@ -3361,9 +3466,10 @@ struct RawTransactionIngestGlobalHydrationRegistry {
 }
 
 impl RawTransactionIngestGlobalHydrationRegistry {
-    fn new(settings: &crate::RawTransactionIngestSettings) -> Self {
+    fn new(max_pending: usize, max_in_flight: usize) -> Self {
         return Self {
-            max_pending: settings.admission_queue_capacity(),
+            hydration_permits: std::sync::Arc::new(tokio::sync::Semaphore::new(max_in_flight)),
+            max_pending,
             pending: std::sync::Mutex::new(std::collections::BTreeMap::new()),
         };
     }
@@ -3416,17 +3522,21 @@ struct RawTransactionIngestHydrationCoordinator {
 impl RawTransactionIngestHydrationCoordinator {
     #[cfg(test)]
     fn new(settings: &crate::RawTransactionIngestSettings) -> Self {
-        return Self::with_global_registry(settings, std::sync::Arc::new(RawTransactionIngestGlobalHydrationRegistry::new(settings)));
+        let pending_limit = settings.admission_queue_capacity();
+        let in_flight_limit = settings.persistence_concurrency().min(pending_limit);
+        let registry = std::sync::Arc::new(RawTransactionIngestGlobalHydrationRegistry::new(pending_limit, settings.persistence_concurrency()));
+        return Self::with_global_registry(registry, pending_limit, in_flight_limit);
     }
 
     fn with_global_registry(
-        settings: &crate::RawTransactionIngestSettings,
         global_registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>,
+        max_pending_signals: usize,
+        max_in_flight: usize,
     ) -> Self {
         return Self {
             global_registry,
-            max_in_flight: settings.persistence_concurrency(),
-            max_pending_signals: settings.admission_queue_capacity(),
+            max_in_flight,
+            max_pending_signals,
             pending_signal_count: 0,
             pending: std::collections::BTreeMap::new(),
             tasks: RawTransactionIngestHydrationTasks::new(),
@@ -3615,6 +3725,33 @@ async fn fetch_hydration(
     return std::result::Result::Ok(RawTransactionIngestHydrationFetch { key, observed });
 }
 
+struct RawTransactionIngestHydrationLeaderGuard {
+    armed: bool,
+    key: RawTransactionIngestHydrationKey,
+    registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>,
+}
+
+impl RawTransactionIngestHydrationLeaderGuard {
+    fn new(registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>, key: RawTransactionIngestHydrationKey) -> Self {
+        return Self { armed: true, key, registry };
+    }
+
+    fn disarm(&mut self) {
+        self.armed = false;
+        return;
+    }
+}
+
+impl std::ops::Drop for RawTransactionIngestHydrationLeaderGuard {
+    fn drop(&mut self) {
+        if self.armed {
+            self.registry
+                .publish_and_remove(&self.key, RawTransactionIngestSharedHydrationResult::Failed(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED));
+        }
+        return;
+    }
+}
+
 async fn fetch_hydration_shared(
     global_registry: std::sync::Arc<RawTransactionIngestGlobalHydrationRegistry>,
     http_pool: ksp_onchain_transport_lib::HttpTransportPool,
@@ -3628,14 +3765,22 @@ async fn fetch_hydration_shared(
         std::result::Result::Err(error) => return std::result::Result::Err(error),
     };
     if leader {
+        let mut leader_guard = RawTransactionIngestHydrationLeaderGuard::new(std::sync::Arc::clone(&global_registry), key.clone());
+        let permit = std::sync::Arc::clone(&global_registry.hydration_permits).acquire_owned().await;
+        let _permit = match permit {
+            std::result::Result::Ok(value) => value,
+            std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.global_hydration_permit_closed")),
+        };
         let fetched = fetch_hydration(http_pool, hydration_role, expected_network, key.clone(), commitment).await;
         return match fetched {
             std::result::Result::Ok(value) => {
                 global_registry.publish_and_remove(&key, RawTransactionIngestSharedHydrationResult::Available(value.observed.clone()));
+                leader_guard.disarm();
                 std::result::Result::Ok(value)
             },
             std::result::Result::Err(error) => {
                 global_registry.publish_and_remove(&key, RawTransactionIngestSharedHydrationResult::Failed(error.code()));
+                leader_guard.disarm();
                 std::result::Result::Err(error)
             },
         };

crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
-// version: 24
+// version: 25
 
 //! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
 
@@ -604,3 +604,41 @@ fn v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_o
     }
     return;
 }
+
+#[test]
+fn v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades() {
+    let admission = include_str!("../src/admission.rs");
+    let persistence = include_str!("../src/persistence.rs");
+    let resources = include_str!("../src/runtime_resources.rs");
+    for required in [
+        "validate_hydration_fairness_capacity",
+        "hydration_pending_limit",
+        "hydration_in_flight_limit",
+        "tokio::sync::Semaphore",
+        "acquire_owned",
+        "RawTransactionIngestHydrationLeaderGuard",
+        "RawTransactionIngestCanonicalState",
+        "content_conflict_error()",
+    ] {
+        assert!(
+            admission.contains(required) || persistence.contains(required) || resources.contains(required),
+            "required pre.009 global-bound contract missing: {required}"
+        );
+    }
+    for forbidden in [
+        "unbounded_channel",
+        "ksp_store_postgres_lib::",
+        "ksp_config_lib::",
+        "ksp_job_backfill_lib::",
+        "reqwest::",
+        "tokio_tungstenite::",
+        "tonic::",
+        "yellowstone_grpc_proto::",
+    ] {
+        assert!(
+            !admission.contains(forbidden) && !persistence.contains(forbidden) && !resources.contains(forbidden),
+            "pre.009 crossed a Worker facade boundary: {forbidden}"
+        );
+    }
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
-// version: 18
+// version: 19
 
 //! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
 
@@ -786,3 +786,42 @@ fn v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_priv
     assert!(!resources.contains("unbounded_channel"));
     return;
 }
+
+#[test]
+fn v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit() {
+    let admission = include_str!("../src/admission.rs");
+    let persistence = include_str!("../src/persistence.rs");
+    let resources = include_str!("../src/runtime_resources.rs");
+    for required in [
+        "runtime_resources.hydration_pending_capacity_insufficient",
+        "runtime_resources.hydration_concurrency_insufficient",
+        "base + usize::from(hydration_entry_index < remainder)",
+        "hydration_in_flight_limit",
+        "tokio::sync::Semaphore::new(max_in_flight)",
+        "RawTransactionIngestHydrationLeaderGuard",
+        "source.global_hydration_pending_saturated",
+        "block_time: transaction.block_time()",
+        "slot: transaction.slot()",
+        "content_hash: transaction.payload().content_hash()",
+    ] {
+        assert!(
+            admission.contains(required) || persistence.contains(required) || resources.contains(required),
+            "required pre.009 adversarial guard missing: {required}"
+        );
+    }
+    for forbidden in [
+        "VecDeque::new()",
+        "unbounded_channel",
+        "first_provider_wins",
+        "provider_priority",
+        "majority_vote",
+        "preferred_provider",
+        "overwrite_conflict",
+    ] {
+        assert!(
+            !admission.contains(forbidden) && !persistence.contains(forbidden) && !resources.contains(forbidden),
+            "pre.009 introduced forbidden unbounded/provider-preference behavior: {forbidden}"
+        );
+    }
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
-// version: 18
+// version: 19
 
 //! External public-surface proofs for the RAW transaction ingest Worker foundation.
 
@@ -351,3 +351,20 @@ fn v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private() {
     }
     return;
 }
+
+#[test]
+fn v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private() {
+    let root = include_str!("../src/lib.rs");
+    for private_name in [
+        "RawTransactionIngestCanonicalState",
+        "RawTransactionIngestGlobalHydrationRegistry",
+        "RawTransactionIngestHydrationLeaderGuard",
+        "validate_hydration_fairness_capacity",
+        "hydration_pending_limit",
+        "hydration_in_flight_limit",
+    ] {
+        assert!(!root.contains(&std::format!("pub use self::runtime_resources::{private_name}")));
+        assert!(!root.contains(&std::format!("pub use self::persistence::{private_name}")));
+    }
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
-// version: 16
+// version: 17
 
 //! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
 
@@ -129,6 +129,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
         "v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted",
         "v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity",
         "v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private",
+        "v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit",
     ] {
         assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
     }
@@ -148,6 +149,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
     assert!(dependency_boundary.contains("v0_3_13_pre_006_http_block_polling_reuses_transport_facades_without_becoming_backfill"));
     assert!(dependency_boundary.contains("v0_3_13_pre_007_multi_source_supervisor_and_inventory_remain_private_bounded_and_source_neutral"));
     assert!(dependency_boundary.contains("v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_only"));
+    assert!(dependency_boundary.contains("v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades"));
     let public_api = include_str!("public_api.rs");
     assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
     assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
@@ -160,6 +162,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
     assert!(public_api.contains("v0_3_13_pre_006_http_block_polling_runtime_resource_surface_is_bounded_and_transport_owned"));
     assert!(public_api.contains("v0_3_13_pre_007_source_inventory_and_logical_keys_remain_private"));
     assert!(public_api.contains("v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private"));
+    assert!(public_api.contains("v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private"));
     assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
     assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
     return;

crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
-// version: 5
+// version: 6
 
 fn material(
     network: &str,
@@ -311,3 +311,104 @@ async fn v0_3_13_pre_008_distinct_source_keys_produce_distinct_idempotent_observ
     assert_ne!(first.observation().observation_key(), second.observation().observation_key());
     return;
 }
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_009_duplicate_storm_cannot_starve_second_ready_source_on_bounded_admission() {
+    let (network, initial_material) = match material("mainnet", 60, "AQID") {
+        std::option::Option::Some(value) => value,
+        std::option::Option::None => return,
+    };
+    let (_, peer_material) = match material("mainnet", 99, "BwgJ") {
+        std::option::Option::Some(value) => value,
+        std::option::Option::None => return,
+    };
+    let initial_provenance = match provenance() {
+        std::option::Option::Some(value) => value,
+        std::option::Option::None => return,
+    };
+    let storm_provenance = match provenance() {
+        std::option::Option::Some(value) => value,
+        std::option::Option::None => return,
+    };
+    let peer_provenance = match provenance() {
+        std::option::Option::Some(value) => value,
+        std::option::Option::None => return,
+    };
+    let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
+    assert!(
+        sender
+            .send(crate::RawTransactionIngress {
+                material: initial_material,
+                network: network.clone(),
+                provenance: initial_provenance,
+                source_key: [60_u8; 32],
+            })
+            .await
+            .is_ok()
+    );
+    let storm_sender = sender.clone();
+    let storm_network = network.clone();
+    let storm_task = tokio::spawn(async move {
+        for _ in 0..8 {
+            let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
+                storm_network.clone(),
+                ksp_store_lib::RawTransactionSignature::new([61_u8; 64]),
+                42,
+                std::option::Option::Some(1_700_000_000),
+                "BAUG",
+                ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
+                ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
+                ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
+            );
+            if storm_sender
+                .send(crate::RawTransactionIngress {
+                    material,
+                    network: storm_network.clone(),
+                    provenance: storm_provenance.clone(),
+                    source_key: [61_u8; 32],
+                })
+                .await
+                .is_err()
+            {
+                return false;
+            }
+        }
+        return true;
+    });
+    tokio::task::yield_now().await;
+    let peer_sender = sender.clone();
+    let peer_network = network.clone();
+    let peer_task = tokio::spawn(async move {
+        return peer_sender
+            .send(crate::RawTransactionIngress { material: peer_material, network: peer_network, provenance: peer_provenance, source_key: [99_u8; 32] })
+            .await
+            .is_ok();
+    });
+    tokio::task::yield_now().await;
+    assert_eq!(admission.queue_depth(), 1);
+    let first = match admission.receive(&network).await {
+        std::result::Result::Ok(std::option::Option::Some(value)) => value,
+        _ => return,
+    };
+    assert_eq!(first.transaction().reference().signature().as_bytes(), &[60_u8; 64]);
+    let second = match admission.receive(&network).await {
+        std::result::Result::Ok(std::option::Option::Some(value)) => value,
+        _ => return,
+    };
+    assert_eq!(second.transaction().reference().signature().as_bytes(), &[61_u8; 64]);
+    let third = match admission.receive(&network).await {
+        std::result::Result::Ok(std::option::Option::Some(value)) => value,
+        _ => return,
+    };
+    assert_eq!(third.transaction().reference().signature().as_bytes(), &[99_u8; 64]);
+    for _ in 0..7 {
+        let next = admission.receive(&network).await;
+        assert!(matches!(next, std::result::Result::Ok(std::option::Option::Some(_))));
+        assert!(admission.queue_depth() <= 1);
+    }
+    let peer_joined = peer_task.await;
+    assert!(matches!(peer_joined, std::result::Result::Ok(true)));
+    let storm_joined = storm_task.await;
+    assert!(matches!(storm_joined, std::result::Result::Ok(true)));
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs
-// version: 2
+// version: 3
 
 #[derive(Clone, Copy)]
 enum PortResponse {
@@ -92,6 +92,17 @@ fn acquisition_with_observation_key(
     network_name: &str,
     signature_byte: u8,
     observation_byte: u8,
+) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
+    return acquisition_with_shape(network_name, signature_byte, observation_byte, 42, std::option::Option::Some(1_700_000_000), "AQID");
+}
+
+fn acquisition_with_shape(
+    network_name: &str,
+    signature_byte: u8,
+    observation_byte: u8,
+    slot: u64,
+    block_time: std::option::Option<i64>,
+    transaction_data: &str,
 ) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
     let network = match ksp_store_lib::RawNetworkId::new(network_name) {
         std::result::Result::Ok(value) => value,
@@ -101,9 +112,9 @@ fn acquisition_with_observation_key(
     let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
         network,
         signature,
-        42,
-        std::option::Option::Some(1_700_000_000),
-        "AQID",
+        slot,
+        block_time,
+        transaction_data,
         ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
         ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
         ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
@@ -357,3 +368,41 @@ async fn v0_3_13_pre_008_replayed_same_observation_is_idempotent_without_raw_res
     assert_eq!(second_outcome.observation(), crate::RawTransactionIngestObservationPersistence::AlreadyPresent);
     return;
 }
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_009_source_skew_and_provider_disagreement_fail_before_additional_observation() {
+    let cases = [
+        (43_u64, std::option::Option::Some(1_700_000_000_i64), "AQID"),
+        (42_u64, std::option::Option::Some(1_700_000_001_i64), "AQID"),
+        (42_u64, std::option::Option::Some(1_700_000_000_i64), "BAUG"),
+    ];
+    for (index, (slot, block_time, transaction_data)) in cases.into_iter().enumerate() {
+        let network = match network("mainnet") {
+            std::option::Option::Some(value) => value,
+            std::option::Option::None => return,
+        };
+        let port = FakePersistencePort::new(
+            network,
+            PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
+        );
+        let convergence = crate::RawTransactionIngestPersistenceConvergence::new(8);
+        let first = match acquisition_with_shape("mainnet", 51, 10, 42, std::option::Option::Some(1_700_000_000), "AQID") {
+            std::option::Option::Some(value) => value,
+            std::option::Option::None => return,
+        };
+        let divergent = match acquisition_with_shape("mainnet", 51, index as u8 + 20, slot, block_time, transaction_data) {
+            std::option::Option::Some(value) => value,
+            std::option::Option::None => return,
+        };
+        assert!(crate::persist_raw_transaction_ingest_converged_acquisition(&port, first, &convergence).await.is_ok());
+        let conflict = crate::persist_raw_transaction_ingest_converged_acquisition(&port, divergent, &convergence).await;
+        let error = match conflict {
+            std::result::Result::Ok(_) => return,
+            std::result::Result::Err(value) => value,
+        };
+        assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT);
+        assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
+        assert_eq!(port.observation_calls.load(std::sync::atomic::Ordering::Acquire), 0);
+    }
+    return;
+}

crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs

@@ -1,5 +1,5 @@
 // file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
-// version: 20
+// version: 21
 
 fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
     return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
@@ -3008,7 +3008,7 @@ fn v0_3_13_pre_008_global_hydration_registry_coalesces_cross_source_key_to_one_l
         network,
         signature: ksp_store_lib::RawTransactionSignature::new([61_u8; 64]),
     };
-    let registry = super::RawTransactionIngestGlobalHydrationRegistry::new(&settings);
+    let registry = super::RawTransactionIngestGlobalHydrationRegistry::new(settings.admission_queue_capacity(), settings.persistence_concurrency());
     let (first_leader, _first_receiver) = match registry.subscribe_or_lead(&key) {
         std::result::Result::Ok(value) => value,
         std::result::Result::Err(_) => return,
@@ -3034,3 +3034,107 @@ fn v0_3_13_pre_008_global_hydration_registry_coalesces_cross_source_key_to_one_l
     assert!(third_leader);
     return;
 }
+
+#[test]
+fn v0_3_13_pre_009_hydration_quotas_partition_exact_bounds_and_reject_underprovision() {
+    assert!(super::validate_hydration_fairness_capacity(8, 5, 3).is_ok());
+    let pending_error = match super::validate_hydration_fairness_capacity(2, 5, 3) {
+        std::result::Result::Ok(()) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(pending_error.context()[0].value(), "runtime_resources.hydration_pending_capacity_insufficient");
+    let concurrency_error = match super::validate_hydration_fairness_capacity(8, 2, 3) {
+        std::result::Result::Ok(()) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(concurrency_error.context()[0].value(), "runtime_resources.hydration_concurrency_insufficient");
+    let pending_limits = [super::hydration_pending_limit(8, 3, 0), super::hydration_pending_limit(8, 3, 1), super::hydration_pending_limit(8, 3, 2)];
+    assert_eq!(pending_limits, [3, 3, 2]);
+    assert_eq!(pending_limits.iter().copied().sum::<usize>(), 8);
+    assert!(pending_limits.iter().copied().all(|limit| return limit >= 1));
+    let in_flight_limits = [super::hydration_in_flight_limit(5, 3, 0), super::hydration_in_flight_limit(5, 3, 1), super::hydration_in_flight_limit(5, 3, 2)];
+    assert_eq!(in_flight_limits, [2, 2, 1]);
+    assert_eq!(in_flight_limits.iter().copied().sum::<usize>(), 5);
+    assert!(in_flight_limits.iter().copied().all(|limit| return limit >= 1));
+    assert!(super::validate_hydration_fairness_capacity(1, 1, 0).is_ok());
+    assert_eq!(super::hydration_pending_limit(1, 0, 0), 0);
+    assert_eq!(super::hydration_in_flight_limit(1, 0, 0), 0);
+    return;
+}
+
+#[test]
+fn v0_3_13_pre_009_duplicate_hydration_storm_keeps_one_global_key_and_distinct_keys_bounded() {
+    let network = match ksp_store_lib::RawNetworkId::new("devnet") {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    let registry = super::RawTransactionIngestGlobalHydrationRegistry::new(2, 1);
+    let first_key = super::RawTransactionIngestHydrationKey {
+        commitment: ksp_onchain_transport_lib::SolanaCommitment::Confirmed.as_str(),
+        network: network.clone(),
+        signature: ksp_store_lib::RawTransactionSignature::new([71_u8; 64]),
+    };
+    for index in 0..32 {
+        let subscribed = registry.subscribe_or_lead(&first_key);
+        let (leader, _receiver) = match subscribed {
+            std::result::Result::Ok(value) => value,
+            std::result::Result::Err(_) => return,
+        };
+        assert_eq!(leader, index == 0);
+    }
+    let second_key = super::RawTransactionIngestHydrationKey {
+        commitment: ksp_onchain_transport_lib::SolanaCommitment::Confirmed.as_str(),
+        network: network.clone(),
+        signature: ksp_store_lib::RawTransactionSignature::new([72_u8; 64]),
+    };
+    let third_key = super::RawTransactionIngestHydrationKey {
+        commitment: ksp_onchain_transport_lib::SolanaCommitment::Confirmed.as_str(),
+        network,
+        signature: ksp_store_lib::RawTransactionSignature::new([73_u8; 64]),
+    };
+    assert!(matches!(registry.subscribe_or_lead(&second_key), std::result::Result::Ok((true, _))));
+    let saturated = registry.subscribe_or_lead(&third_key);
+    let error = match saturated {
+        std::result::Result::Ok(_) => return,
+        std::result::Result::Err(value) => value,
+    };
+    assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
+    assert_eq!(error.context()[0].value(), "source.global_hydration_pending_saturated");
+    return;
+}
+
+#[tokio::test(flavor = "current_thread")]
+async fn v0_3_13_pre_009_global_hydration_permit_and_leader_cleanup_are_bounded_and_fail_closed() {
+    let network = match ksp_store_lib::RawNetworkId::new("devnet") {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    let registry = std::sync::Arc::new(super::RawTransactionIngestGlobalHydrationRegistry::new(4, 1));
+    let first_permit = std::sync::Arc::clone(&registry.hydration_permits).try_acquire_owned();
+    let first_permit = match first_permit {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    assert!(std::sync::Arc::clone(&registry.hydration_permits).try_acquire_owned().is_err());
+    std::mem::drop(first_permit);
+    assert!(std::sync::Arc::clone(&registry.hydration_permits).try_acquire_owned().is_ok());
+    let key = super::RawTransactionIngestHydrationKey {
+        commitment: ksp_onchain_transport_lib::SolanaCommitment::Confirmed.as_str(),
+        network,
+        signature: ksp_store_lib::RawTransactionSignature::new([74_u8; 64]),
+    };
+    let subscribed = registry.subscribe_or_lead(&key);
+    let (leader, receiver) = match subscribed {
+        std::result::Result::Ok(value) => value,
+        std::result::Result::Err(_) => return,
+    };
+    assert!(leader);
+    {
+        let _guard = super::RawTransactionIngestHydrationLeaderGuard::new(std::sync::Arc::clone(&registry), key.clone());
+    }
+    let published = receiver.borrow().clone();
+    assert!(matches!(published, std::option::Option::Some(super::RawTransactionIngestSharedHydrationResult::Failed(_))));
+    let resubscribed = registry.subscribe_or_lead(&key);
+    assert!(matches!(resubscribed, std::result::Result::Ok((true, _))));
+    return;
+}