v0.3.13-pre.011
This commit is contained in:
@@ -1,12 +1,12 @@
|
||||
# file: Cargo.toml
|
||||
# version: 559
|
||||
# version: 560
|
||||
|
||||
[workspace]
|
||||
resolver = "3"
|
||||
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api", "crates/ksp-worker-raw-transaction-ingest-lib"]
|
||||
|
||||
[workspace.package]
|
||||
version = "0.3.13-pre.10"
|
||||
version = "0.3.13-pre.11"
|
||||
edition = "2024"
|
||||
license = "MIT"
|
||||
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
|
||||
<!-- version: 10 -->
|
||||
<!-- version: 11 -->
|
||||
|
||||
# ksp-worker-raw-transaction-ingest-lib
|
||||
|
||||
@@ -165,7 +165,7 @@ Le Worker s'exécute sur le runtime Tokio courant du caller. Il ne crée pas de
|
||||
- utiliser la même source via `WorkerSnapshotSource` ;
|
||||
- attendre le terminal après drain et join des tâches possédées.
|
||||
|
||||
Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source relaie le stop à toutes les sources et les rejoint avant de rendre son résultat au supervisor Worker ; les tâches source, hydration et persistence possédées sont ensuite drainées ou abort+join avant publication terminale. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
|
||||
Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source relaie le stop à toutes les sources et les rejoint avant de rendre son résultat au supervisor Worker ; les tâches source, hydration et persistence possédées sont ensuite drainées ou abort+join avant publication terminale. Si la deadline expire, l'abort du wrapper source détruit aussi son `JoinSet` interne et annule ses tâches imbriquées avant le terminal. Une faute déjà observée n'est pas remplacée par un stop concurrent, sauf le `drain_timeout` terminal lorsqu'une récupération bornée dépasse sa deadline. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
|
||||
|
||||
## Admission, coalescence et backpressure
|
||||
|
||||
@@ -183,7 +183,7 @@ source reference-bearing active => quota pending >= 1 et quota in-flight >= 1
|
||||
|
||||
Pour garantir simultanément ces bornes et l'absence de starvation structurelle, le démarrage échoue avant spawn si le nombre de sources reference-bearing dépasse `admission_queue_capacity` ou `persistence_concurrency`. Un sémaphore global protège en plus l'ouverture effective des hydrations HTTP.
|
||||
|
||||
Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu canonique identique ajoutent uniquement leur observation déterministe. Une divergence de slot, block time, format ou hash canonique devient un content conflict terminal ; aucune majorité, préférence provider ou overwrite n'est appliqué. Le Store conserve son guard durable final.
|
||||
Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. La publication du résultat partagé notifie les followers sous le verrou de registry avant de retirer la clé : une nouvelle génération de leader ne peut donc pas s'intercaler entre retrait et notification. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu canonique identique ajoutent uniquement leur observation déterministe. Une divergence de slot, block time, format ou hash canonique devient un content conflict terminal ; aucune majorité, préférence provider ou overwrite n'est appliqué. Le Store conserve son guard durable final.
|
||||
|
||||
Les retries/reroutages HTTP appartiennent à `ksp-onchain-transport-lib`. Le Worker ne possède pas une seconde boucle de retry autour de `getTransaction`.
|
||||
|
||||
@@ -279,7 +279,7 @@ worker_raw_transaction_ingest.source_failed
|
||||
worker_raw_transaction_ingest.drain_timeout
|
||||
```
|
||||
|
||||
Les diagnostics et `Debug` ne recopient pas de payload RAW, signature, URL, credential, filtre provider, texte backend/provider arbitraire ou client inférieur.
|
||||
Les diagnostics et `Debug` ne recopient pas de payload RAW, signature, URL, credential, filtre provider, texte backend/provider arbitraire ou client inférieur. Les agrégations de compteurs de continuité multi-source utilisent une arithmétique vérifiée ; un overflow devient `counter_exhausted` au lieu d'être saturé silencieusement.
|
||||
|
||||
## Dépendances
|
||||
|
||||
@@ -302,8 +302,7 @@ La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Sto
|
||||
|
||||
La verticale actuelle ne possède pas :
|
||||
|
||||
- plusieurs sources productives simultanées dans `RawTransactionIngestRuntimeResources` ;
|
||||
- sélection Config interne au Worker ;
|
||||
- sélection Config interne, reconfiguration dynamique ou policy de failover entre sources ;
|
||||
- checkpoint persistent de processing frontier ;
|
||||
- campagne de réparation historique automatique ;
|
||||
- application Desk ou process autonome ;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
|
||||
<!-- version: 10 -->
|
||||
<!-- version: 11 -->
|
||||
|
||||
# Utilisation de ksp-worker-raw-transaction-ingest-lib
|
||||
|
||||
@@ -333,7 +333,7 @@ let replay_attempts = snapshot.source_replay_attempt_total();
|
||||
let proven_gaps = snapshot.source_continuity_gap_total();
|
||||
```
|
||||
|
||||
`admission_queue_depth()`, `in_flight_persistence()`, `hydration_pending()`, `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` sont des gauges latest-value. Les compteurs cumulés ne wrapent jamais silencieusement ; l'épuisement est terminal avec `worker_raw_transaction_ingest.counter_exhausted`.
|
||||
`admission_queue_depth()`, `in_flight_persistence()`, `hydration_pending()`, `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` sont des gauges latest-value. Les compteurs cumulés et les agrégats multi-source ne wrapent ni ne saturent silencieusement ; l'épuisement est terminal avec `worker_raw_transaction_ingest.counter_exhausted`.
|
||||
|
||||
## Interpréter la processing frontier
|
||||
|
||||
@@ -386,7 +386,7 @@ if accepted {
|
||||
}
|
||||
```
|
||||
|
||||
`request_stop()` est idempotent. Le terminal n'est publié qu'après le drain borné et la récupération des tâches possédées.
|
||||
`request_stop()` est idempotent. Le terminal n'est publié qu'après le drain borné et la récupération des tâches possédées. Si la deadline de drain expire, toutes les tâches source/persistence encore possédées sont abortées puis jointes avant publication terminale ; une persistence libérée après ce terminal ne peut donc pas produire une complétion tardive. Une faute source déjà observée reste prioritaire face à un stop concurrent, sauf si le drain lui-même expire et devient le terminal `drain_timeout`.
|
||||
|
||||
## Interpréter les faults
|
||||
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||
// version: 25
|
||||
// version: 26
|
||||
|
||||
#![warn(missing_docs)]
|
||||
#![deny(unreachable_pub)]
|
||||
#![forbid(unsafe_code)]
|
||||
|
||||
//! Source-neutral runtime foundation for continuous KSP RAW transaction ingestion.
|
||||
//! Owned source, hydration and persistence tasks are bounded and joined before terminal publication.
|
||||
//!
|
||||
//! This tranche owns the concrete Worker family identity, validated technical settings
|
||||
//! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||
// version: 16
|
||||
// version: 17
|
||||
|
||||
type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
|
||||
type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
|
||||
@@ -343,13 +343,16 @@ fn source_completion(
|
||||
in_flight_persistence: usize,
|
||||
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||
match joined {
|
||||
let source_code = match joined {
|
||||
std::result::Result::Ok(std::result::Result::Ok(())) => return std::option::Option::None,
|
||||
std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => {},
|
||||
}
|
||||
std::result::Result::Ok(std::result::Result::Err(error)) if error.code() == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED => {
|
||||
crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED
|
||||
},
|
||||
std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED,
|
||||
};
|
||||
let published = snapshots.record_source_failure(state, admission_queue_depth, in_flight_persistence);
|
||||
return match published {
|
||||
std::result::Result::Ok(()) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED),
|
||||
std::result::Result::Ok(()) => std::option::Option::Some(source_code),
|
||||
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||
// version: 25
|
||||
// version: 26
|
||||
|
||||
use sha2::Digest; // rust-rules: trait-import
|
||||
|
||||
@@ -156,14 +156,18 @@ impl RawTransactionIngestLiveSource {
|
||||
} else {
|
||||
crate::RawTransactionIngestSourceState::Failed
|
||||
};
|
||||
inventory_publisher.publish(source_projection_with_state(latest, terminal_state));
|
||||
if let std::result::Result::Err(error) = inventory_publisher.publish(source_projection_with_state(latest, terminal_state)) {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
changed = source_frontier_receiver.changed() => {
|
||||
if changed.is_err() {
|
||||
return std::result::Result::Err(crate::runtime_error("source.frontier_channel_closed"));
|
||||
}
|
||||
inventory_publisher.publish(*source_frontier_receiver.borrow_and_update());
|
||||
if let std::result::Result::Err(error) = inventory_publisher.publish(*source_frontier_receiver.borrow_and_update()) {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -171,7 +175,7 @@ impl RawTransactionIngestLiveSource {
|
||||
}
|
||||
|
||||
impl RawTransactionIngestSourceInventory {
|
||||
fn aggregate(&self) -> crate::RawTransactionIngestProcessingFrontierProjection {
|
||||
fn aggregate(&self) -> ksp_core_lib::Result<crate::RawTransactionIngestProcessingFrontierProjection> {
|
||||
let mut hydration_pending = 0_usize;
|
||||
let mut oldest_pending_slot = std::option::Option::None;
|
||||
let mut processing_frontier_slot = std::option::Option::None;
|
||||
@@ -189,7 +193,10 @@ impl RawTransactionIngestSourceInventory {
|
||||
let mut any_reconnecting = false;
|
||||
let mut all_closed = !self.source_projections.is_empty();
|
||||
for projection in &self.source_projections {
|
||||
hydration_pending = hydration_pending.saturating_add(projection.hydration_pending());
|
||||
hydration_pending = match hydration_pending.checked_add(projection.hydration_pending()) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("hydration_pending")),
|
||||
};
|
||||
oldest_pending_slot = minimum_optional_slot(oldest_pending_slot, projection.oldest_pending_slot());
|
||||
match projection.processing_frontier_slot() {
|
||||
std::option::Option::Some(slot) => {
|
||||
@@ -199,9 +206,18 @@ impl RawTransactionIngestSourceInventory {
|
||||
all_frontiers_present = false;
|
||||
},
|
||||
}
|
||||
source_continuity_gap_total = source_continuity_gap_total.saturating_add(projection.source_continuity_gap_total());
|
||||
source_reconnect_total = source_reconnect_total.saturating_add(projection.source_reconnect_total());
|
||||
source_replay_attempt_total = source_replay_attempt_total.saturating_add(projection.source_replay_attempt_total());
|
||||
source_continuity_gap_total = match source_continuity_gap_total.checked_add(projection.source_continuity_gap_total()) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("source_continuity_gap_total")),
|
||||
};
|
||||
source_reconnect_total = match source_reconnect_total.checked_add(projection.source_reconnect_total()) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("source_reconnect_total")),
|
||||
};
|
||||
source_replay_attempt_total = match source_replay_attempt_total.checked_add(projection.source_replay_attempt_total()) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return std::result::Result::Err(crate::counter_exhausted_error("source_replay_attempt_total")),
|
||||
};
|
||||
match projection.source_state() {
|
||||
std::option::Option::Some(crate::RawTransactionIngestSourceState::Active) => {
|
||||
source_active += 1;
|
||||
@@ -244,9 +260,11 @@ impl RawTransactionIngestSourceInventory {
|
||||
} else {
|
||||
std::option::Option::None
|
||||
};
|
||||
return crate::RawTransactionIngestProcessingFrontierProjection::new(hydration_pending, processing_frontier_slot, oldest_pending_slot)
|
||||
.with_source_continuity(source_state, source_reconnect_total, source_replay_attempt_total, source_continuity_gap_total)
|
||||
.with_source_counts(source_total, source_active, source_reconnecting, source_failed);
|
||||
return std::result::Result::Ok(
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(hydration_pending, processing_frontier_slot, oldest_pending_slot)
|
||||
.with_source_continuity(source_state, source_reconnect_total, source_replay_attempt_total, source_continuity_gap_total)
|
||||
.with_source_counts(source_total, source_active, source_reconnecting, source_failed),
|
||||
);
|
||||
}
|
||||
|
||||
fn new(source_keys: std::vec::Vec<[u8; 32]>) -> Self {
|
||||
@@ -254,35 +272,45 @@ impl RawTransactionIngestSourceInventory {
|
||||
return Self { source_keys, source_projections };
|
||||
}
|
||||
|
||||
fn update(&mut self, entry_index: usize, source_key: [u8; 32], projection: crate::RawTransactionIngestProcessingFrontierProjection) {
|
||||
fn update(
|
||||
&mut self,
|
||||
entry_index: usize,
|
||||
source_key: [u8; 32],
|
||||
projection: crate::RawTransactionIngestProcessingFrontierProjection,
|
||||
) -> ksp_core_lib::Result<()> {
|
||||
let expected_key = match self.source_keys.get(entry_index) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return,
|
||||
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.inventory_entry_missing")),
|
||||
};
|
||||
if expected_key != &source_key {
|
||||
return;
|
||||
return std::result::Result::Err(crate::runtime_error("source.inventory_key_mismatch"));
|
||||
}
|
||||
let current = match self.source_projections.get_mut(entry_index) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return,
|
||||
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.inventory_projection_missing")),
|
||||
};
|
||||
*current = projection;
|
||||
return;
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
}
|
||||
|
||||
impl RawTransactionIngestSourceInventoryPublisher {
|
||||
fn publish(&self, projection: crate::RawTransactionIngestProcessingFrontierProjection) {
|
||||
fn publish(&self, projection: crate::RawTransactionIngestProcessingFrontierProjection) -> ksp_core_lib::Result<()> {
|
||||
let aggregate = {
|
||||
let mut inventory = match self.inventory.lock() {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(poisoned) => poisoned.into_inner(),
|
||||
};
|
||||
inventory.update(self.entry_index, self.source_key, projection);
|
||||
inventory.aggregate()
|
||||
if let std::result::Result::Err(error) = inventory.update(self.entry_index, self.source_key, projection) {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
match inventory.aggregate() {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
}
|
||||
};
|
||||
self.aggregate_sender.send_replace(aggregate);
|
||||
return;
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3502,15 +3530,19 @@ impl RawTransactionIngestGlobalHydrationRegistry {
|
||||
}
|
||||
|
||||
fn publish_and_remove(&self, key: &RawTransactionIngestHydrationKey, result: RawTransactionIngestSharedHydrationResult) {
|
||||
let sender = {
|
||||
let mut pending = match self.pending.lock() {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(poisoned) => poisoned.into_inner(),
|
||||
};
|
||||
pending.remove(key)
|
||||
let mut pending = match self.pending.lock() {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(poisoned) => poisoned.into_inner(),
|
||||
};
|
||||
if let std::option::Option::Some(sender) = sender {
|
||||
sender.send_replace(std::option::Option::Some(result));
|
||||
let published = match pending.get(key) {
|
||||
std::option::Option::Some(sender) => {
|
||||
sender.send_replace(std::option::Option::Some(result));
|
||||
true
|
||||
},
|
||||
std::option::Option::None => false,
|
||||
};
|
||||
if published {
|
||||
let _removed = pending.remove(key);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||
// version: 27
|
||||
// version: 28
|
||||
|
||||
//! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
|
||||
|
||||
@@ -663,3 +663,42 @@ fn v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backe
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades() {
|
||||
let runtime = include_str!("../src/runtime.rs");
|
||||
let resources = include_str!("../src/runtime_resources.rs");
|
||||
for required in [
|
||||
"tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
|
||||
"persistence.abort_all()",
|
||||
"children.abort_all()",
|
||||
"while persistence.join_next().await.is_some() {}",
|
||||
"while children.join_next().await.is_some() {}",
|
||||
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||
] {
|
||||
assert!(runtime.contains(required), "required pre.011 runtime shutdown guard missing: {required}");
|
||||
}
|
||||
for required in [
|
||||
"RawTransactionIngestHydrationLeaderGuard",
|
||||
"publish_and_remove",
|
||||
"pending.remove(key)",
|
||||
"source.inventory_key_mismatch",
|
||||
"checked_add(projection.hydration_pending())",
|
||||
"checked_add(projection.source_reconnect_total())",
|
||||
] {
|
||||
assert!(resources.contains(required), "required pre.011 source-race guard missing: {required}");
|
||||
}
|
||||
for forbidden in [
|
||||
"unbounded_channel",
|
||||
"ksp_store_postgres_lib::",
|
||||
"ksp_config_lib::",
|
||||
"ksp_job_backfill_lib::",
|
||||
"reqwest::",
|
||||
"tokio_tungstenite::",
|
||||
"tonic::",
|
||||
"yellowstone_grpc_proto::",
|
||||
] {
|
||||
assert!(!runtime.contains(forbidden) && !resources.contains(forbidden), "pre.011 crossed a Worker/facade boundary: {forbidden}");
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||
// version: 22
|
||||
// version: 23
|
||||
|
||||
//! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
|
||||
//! External public, security, redaction and release-boundary hardening canaries through `pre.011`.
|
||||
|
||||
fn network(value: &'static str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||
let result = ksp_store_lib::RawNetworkId::new(value);
|
||||
@@ -854,3 +854,81 @@ fn v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted() {
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe() {
|
||||
let runtime = include_str!("../src/runtime.rs");
|
||||
let resources = include_str!("../src/runtime_resources.rs");
|
||||
let supervisor = match runtime.split_once("async fn run_supervisor<Spawner>(") {
|
||||
std::option::Option::Some((_, tail)) => match tail.split_once("fn spawn_persistence(") {
|
||||
std::option::Option::Some((value, _)) => value,
|
||||
std::option::Option::None => "",
|
||||
},
|
||||
std::option::Option::None => "",
|
||||
};
|
||||
let active_shutdown = match supervisor.split_once("let mut fault = supervise_until_stop(") {
|
||||
std::option::Option::Some((_, value)) => value,
|
||||
std::option::Option::None => "",
|
||||
};
|
||||
let stop_position = active_shutdown.find("source_stop_sender.send_replace(true);");
|
||||
let stopping_position = active_shutdown.find("begin_stopping(");
|
||||
let drain_position = active_shutdown.find("drain_owned_work(");
|
||||
let terminal_position = active_shutdown.find("match fault {");
|
||||
assert!(matches!((stop_position, stopping_position), (std::option::Option::Some(stop), std::option::Option::Some(stopping)) if stop < stopping));
|
||||
assert!(matches!((stopping_position, drain_position), (std::option::Option::Some(stopping), std::option::Option::Some(drain)) if stopping < drain));
|
||||
assert!(matches!((drain_position, terminal_position), (std::option::Option::Some(drain), std::option::Option::Some(terminal)) if drain < terminal));
|
||||
for required in [
|
||||
"tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
|
||||
"persistence.abort_all()",
|
||||
"children.abort_all()",
|
||||
"while persistence.join_next().await.is_some() {}",
|
||||
"while children.join_next().await.is_some() {}",
|
||||
] {
|
||||
assert!(runtime.contains(required), "required pre.011 bounded-drain guard missing: {required}");
|
||||
}
|
||||
let registry_publish = match resources.split_once("fn publish_and_remove(&self, key: &RawTransactionIngestHydrationKey") {
|
||||
std::option::Option::Some((_, tail)) => match tail.split_once("type RawTransactionIngestHydrationTasks") {
|
||||
std::option::Option::Some((value, _)) => value,
|
||||
std::option::Option::None => "",
|
||||
},
|
||||
std::option::Option::None => "",
|
||||
};
|
||||
let notify_position = registry_publish.find("sender.send_replace(std::option::Option::Some(result));");
|
||||
let remove_position = registry_publish.find("pending.remove(key);");
|
||||
assert!(matches!((notify_position, remove_position), (std::option::Option::Some(notify), std::option::Option::Some(remove)) if notify < remove));
|
||||
let inventory = match resources.split_once("impl RawTransactionIngestSourceInventory {") {
|
||||
std::option::Option::Some((_, tail)) => match tail.split_once("impl RawTransactionIngestSourceInventoryPublisher") {
|
||||
std::option::Option::Some((value, _)) => value,
|
||||
std::option::Option::None => "",
|
||||
},
|
||||
std::option::Option::None => "",
|
||||
};
|
||||
for required in [
|
||||
"checked_add(projection.hydration_pending())",
|
||||
"checked_add(projection.source_continuity_gap_total())",
|
||||
"checked_add(projection.source_reconnect_total())",
|
||||
"checked_add(projection.source_replay_attempt_total())",
|
||||
"source.inventory_entry_missing",
|
||||
"source.inventory_key_mismatch",
|
||||
"source.inventory_projection_missing",
|
||||
] {
|
||||
assert!(inventory.contains(required), "required pre.011 inventory guard missing: {required}");
|
||||
}
|
||||
assert!(!inventory.contains("saturating_add"), "pre.011 source inventory must not silently saturate aggregate counters");
|
||||
for source_marker in [
|
||||
"impl crate::RawTransactionIngestYellowstoneSource",
|
||||
"impl crate::RawTransactionIngestHeliusTransactionSource",
|
||||
"impl crate::RawTransactionIngestHttpBlockPollingSource",
|
||||
"impl crate::RawTransactionIngestStandardBlockSource",
|
||||
"impl crate::RawTransactionIngestStandardLogsSource",
|
||||
] {
|
||||
let tail = match resources.split_once(source_marker) {
|
||||
std::option::Option::Some((_, value)) => value,
|
||||
std::option::Option::None => "",
|
||||
};
|
||||
assert!(tail.contains("stop_receiver.changed()"), "pre.011 source lacks stop-preemptible wait: {source_marker}");
|
||||
}
|
||||
assert!(!runtime.contains("unbounded_channel"));
|
||||
assert!(!resources.contains("unbounded_channel"));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||
// version: 20
|
||||
// version: 21
|
||||
|
||||
//! External public-surface proofs for the RAW transaction ingest Worker foundation.
|
||||
|
||||
@@ -385,3 +385,27 @@ fn v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral()
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface() {
|
||||
let root = include_str!("../src/lib.rs");
|
||||
for forbidden in [
|
||||
"pub use self::runtime_resources::RawTransactionIngestSourceInventory",
|
||||
"pub use self::runtime_resources::RawTransactionIngestSourceInventoryPublisher",
|
||||
"pub use self::runtime_resources::RawTransactionIngestGlobalHydrationRegistry",
|
||||
"pub use self::runtime_resources::RawTransactionIngestHydrationLeaderGuard",
|
||||
"publish_and_remove",
|
||||
"source.inventory_key_mismatch",
|
||||
] {
|
||||
assert!(!root.contains(forbidden), "pre.011 private race-hardening implementation leaked publicly: {forbidden}");
|
||||
}
|
||||
assert_eq!(
|
||||
ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED,
|
||||
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "counter_exhausted")
|
||||
);
|
||||
assert_eq!(
|
||||
ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT,
|
||||
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "drain_timeout")
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||
// version: 18
|
||||
// version: 19
|
||||
|
||||
//! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
|
||||
//! Release-completeness canaries through the `pre.011` races/shutdown hardening tranche.
|
||||
|
||||
#[test]
|
||||
fn pre_010_production_module_inventory_is_exact() -> std::io::Result<()> {
|
||||
@@ -131,6 +131,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
||||
"v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private",
|
||||
"v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit",
|
||||
"v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted",
|
||||
"v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe",
|
||||
] {
|
||||
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
|
||||
}
|
||||
@@ -152,6 +153,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
||||
assert!(dependency_boundary.contains("v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_only"));
|
||||
assert!(dependency_boundary.contains("v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades"));
|
||||
assert!(dependency_boundary.contains("v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backend_free"));
|
||||
assert!(dependency_boundary.contains("v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades"));
|
||||
let public_api = include_str!("public_api.rs");
|
||||
assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
|
||||
assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
|
||||
@@ -166,6 +168,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
||||
assert!(public_api.contains("v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private"));
|
||||
assert!(public_api.contains("v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private"));
|
||||
assert!(public_api.contains("v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral"));
|
||||
assert!(public_api.contains("v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface"));
|
||||
assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
|
||||
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
||||
return;
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||
// version: 10
|
||||
// version: 11
|
||||
|
||||
struct ActiveTaskGuard {
|
||||
active: std::sync::Arc<std::sync::atomic::AtomicUsize>,
|
||||
@@ -899,3 +899,98 @@ async fn pre_009_drain_timeout_aborts_and_joins_all_owned_source_and_persistence
|
||||
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||
return;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
async fn v0_3_13_pre_011_source_counter_exhaustion_stays_terminal_and_is_not_collapsed_to_source_failed() {
|
||||
let settings = match settings("mainnet") {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return,
|
||||
};
|
||||
let handle = match super::start_foundation_with_source_spawner(
|
||||
settings,
|
||||
tokio::runtime::Handle::current(),
|
||||
std::option::Option::None,
|
||||
move |children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>, _stop_receiver, _admission_sender| {
|
||||
let _abort_handle = children.spawn(async move {
|
||||
return std::result::Result::Err(crate::counter_exhausted_error("source_reconnect_total"));
|
||||
});
|
||||
},
|
||||
) {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_) => return,
|
||||
};
|
||||
let source = handle.snapshot_source();
|
||||
let terminal = match handle.wait_terminal().await {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_) => return,
|
||||
};
|
||||
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||
let snapshot = source.current();
|
||||
assert_eq!(snapshot.source_failure_total(), 1);
|
||||
assert_eq!(snapshot.worker_snapshot().state(), terminal);
|
||||
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||
let terminal_sequence = snapshot.worker_snapshot().sequence();
|
||||
assert!(!handle.request_stop());
|
||||
let retained = source.current();
|
||||
assert_eq!(retained.worker_snapshot().state(), terminal);
|
||||
assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
|
||||
return;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
async fn v0_3_13_pre_011_drain_timeout_prevents_late_persistence_completion_after_terminal() {
|
||||
let settings = match settings_with_runtime_limits(1, 1, crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return,
|
||||
};
|
||||
let network = settings.network().clone();
|
||||
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||
let runtime_port: super::PersistencePort = port.clone();
|
||||
let source_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||
let source_active_for_task = std::sync::Arc::clone(&source_active);
|
||||
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||
settings,
|
||||
tokio::runtime::Handle::current(),
|
||||
std::option::Option::Some(runtime_port),
|
||||
move |children, _stop_receiver, admission_sender| {
|
||||
let _abort_handle = children.spawn(async move {
|
||||
let ingress = match runtime_ingress(&network, 81) {
|
||||
std::option::Option::Some(value) => value,
|
||||
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||
};
|
||||
if admission_sender.send(ingress).await.is_err() {
|
||||
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||
}
|
||||
let _guard = ActiveTaskGuard::new(source_active_for_task);
|
||||
return std::future::pending::<ksp_core_lib::Result<()>>().await;
|
||||
});
|
||||
},
|
||||
) {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_) => return,
|
||||
};
|
||||
assert!(wait_for_max_active(&port, 1).await);
|
||||
assert!(wait_for_active_count(&source_active, 1).await);
|
||||
assert!(handle.request_stop());
|
||||
let terminal = match handle.wait_terminal().await {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_) => return,
|
||||
};
|
||||
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT));
|
||||
let snapshot = handle.snapshot_source().current();
|
||||
let terminal_sequence = snapshot.worker_snapshot().sequence();
|
||||
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||
assert_eq!(source_active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||
port.release();
|
||||
for _ in 0..64 {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||
let retained = handle.snapshot_source().current();
|
||||
assert_eq!(retained.worker_snapshot().state(), terminal);
|
||||
assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
|
||||
assert_eq!(retained.in_flight_persistence(), 0);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||
// version: 22
|
||||
// version: 23
|
||||
|
||||
fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
|
||||
return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
|
||||
@@ -971,13 +971,13 @@ fn v0_3_13_pre_007_source_inventory_aggregates_frontier_and_lifecycle_conservati
|
||||
source_key: [1_u8; 32],
|
||||
};
|
||||
let second = super::RawTransactionIngestSourceInventoryPublisher { aggregate_sender, entry_index: 1, inventory, source_key: [2_u8; 32] };
|
||||
first.publish(
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(50), std::option::Option::Some(45)).with_source_continuity(
|
||||
std::option::Option::Some(crate::RawTransactionIngestSourceState::Active),
|
||||
1,
|
||||
2,
|
||||
0,
|
||||
),
|
||||
assert!(
|
||||
first
|
||||
.publish(
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(50), std::option::Option::Some(45))
|
||||
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 2, 0),
|
||||
)
|
||||
.is_ok()
|
||||
);
|
||||
let first_aggregate = *aggregate_receiver.borrow();
|
||||
assert_eq!(first_aggregate.hydration_pending(), 1);
|
||||
@@ -988,13 +988,13 @@ fn v0_3_13_pre_007_source_inventory_aggregates_frontier_and_lifecycle_conservati
|
||||
assert_eq!(first_aggregate.source_active(), 1);
|
||||
assert_eq!(first_aggregate.source_reconnecting(), 0);
|
||||
assert_eq!(first_aggregate.source_failed(), 0);
|
||||
second.publish(
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(42), std::option::Option::Some(40)).with_source_continuity(
|
||||
std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting),
|
||||
3,
|
||||
4,
|
||||
1,
|
||||
),
|
||||
assert!(
|
||||
second
|
||||
.publish(
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(42), std::option::Option::Some(40))
|
||||
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting), 3, 4, 1),
|
||||
)
|
||||
.is_ok()
|
||||
);
|
||||
let aggregate = *aggregate_receiver.borrow();
|
||||
assert_eq!(aggregate.hydration_pending(), 3);
|
||||
@@ -3146,3 +3146,204 @@ async fn v0_3_13_pre_009_global_hydration_permit_and_leader_cleanup_are_bounded_
|
||||
assert!(matches!(resubscribed, std::result::Result::Ok((true, _))));
|
||||
return;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v0_3_13_pre_011_source_inventory_rejects_stale_identity_and_counter_exhaustion() {
|
||||
let mut identity_inventory = super::RawTransactionIngestSourceInventory::new(std::vec![[1_u8; 32]]);
|
||||
let mismatch = identity_inventory.update(0, [2_u8; 32], crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||
let mismatch = match mismatch {
|
||||
std::result::Result::Ok(()) => return,
|
||||
std::result::Result::Err(value) => value,
|
||||
};
|
||||
assert_eq!(mismatch.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||
assert_eq!(mismatch.context()[0].value(), "source.inventory_key_mismatch");
|
||||
let missing = identity_inventory.update(1, [1_u8; 32], crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||
let missing = match missing {
|
||||
std::result::Result::Ok(()) => return,
|
||||
std::result::Result::Err(value) => value,
|
||||
};
|
||||
assert_eq!(missing.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||
assert_eq!(missing.context()[0].value(), "source.inventory_entry_missing");
|
||||
let mut pending_inventory = super::RawTransactionIngestSourceInventory::new(std::vec![[3_u8; 32], [4_u8; 32]]);
|
||||
assert!(
|
||||
pending_inventory
|
||||
.update(
|
||||
0,
|
||||
[3_u8; 32],
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(usize::MAX, std::option::Option::Some(10), std::option::Option::None),
|
||||
)
|
||||
.is_ok()
|
||||
);
|
||||
assert!(
|
||||
pending_inventory
|
||||
.update(1, [4_u8; 32], crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(11), std::option::Option::None),)
|
||||
.is_ok()
|
||||
);
|
||||
let pending_error = match pending_inventory.aggregate() {
|
||||
std::result::Result::Ok(_) => return,
|
||||
std::result::Result::Err(value) => value,
|
||||
};
|
||||
assert_eq!(pending_error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED);
|
||||
assert_eq!(pending_error.context()[0].value(), "hydration_pending");
|
||||
let mut continuity_inventory = super::RawTransactionIngestSourceInventory::new(std::vec![[5_u8; 32], [6_u8; 32]]);
|
||||
assert!(
|
||||
continuity_inventory
|
||||
.update(
|
||||
0,
|
||||
[5_u8; 32],
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(20), std::option::Option::None)
|
||||
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), u64::MAX, 0, 0),
|
||||
)
|
||||
.is_ok()
|
||||
);
|
||||
assert!(
|
||||
continuity_inventory
|
||||
.update(
|
||||
1,
|
||||
[6_u8; 32],
|
||||
crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(21), std::option::Option::None)
|
||||
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 0, 0),
|
||||
)
|
||||
.is_ok()
|
||||
);
|
||||
let continuity_error = match continuity_inventory.aggregate() {
|
||||
std::result::Result::Ok(_) => return,
|
||||
std::result::Result::Err(value) => value,
|
||||
};
|
||||
assert_eq!(continuity_error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED);
|
||||
assert_eq!(continuity_error.context()[0].value(), "source_reconnect_total");
|
||||
return;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
async fn v0_3_13_pre_011_aborted_hydration_leader_notifies_followers_and_releases_registry_key() {
|
||||
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_) => return,
|
||||
};
|
||||
let registry = std::sync::Arc::new(super::RawTransactionIngestGlobalHydrationRegistry::new(2, 1));
|
||||
let key = super::RawTransactionIngestHydrationKey {
|
||||
commitment: ksp_onchain_transport_lib::SolanaCommitment::Confirmed.as_str(),
|
||||
network,
|
||||
signature: ksp_store_lib::RawTransactionSignature::new([91_u8; 64]),
|
||||
};
|
||||
let subscribed = registry.subscribe_or_lead(&key);
|
||||
let (leader, follower) = match subscribed {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_) => return,
|
||||
};
|
||||
assert!(leader);
|
||||
let task_registry = std::sync::Arc::clone(®istry);
|
||||
let task_key = key.clone();
|
||||
let leader_task = tokio::spawn(async move {
|
||||
let _guard = super::RawTransactionIngestHydrationLeaderGuard::new(task_registry, task_key);
|
||||
return std::future::pending::<()>().await;
|
||||
});
|
||||
tokio::task::yield_now().await;
|
||||
leader_task.abort();
|
||||
let _joined = leader_task.await;
|
||||
let published = follower.borrow().clone();
|
||||
match published {
|
||||
std::option::Option::Some(super::RawTransactionIngestSharedHydrationResult::Failed(code)) => {
|
||||
assert_eq!(code, crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED);
|
||||
},
|
||||
_ => return,
|
||||
}
|
||||
assert!(matches!(registry.subscribe_or_lead(&key), std::result::Result::Ok((true, _))));
|
||||
return;
|
||||
}
|
||||
|
||||
struct Pre011SourceActiveGuard {
|
||||
active: std::sync::Arc<std::sync::atomic::AtomicUsize>,
|
||||
}
|
||||
|
||||
impl Pre011SourceActiveGuard {
|
||||
fn new(active: std::sync::Arc<std::sync::atomic::AtomicUsize>) -> Self {
|
||||
active.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||
return Self { active };
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::Drop for Pre011SourceActiveGuard {
|
||||
fn drop(&mut self) {
|
||||
self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
async fn v0_3_13_pre_011_aborting_outer_source_supervisor_aborts_nested_source_tasks() {
|
||||
let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||
let (_stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||
let (source_stop_sender, _source_stop_receiver) = tokio::sync::watch::channel(false);
|
||||
let mut children = tokio::task::JoinSet::new();
|
||||
let task_active = std::sync::Arc::clone(&active);
|
||||
let _abort_handle = children.spawn(async move {
|
||||
let _guard = Pre011SourceActiveGuard::new(task_active);
|
||||
return std::future::pending::<ksp_core_lib::Result<()>>().await;
|
||||
});
|
||||
let supervisor = tokio::spawn(super::supervise_live_source_tasks(stop_receiver, source_stop_sender, children));
|
||||
for _ in 0..64 {
|
||||
if active.load(std::sync::atomic::Ordering::Acquire) == 1 {
|
||||
break;
|
||||
}
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||
supervisor.abort();
|
||||
let _joined = supervisor.await;
|
||||
for _ in 0..64 {
|
||||
if active.load(std::sync::atomic::Ordering::Acquire) == 0 {
|
||||
break;
|
||||
}
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||
return;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
async fn v0_3_13_pre_011_stop_racing_ready_source_fault_preserves_fault_and_joins_sibling() {
|
||||
let sibling_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||
let fault_gate = std::sync::Arc::new(tokio::sync::Notify::new());
|
||||
let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||
let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
|
||||
let mut children = tokio::task::JoinSet::new();
|
||||
let task_fault_gate = std::sync::Arc::clone(&fault_gate);
|
||||
let _fault_abort_handle = children.spawn(async move {
|
||||
task_fault_gate.notified().await;
|
||||
return std::result::Result::Err(crate::runtime_error("test.pre_011_ready_source_fault"));
|
||||
});
|
||||
let sibling_active_for_task = std::sync::Arc::clone(&sibling_active);
|
||||
let mut sibling_stop_receiver = source_stop_receiver.clone();
|
||||
let _sibling_abort_handle = children.spawn(async move {
|
||||
let _guard = Pre011SourceActiveGuard::new(sibling_active_for_task);
|
||||
loop {
|
||||
let changed = sibling_stop_receiver.changed().await;
|
||||
if changed.is_err() || *sibling_stop_receiver.borrow() {
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
}
|
||||
});
|
||||
let supervisor = tokio::spawn(super::supervise_live_source_tasks(stop_receiver, source_stop_sender, children));
|
||||
for _ in 0..64 {
|
||||
if sibling_active.load(std::sync::atomic::Ordering::Acquire) == 1 {
|
||||
break;
|
||||
}
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
assert_eq!(sibling_active.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||
stop_sender.send_replace(true);
|
||||
fault_gate.notify_one();
|
||||
let result = match supervisor.await {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_) => return,
|
||||
};
|
||||
let error = match result {
|
||||
std::result::Result::Ok(()) => return,
|
||||
std::result::Result::Err(value) => value,
|
||||
};
|
||||
assert_eq!(error.context()[0].value(), "test.pre_011_ready_source_fault");
|
||||
assert_eq!(sibling_active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||
return;
|
||||
}
|
||||
|
||||
62
deltas/0.3.13/pre.011.md
Normal file
62
deltas/0.3.13/pre.011.md
Normal file
@@ -0,0 +1,62 @@
|
||||
<!-- file: deltas/0.3.13/pre.011.md -->
|
||||
<!-- version: 1 -->
|
||||
|
||||
# Delta 0.3.13-pre.011 — races/shutdown hardening
|
||||
|
||||
## Objet
|
||||
|
||||
Fermer les races stop/fault et l'ownership des tâches source/hydration/persistence avant la tranche de completeness/security cross-layer, sans modifier Transport, le graphe de dépendances ou les politiques de convergence/fairness déjà acquises.
|
||||
|
||||
## Shutdown et ownership
|
||||
|
||||
Le supervisor conserve l'ordre terminal explicite : signal de stop source, passage Worker à `Stopping`, drain borné, abort+join si la deadline expire, puis publication du terminal. L'abort du wrapper multi-source détruit son `JoinSet` privé et annule donc aussi les tâches source imbriquées avant retour.
|
||||
|
||||
Une faute source déjà observée n'est pas masquée par un stop concurrent. Le `drain_timeout` reste prioritaire lorsqu'une récupération bornée dépasse réellement sa deadline. Une persistence encore bloquée à cette échéance est abortée et jointe ; elle ne peut pas publier de succès tardif après le snapshot terminal.
|
||||
|
||||
## Hydration partagée
|
||||
|
||||
`RawTransactionIngestGlobalHydrationRegistry::publish_and_remove` notifie désormais les followers sous le mutex de registry avant de retirer la clé. Une nouvelle génération de leader ne peut donc pas s'intercaler entre le retrait de la clé et la notification des followers existants.
|
||||
|
||||
Le leader guard conserve son cleanup fail-closed : si un leader est aborté avant publication normale, les followers reçoivent une faute source sûre et la clé redevient ensuite disponible.
|
||||
|
||||
## Inventaire et compteurs
|
||||
|
||||
L'agrégation multi-source remplace les additions saturantes par `checked_add` pour `hydration_pending`, `source_reconnect_total`, `source_replay_attempt_total` et `source_continuity_gap_total`. Un overflow devient `worker_raw_transaction_ingest.counter_exhausted`.
|
||||
|
||||
Une mise à jour d'inventaire avec index absent, `source_key` incohérente ou projection absente échoue explicitement par `runtime_invalid` au lieu d'être ignorée. Lorsqu'une source remonte `counter_exhausted`, le supervisor préserve ce code terminal et ne le remappe pas en `source_failed`.
|
||||
|
||||
## Preuves ajoutées
|
||||
|
||||
Les tests couvrent l'identité/index d'inventaire invalides, les overflows d'agrégats, l'abort d'un leader hydration, l'abort du supervisor externe et de ses enfants imbriqués, la race stop/source fault, la conservation de `counter_exhausted`, le drain timeout sans completion tardive, l'ordre terminal et l'atomicité notification-avant-retrait.
|
||||
|
||||
Les canaris externes verrouillent l'absence de queue non bornée, backend Store direct, client Transport secondaire ou nouvelle surface publique d'identité source.
|
||||
|
||||
## Frontière
|
||||
|
||||
Aucun changement de Transport, Store API, dépendance, feature, convergence, quotas/fairness, snapshot public ou politique de failover. La completeness/security exhaustive reste `pre.012`; le gate technique/live complet et les graphes restent `pre.013`.
|
||||
|
||||
## Gate reçu avant cette tranche
|
||||
|
||||
Le gate opérateur `0.3.13-pre.010` est vert pour toutes les commandes exécutées : fmt, audits, Markdown `340/848`, workspace check, Clippy strict, puis Worker `100` unit + `4` cross-layer + `17` dependency-boundary + `24` hardening + `18` public-api + `4` release-completeness et `0` doc-test.
|
||||
|
||||
Le Transport n'était pas modifié par `pre.010` et n'a pas été rejoué. Aucun `cargo tree` n'était requis ni exécuté.
|
||||
|
||||
## Validation locale d'assemblage
|
||||
|
||||
Le toolchain Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Aucun résultat Cargo post-modification n'est déclaré PASS localement. Les audits statiques et contrôles de packaging effectivement exécutés sont les seules preuves locales consignées pour cette livraison.
|
||||
|
||||
Les audits statiques finaux d'assemblage sont clean : règles Rust générales, export completeness `0`, règles KSP, Markdown `340/849`, et inventaire normatif `489/489` sans doublon.
|
||||
|
||||
## Gate opérateur avant pre.012
|
||||
|
||||
```bash
|
||||
cargo fmt --all
|
||||
cargo fmt --all -- --check
|
||||
python3 scripts/audit_rust_workspace_rules.py
|
||||
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||
cargo check --workspace
|
||||
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||
```
|
||||
|
||||
Aucun `cargo tree` n'est requis ; aucune dépendance ni feature n'a changé. Le Transport n'est pas modifié par cette tranche de hardening shutdown/races.
|
||||
@@ -1,5 +1,5 @@
|
||||
<!-- file: docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md -->
|
||||
<!-- version: 11 -->
|
||||
<!-- version: 12 -->
|
||||
|
||||
# Plan v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source RawTransaction
|
||||
|
||||
@@ -1479,3 +1479,80 @@ cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||
```
|
||||
|
||||
Aucun `cargo tree` n'est requis : `pre.010` ne modifie ni dépendance ni feature. Le Transport n'est pas modifié par cette tranche snapshot/health.
|
||||
|
||||
## 80. Gate opérateur pre.010 reçu
|
||||
|
||||
Le 10 septembre 2026, le gate intermédiaire communiqué pour `0.3.13-pre.010` est entièrement vert pour les commandes exécutées : fmt, audits Rust, Markdown `340/848`, workspace check, Clippy strict, puis Worker `100` unit + `4` cross-layer + `17` dependency-boundary + `24` hardening + `18` public-api + `4` release-completeness et `0` doc-test.
|
||||
|
||||
Le Transport n'était pas modifié par `pre.010` et n'a pas été rejoué. Aucun `cargo tree` n'était requis ni exécuté.
|
||||
|
||||
## 81. Races et shutdown pre.011
|
||||
|
||||
La fermeture `pre.011` conserve l'ownership déjà établi mais rend les transitions concurrentes explicitement fail-closed :
|
||||
|
||||
```text
|
||||
stop externe -> signal stop source avant passage Worker à Stopping
|
||||
source fault déjà observée -> conservée face au stop concurrent
|
||||
shutdown_drain_timeout expiré -> drain_timeout terminal prioritaire
|
||||
abort du wrapper multi-source -> drop du JoinSet interne -> abort des enfants imbriqués
|
||||
abort persistence -> join avant publication terminale
|
||||
leader hydration abandonné -> followers notifiés Failed + clé réouvrable
|
||||
publication hydration -> notification followers avant retrait de la clé sous le même mutex
|
||||
```
|
||||
|
||||
Le Worker ne publie donc pas son terminal avant la récupération des tâches qu'il possède. Une tâche persistence libérée après un `drain_timeout` ne peut pas réapparaître comme succès tardif ni modifier le dernier snapshot terminal.
|
||||
|
||||
## 82. Compteurs et inventaire fail-closed pre.011
|
||||
|
||||
L'inventaire multi-source n'utilise plus d'addition saturante pour agréger :
|
||||
|
||||
```text
|
||||
hydration_pending
|
||||
source_reconnect_total
|
||||
source_replay_attempt_total
|
||||
source_continuity_gap_total
|
||||
```
|
||||
|
||||
Chaque agrégat utilise `checked_add`; un overflow devient `worker_raw_transaction_ingest.counter_exhausted`. Une publication d'inventaire portant un index absent, une `source_key` incohérente ou une projection absente est rejetée par `runtime_invalid` au lieu d'être ignorée.
|
||||
|
||||
Si une source remonte précisément `counter_exhausted`, le supervisor conserve ce code terminal au lieu de le rabattre sur le code générique `source_failed`.
|
||||
|
||||
## 83. Preuves déterministes pre.011
|
||||
|
||||
Les preuves ajoutées couvrent au minimum :
|
||||
|
||||
```text
|
||||
mauvaise source_key -> runtime_invalid fail-closed
|
||||
index inventaire absent -> runtime_invalid fail-closed
|
||||
overflow hydration_pending agrégé -> counter_exhausted
|
||||
overflow reconnect agrégé -> counter_exhausted
|
||||
abort leader hydration -> follower Failed + clé libérée
|
||||
abort supervisor externe -> enfants source imbriqués abortés
|
||||
stop concurrent + source fault prête -> fault conservée + sibling rejoint
|
||||
counter_exhausted source -> terminal counter_exhausted non remappé
|
||||
shutdown drain timeout -> aucune persistence tardive après terminal
|
||||
ordre stop -> Stopping -> drain -> terminal verrouillé par canari
|
||||
publication hydration atomique notification-avant-remove verrouillée par canari
|
||||
```
|
||||
|
||||
Les canaris externes vérifient que ce durcissement reste privé au Worker, réutilise uniquement les façades existantes et n'ajoute aucune identité source/provider à l'API publique.
|
||||
|
||||
## 84. Frontière de tranche pre.011
|
||||
|
||||
`pre.011` ne change ni la politique de convergence, ni les quotas/fairness `pre.009`, ni la projection health `pre.010`. Il n'ajoute pas de failover, de scheduler, de retry Worker supplémentaire, de backend Store direct ou de client Transport secondaire.
|
||||
|
||||
La completeness/security exhaustive cross-layer appartient à `pre.012`; le gate technique complet, les graphes et les smokes accessibles appartiennent à `pre.013`.
|
||||
|
||||
## 85. Gate opérateur requis avant pre.012
|
||||
|
||||
```bash
|
||||
cargo fmt --all
|
||||
cargo fmt --all -- --check
|
||||
python3 scripts/audit_rust_workspace_rules.py
|
||||
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||
cargo check --workspace
|
||||
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||
```
|
||||
|
||||
Aucun `cargo tree` n'est requis : `pre.011` ne modifie ni dépendance ni feature. Le Transport n'est pas modifié par cette tranche de hardening shutdown/races.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
<!-- file: docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md -->
|
||||
<!-- version: 11 -->
|
||||
<!-- version: 12 -->
|
||||
|
||||
# Validation v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source
|
||||
|
||||
@@ -1485,3 +1485,98 @@ cargo check --workspace
|
||||
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||
```
|
||||
|
||||
## 82. Gate complet pre.010 reçu
|
||||
|
||||
Preuve opérateur du 10 septembre 2026 :
|
||||
|
||||
```text
|
||||
fmt : PASS
|
||||
audits Rust : clean / export completeness 0
|
||||
Markdown : clean, 340 tables / 848 files
|
||||
cargo check --workspace : PASS
|
||||
clippy strict : PASS
|
||||
Worker : 100 unit + 4 cross-layer + 17 dependency-boundary + 24 hardening + 18 public-api + 4 release-completeness, 0 échec
|
||||
Worker doc-tests : 0
|
||||
```
|
||||
|
||||
Le Transport n'était pas modifié par `pre.010` et n'a pas été rejoué. Aucun `cargo tree` n'est revendiqué.
|
||||
|
||||
## 83. Durcissement races/shutdown pre.011 matérialisé
|
||||
|
||||
La tranche ferme les transitions concurrentes suivantes sans changer les frontières de couche :
|
||||
|
||||
```text
|
||||
stop vs source fault
|
||||
abort supervisor vs enfants source imbriqués
|
||||
abort leader hydration vs followers
|
||||
publication hydration vs arrivée d'un nouveau leader
|
||||
shutdown deadline vs persistence bloquée
|
||||
counter exhaustion vs remapping générique source_failed
|
||||
publication inventaire vs identité/index incohérent
|
||||
```
|
||||
|
||||
`publish_and_remove` notifie les followers sous le mutex de registry avant de retirer la clé. Le retrait rend ensuite seulement la clé disponible à une nouvelle génération de leader. Le `Drop` du leader guard reste le chemin fail-closed pour un leader aborté avant publication normale.
|
||||
|
||||
## 84. Agrégation et terminalité vérifiées pre.011
|
||||
|
||||
L'inventaire agrégé utilise désormais `checked_add` pour `hydration_pending`, `source_reconnect_total`, `source_replay_attempt_total` et `source_continuity_gap_total`. Aucun clamp par `saturating_add` n'est admis dans cette agrégation.
|
||||
|
||||
Un overflow devient `worker_raw_transaction_ingest.counter_exhausted`. Ce code est préservé par le supervisor lorsqu'il provient d'une source, au lieu d'être converti en `worker_raw_transaction_ingest.source_failed`.
|
||||
|
||||
Les mises à jour d'inventaire invalides échouent explicitement sur des codes internes stables :
|
||||
|
||||
```text
|
||||
source.inventory_entry_missing
|
||||
source.inventory_key_mismatch
|
||||
source.inventory_projection_missing
|
||||
```
|
||||
|
||||
Aucune valeur source/provider n'est recopiée dans l'erreur.
|
||||
|
||||
## 85. Preuves unitaires et externes pre.011
|
||||
|
||||
Les tests ajoutés prouvent :
|
||||
|
||||
```text
|
||||
stale identity/index inventaire rejetés
|
||||
counter overflow agrégé détecté
|
||||
leader hydration aborté -> follower réveillé + registry nettoyée
|
||||
abort du supervisor source -> enfant imbriqué détruit
|
||||
stop et fault simultanés -> fault non masquée et sibling rejoint
|
||||
drain timeout -> persistence abortée/jointe et aucune complétion tardive
|
||||
terminal counter_exhausted retenu après source completion
|
||||
```
|
||||
|
||||
Les canaris `dependency_boundary`, `hardening`, `public_api` et `release_completeness` verrouillent l'ownership, l'ordre terminal, l'atomicité notification/retrait, l'absence de `unbounded_channel`, l'absence de backend/client direct et l'absence de nouvelle surface publique d'identité source.
|
||||
|
||||
## 86. Non-claims pre.011
|
||||
|
||||
```text
|
||||
pas de changement Transport
|
||||
pas de nouvelle dépendance/feature
|
||||
pas de retry/failover supplémentaire
|
||||
pas de politique de réparation historique
|
||||
pas de changement convergence/fairness
|
||||
pas de nouvelle surface snapshot publique
|
||||
pas encore de completeness/security cross-layer exhaustive pre.012
|
||||
pas encore de gate technique/live pre.013
|
||||
```
|
||||
|
||||
## 87. Validation locale d'assemblage et gate avant pre.012
|
||||
|
||||
Le toolchain Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-modification restent `NON EXÉCUTÉ LOCAL`.
|
||||
|
||||
Gate opérateur demandé :
|
||||
|
||||
```bash
|
||||
cargo fmt --all
|
||||
cargo fmt --all -- --check
|
||||
python3 scripts/audit_rust_workspace_rules.py
|
||||
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||
cargo check --workspace
|
||||
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||
```
|
||||
|
||||
Aucun `cargo tree` n'est requis et le Transport n'a pas besoin d'être rejoué pour cette tranche qui ne modifie ni le graphe ni `ksp-onchain-transport-lib`.
|
||||
|
||||
Reference in New Issue
Block a user