diff --git a/Cargo.toml b/Cargo.toml index 892bce2..4af63e5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,12 +1,12 @@ # file: Cargo.toml -# version: 547 +# version: 548 [workspace] resolver = "3" members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api", "crates/ksp-worker-raw-transaction-ingest-lib"] [workspace.package] -version = "0.3.13-pre.4.fix.1" +version = "0.3.13-pre.5" edition = "2024" license = "MIT" repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project" diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/README.md b/crates/ksp-worker-raw-transaction-ingest-lib/README.md index 6cd0d57..e49a9e7 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/README.md +++ b/crates/ksp-worker-raw-transaction-ingest-lib/README.md @@ -1,5 +1,5 @@ - + # ksp-worker-raw-transaction-ingest-lib @@ -13,7 +13,7 @@ RawTransactionIngestWorker::start RawTransactionIngestWorker::start_with_runtime_resources -> même runtime + une source productive supervisée - Yellowstone, WS standard logsSubscribe ou WS standard blockSubscribe + Yellowstone, WS standard logsSubscribe, WS standard blockSubscribe ou Helius transactionSubscribe + hydration HTTP getTransaction lorsque la source produit une référence ``` @@ -39,6 +39,12 @@ Solana standard WS blockSubscribe -> qualification explicite Legacy / V0 / V1 -> matériau Common RAW direct par transaction du bloc +Helius transactionSubscribe + -> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false + -> signature + slot + transactionIndex uniquement dans le signal Worker + -> coalescence bornée par (network, signature, commitment) + -> HTTP getTransaction observed avant Common RAW + les chemins productifs -> ksp-raw-transaction-lib -> admission centrale bornée @@ -70,7 +76,7 @@ La construction est sans I/O et refuse notamment : - une provenance provider/endpoint non représentable ; - l'absence d'une route HTTP compatible pour `getTransaction` sur le même réseau. -Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques. `pre.004` sait exécuter une source unique Yellowstone, Standard Logs ou Standard Block ; une activation simultanée de plusieurs sources reste rejetée avant spawn jusqu'au supervisor dédié. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés. +Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques. `pre.005` sait exécuter une source unique Yellowstone, Standard Logs, Standard Block ou Helius Transaction ; une activation simultanée de plusieurs sources reste rejetée avant spawn jusqu'au supervisor dédié. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés. ## Contrat de source Standard Logs + HTTP @@ -104,6 +110,24 @@ La qualification est fermée : une version omise/nulle ou supérieure à `1`, un Pour un bloc multi-transaction, le slot n'est projeté settled qu'après l'admission réussie de toutes ses transactions. Cette voie RAW-direct n'incrémente pas `hydration_pending`; un stop ou une erreur au milieu du bloc ne produit aucune progression artificielle. Reconnect, resubscribe et backpressure WebSocket restent possédés par Transport. +## Contrat de source Helius Transaction + HTTP + +`RawTransactionIngestHeliusTransactionSource::new` reçoit : + +```text +WsEndpointSettings kind helius_laserstream +HeliusTransactionSubscribeFilter +SolanaCommitment Confirmed ou Finalized +HttpTransportPool +HttpRoleName d'hydration +``` + +La construction est sans I/O. Elle réutilise le contrat Transport existant et impose `Full`, `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`. Le Worker ne lit pas Config, ne lit pas `KSP_SECRET_HELIUS_API_KEY` et ne code aucun tier provider ; l'URL résolue et le credential restent dans l'endpoint Transport fourni par le caller. + +Au runtime, `HeliusLaserStreamWsSession::connect` puis `transaction_subscribe` sont utilisés. Seule une notification `Full` conforme au mode demandé est admise ; une forme `Signature`, `Unknown` ou future devient une faute source sûre. Le payload Helius `transaction` n'est jamais copié dans l'état Worker : la projection conserve uniquement signature, slot et transaction index, puis réutilise le coordinateur d'hydration commun `getTransaction observed`. + +La clé logique Helius inclut réseau, identités provider/endpoint sûres, commitment et empreinte privée du filtre. Les listes de pubkeys du filtre sont normalisées avant hash afin que leur ordre ne crée pas artificiellement deux sources logiques ; le rôle HTTP d'hydration reste exclu de l'identité live. + ## Runtime et lifecycle Le Worker s'exécute sur le runtime Tokio courant du caller. Il ne crée pas de runtime global et n'expose aucun `JoinHandle` public. @@ -121,7 +145,7 @@ Le shutdown est borné par `shutdown_drain_timeout`. Les tâches source, hydrati La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes subissent la backpressure ; aucune queue non bornée ni silent drop n'est autorisé. -Le Worker possède un coordinateur d'hydration source-neutral borné, réutilisé par Yellowstone et Standard Logs. Standard Block n'entre pas dans ce coordinateur lorsqu'une transaction est direct-qualified : +Le Worker possède un coordinateur d'hydration source-neutral borné, réutilisé par Yellowstone, Standard Logs et Helius Transaction. Standard Block n'entre pas dans ce coordinateur lorsqu'une transaction est direct-qualified : ```text in-flight hydration <= persistence_concurrency @@ -240,7 +264,7 @@ La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Sto La verticale actuelle ne possède pas : - plusieurs sources productives simultanées dans `RawTransactionIngestRuntimeResources` ; -- source `blockSubscribe`, Helius `transactionSubscribe` ou HTTP polling Worker ; +- source HTTP live polling Worker ; - sélection Config interne au Worker ; - checkpoint persistent de processing frontier ; - campagne de réparation historique automatique ; diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md b/crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md index 9f836f6..4e4ab21 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md +++ b/crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md @@ -1,5 +1,5 @@ - + # Utilisation de ksp-worker-raw-transaction-ingest-lib @@ -101,7 +101,7 @@ let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWo }; ``` -Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. Une exécution productive accepte une source unique Yellowstone, Standard Logs ou Standard Block ; une collection de plusieurs sources est validable à la composition mais reste rejetée avant spawn tant que la supervision simultanée n'est pas disponible. +Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. Une exécution productive accepte une source unique Yellowstone, Standard Logs, Standard Block ou Helius Transaction ; une collection de plusieurs sources est validable à la composition mais reste rejetée avant spawn tant que la supervision simultanée n'est pas disponible. ### Source Standard Logs productive @@ -151,6 +151,36 @@ fn standard_block_runtime_resources( Le Worker demande `Base64`, `Full`, `maxSupportedTransactionVersion = 1` et `showRewards = false`. Une transaction n'est RAW-direct que si sa version est explicitement `Legacy`, `0` ou `1`. Une version omise/nulle ou supérieure, `block: null`, une erreur de bloc, un champ transactions absent/nul ou une transaction non Base64 provoque une faute sûre ; ces cas ne sont jamais assimilés à une progression vide. +### Source Helius Transaction productive + +Une source Helius LaserStream `transactionSubscribe` se compose avec le même pattern caller-owned : + +```rust +fn helius_transaction_runtime_resources( + ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings, + filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter, + commitment: ksp_onchain_transport_lib::SolanaCommitment, + http_pool: ksp_onchain_transport_lib::HttpTransportPool, + hydration_role: ksp_onchain_transport_lib::HttpRoleName, +) -> ksp_core_lib::Result { + let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new( + ws_endpoint, + filter, + commitment, + http_pool, + hydration_role, + ) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(error) => return std::result::Result::Err(error), + }; + return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source(source)); +} +``` + +`ws_endpoint` doit être un endpoint Transport de kind `helius_laserstream`. Le caller supérieur résout éventuellement `KSP_SECRET_HELIUS_API_KEY` via Config avant de construire l'endpoint ; le Worker ne lit jamais l'environnement ni Config. Le commitment est limité à `Confirmed`/`Finalized` et la route HTTP doit supporter `getTransaction` sur le même réseau. + +Le Worker demande la forme Helius `Full` avec `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`, mais ne fait pas confiance au nested payload pour construire directement le Common RAW. Il conserve seulement signature/slot/index et hydrate par `getTransaction observed`. Une notification d'une autre forme est fail-closed. + ## Préparer la source Yellowstone La `YellowstoneSubscribeRequest` doit : @@ -174,6 +204,7 @@ Yellowstone TransactionStatus -> signal -> HTTP getTransaction -> Common RAW -> Yellowstone Block -> un signal par transaction -> HTTP getTransaction -> Common RAW -> admission Standard WS logsSubscribe -> context.slot + signature -> HTTP getTransaction -> Common RAW -> admission Standard WS blockSubscribe -> Full/Base64 Legacy|V0|V1 -> Common RAW direct par transaction -> admission +Helius transactionSubscribe -> Full envelope -> signature/slot/index -> HTTP getTransaction -> Common RAW -> admission Yellowstone BlockMeta -> continuity-only Yellowstone Slot -> continuity-only Yellowstone Account/Ping/Pong/Entry -> sans RAW Transaction dans cette verticale @@ -316,7 +347,7 @@ Le pattern attendu est : ```text Config / application / service owner -> résout endpoints, credentials et rôles - -> construit une source Transport Yellowstone, Standard Logs ou Standard Block + -> construit une source Transport Yellowstone, Standard Logs, Standard Block ou Helius Transaction -> construit HttpTransportPool + hydration role seulement pour les sources hydratées -> construit le Store -> construit RawTransactionIngestRuntimeResources diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs b/crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs index 5c75f7f..e175233 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs +++ b/crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs @@ -1,5 +1,5 @@ // file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs -// version: 21 +// version: 22 #![warn(missing_docs)] #![deny(unreachable_pub)] @@ -10,10 +10,11 @@ //! This tranche owns the concrete Worker family identity, validated technical settings //! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also //! owns bounded source-neutral admission, common RAW canonicalization/assembly and backend-neutral -//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. `pre.004` keeps the bounded 1..32 caller-composed -//! aggregate and adds a productive standard Solana `blockSubscribe` direct RAW source beside Yellowstone and Standard Logs while simultaneous multi-source -//! activation remains gated until the dedicated supervisor tranche. Standard Logs and Yellowstone reference paths converge into one source-neutral hydration -//! coordinator contract; qualified Standard Block Legacy/V0/V1 transactions enter the existing central admission path directly. Yellowstone +//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. `pre.005` keeps the bounded 1..32 caller-composed +//! aggregate and adds a productive Helius `transactionSubscribe` + HTTP hydration source beside Yellowstone, Standard Logs and Standard Block while +//! simultaneous multi-source activation remains gated until the dedicated supervisor tranche. Helius Full notifications, Standard Logs and Yellowstone +//! reference paths converge into one source-neutral hydration coordinator contract; qualified Standard Block Legacy/V0/V1 transactions enter the existing +//! central admission path directly. Yellowstone //! Transaction/TransactionStatus/Block and standard logs notifications become signature/slot references; BlockMeta/Slot remain continuity-only signals. //! Hydration is coalesced by network/signature/commitment under bounded in-flight and pending budgets. A bounded run-local processing frontier projects //! hydration pending, oldest pending slot and highest unblocked actually observed slot. The productive source also projects safe Transport reconnect/replay @@ -52,6 +53,8 @@ pub use self::runtime::RawTransactionIngestTerminalFuture; pub use self::runtime::RawTransactionIngestWorker; /// Maximum number of logical live sources accepted by one runtime-resource aggregate. pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES; +/// Validated Helius `transactionSubscribe` + HTTP hydration source contract owned by the continuous RAW transaction ingest Worker. +pub use self::runtime_resources::RawTransactionIngestHeliusTransactionSource; /// Caller-composed bounded runtime resources for supported continuous RAW transaction live-source families. pub use self::runtime_resources::RawTransactionIngestRuntimeResources; /// Validated standard Solana `blockSubscribe` direct RAW source contract owned by the continuous RAW transaction ingest Worker. diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs b/crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs index da896a7..e82c0cd 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs +++ b/crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs @@ -1,11 +1,14 @@ // file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs -// version: 16 +// version: 17 use sha2::Digest; // rust-rules: trait-import /// Maximum number of logical live sources accepted in one RAW transaction ingest runtime-resource aggregate. pub const MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES: usize = 32; +const RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.helius_transaction.filter.v1\0"; +const RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL: &str = "helius_ws_http"; +const RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.helius_transaction_http.source_key.v1\0"; const RAW_TRANSACTION_INGEST_LIVE_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.live_source.source_key.v1\0"; const RAW_TRANSACTION_INGEST_STANDARD_BLOCK_ACQUISITION_METHOD: &str = "block_subscribe"; const RAW_TRANSACTION_INGEST_STANDARD_BLOCK_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.standard_block.filter.v1\0"; @@ -26,6 +29,7 @@ enum RawTransactionIngestSourceFamily { } enum RawTransactionIngestLiveSource { + HeliusTransaction(crate::RawTransactionIngestHeliusTransactionSource), StandardBlock(crate::RawTransactionIngestStandardBlockSource), StandardLogs(crate::RawTransactionIngestStandardLogsSource), Yellowstone(crate::RawTransactionIngestYellowstoneSource), @@ -34,6 +38,7 @@ enum RawTransactionIngestLiveSource { impl RawTransactionIngestLiveSource { fn network(&self) -> &ksp_store_lib::RawNetworkId { return match self { + Self::HeliusTransaction(source) => &source.network, Self::StandardBlock(source) => &source.network, Self::StandardLogs(source) => &source.network, Self::Yellowstone(source) => &source.network, @@ -42,6 +47,7 @@ impl RawTransactionIngestLiveSource { fn source_key(&self) -> [u8; 32] { return match self { + Self::HeliusTransaction(source) => source.source_key, Self::StandardBlock(source) => source.source_key, Self::StandardLogs(source) => source.source_key, Self::Yellowstone(source) => source.source_key, @@ -56,6 +62,7 @@ impl RawTransactionIngestLiveSource { processing_frontier_sender: tokio::sync::watch::Sender, ) -> ksp_core_lib::Result<()> { return match self { + Self::HeliusTransaction(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await, Self::StandardBlock(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await, Self::StandardLogs(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await, Self::Yellowstone(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await, @@ -578,6 +585,276 @@ impl crate::RawTransactionIngestYellowstoneSource { } } +/// Validated Helius `transactionSubscribe` plus HTTP hydration source owned by the continuous RAW transaction ingest Worker. +/// +/// The caller provides one Transport-owned Helius LaserStream WebSocket endpoint, one Helius transaction filter, a Confirmed/Finalized commitment and one HTTP +/// hydration pool/role. The Worker always requests Full/Base64 notifications with `maxSupportedTransactionVersion = 1` but projects only the provider +/// signature/slot/index reference into the source-neutral hydration coordinator; the nested Helius payload is never copied into Worker state, while endpoint +/// URLs and credentials remain encapsulated by Transport-owned settings and are never exposed by Worker APIs or diagnostics. +pub struct RawTransactionIngestHeliusTransactionSource { + ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings, + filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter, + commitment: ksp_onchain_transport_lib::SolanaCommitment, + http_pool: ksp_onchain_transport_lib::HttpTransportPool, + hydration_role: ksp_onchain_transport_lib::HttpRoleName, + network: ksp_store_lib::RawNetworkId, + route: RawTransactionIngestSourceRoute, + filter_fingerprint: [u8; 32], + source_key: [u8; 32], +} + +impl crate::RawTransactionIngestHeliusTransactionSource { + /// Creates one validated Helius transaction source without opening WebSocket or HTTP network I/O. + pub fn new( + ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings, + filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter, + commitment: ksp_onchain_transport_lib::SolanaCommitment, + http_pool: ksp_onchain_transport_lib::HttpTransportPool, + hydration_role: ksp_onchain_transport_lib::HttpRoleName, + ) -> ksp_core_lib::Result { + let ws_settings = ksp_onchain_transport_lib::WsTransportSettings::new(std::vec![ws_endpoint.clone()]); + if ws_settings.validate().is_err() { + return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_endpoint_invalid")); + } + if ws_endpoint.protocol() != ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream { + return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_protocol_invalid")); + } + match commitment { + ksp_onchain_transport_lib::SolanaCommitment::Confirmed | ksp_onchain_transport_lib::SolanaCommitment::Finalized => {}, + ksp_onchain_transport_lib::SolanaCommitment::Processed => { + return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_commitment_invalid")); + }, + } + let request = helius_transaction_request(filter.clone(), commitment); + if request.validate().is_err() { + return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_request_invalid")); + } + let network = match ksp_store_lib::RawNetworkId::new(ws_endpoint.cluster().as_str()) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => { + return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_network_unrepresentable")); + }, + }; + let provider = match ksp_store_lib::RawProvenanceCode::new(ws_endpoint.provider().as_str()) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => { + return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_provider_unrepresentable")); + }, + }; + let endpoint_id = match ksp_store_lib::RawProvenanceCode::new(ws_endpoint.name()) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => { + return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_endpoint_unrepresentable")); + }, + }; + let route = RawTransactionIngestSourceRoute { endpoint_id, provider }; + let method = match ksp_onchain_transport_lib::find_http_rpc_method("getTransaction") { + std::option::Option::Some(value) => value, + std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_method_missing")), + }; + let compatible_http_routes = compatible_http_route_count(&http_pool, &hydration_role, method.request_kind(), network.as_str(), &route, "hx"); + let compatible_http_routes = match compatible_http_routes { + std::result::Result::Ok(value) => value, + std::result::Result::Err(error) => return std::result::Result::Err(error), + }; + if compatible_http_routes == 0 { + return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_role_unsupported")); + } + let filter_fingerprint = helius_transaction_filter_fingerprint(&filter); + let source_key = helius_transaction_live_source_key(&network, &route, commitment, &filter_fingerprint); + return std::result::Result::Ok(Self { + ws_endpoint, + filter, + commitment, + http_pool, + hydration_role, + network, + route, + filter_fingerprint, + source_key, + }); + } + + fn hydration_context(&self) -> RawTransactionIngestHydrationContext { + return RawTransactionIngestHydrationContext { + commitment: self.commitment, + http_pool: self.http_pool.clone(), + hydration_role: self.hydration_role.clone(), + network: self.network.clone(), + protocol: RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL, + route: self.route.clone(), + route_prefix: "hx", + source_key_domain: RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_SOURCE_KEY_DOMAIN, + }; + } + + /// Runs one productive Helius `transactionSubscribe` source until cooperative stop or one safe terminal source failure. + pub(crate) async fn run( + self, + settings: crate::RawTransactionIngestSettings, + mut stop_receiver: tokio::sync::watch::Receiver, + admission_sender: tokio::sync::mpsc::Sender, + processing_frontier_sender: tokio::sync::watch::Sender, + ) -> ksp_core_lib::Result<()> { + let connected = tokio::select! { + biased; + _ = stop_receiver.changed() => { + return std::result::Result::Ok(()); + } + result = ksp_onchain_transport_lib::HeliusLaserStreamWsSession::connect(self.ws_endpoint.clone()) => result, + }; + let session = match connected { + std::result::Result::Ok(value) => value, + std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())), + }; + let request = helius_transaction_request(self.filter.clone(), self.commitment); + let subscribed = tokio::select! { + biased; + _ = stop_receiver.changed() => { + let _closed = session.close().await; + return std::result::Result::Ok(()); + } + result = session.transaction_subscribe(&request) => result, + }; + let mut subscription = match subscribed { + std::result::Result::Ok(value) => value, + std::result::Result::Err(error) => { + let _closed = session.close().await; + return std::result::Result::Err(source_transport_error(error.code())); + }, + }; + let hydration = self.hydration_context(); + let mut coordinator = RawTransactionIngestHydrationCoordinator::new(&settings); + let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender); + processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Active); + let mut fault = std::option::Option::None; + loop { + if *stop_receiver.borrow() { + break; + } + if let std::result::Result::Err(error) = coordinator.start_hydrations(&hydration) { + fault = std::option::Option::Some(error); + break; + } + let can_receive = coordinator.can_receive(); + if !can_receive && coordinator.tasks.is_empty() { + fault = std::option::Option::Some(crate::runtime_error("source.hydration_stalled")); + break; + } + tokio::select! { + biased; + _ = stop_receiver.changed() => { + break; + } + joined = coordinator.tasks.join_next(), if !coordinator.tasks.is_empty() => { + let joined = match joined { + std::option::Option::Some(value) => value, + std::option::Option::None => { + fault = std::option::Option::Some(crate::runtime_error("source.hydration_join_missing")); + break; + }, + }; + let handled = coordinator + .handle_joined( + joined, + &hydration, + &settings, + &admission_sender, + &mut stop_receiver, + &mut processing_frontier, + ) + .await; + match handled { + std::result::Result::Ok(true) => {}, + std::result::Result::Ok(false) => break, + std::result::Result::Err(error) => { + fault = std::option::Option::Some(error); + break; + }, + } + } + notification = subscription.recv(), if can_receive => { + let notification = match notification { + std::option::Option::Some(std::result::Result::Ok(value)) => value, + std::option::Option::Some(std::result::Result::Err(error)) => { + fault = std::option::Option::Some(source_transport_error(error.code())); + break; + }, + std::option::Option::None => { + let error = match subscription.terminal_error_code() { + std::option::Option::Some(code) => source_transport_error(code), + std::option::Option::None => crate::runtime_error("source.helius_transaction_subscription_closed"), + }; + fault = std::option::Option::Some(error); + break; + }, + }; + let received_at = match current_raw_timestamp() { + std::result::Result::Ok(value) => value, + std::result::Result::Err(error) => { + fault = std::option::Option::Some(error); + break; + }, + }; + let full = match notification { + ksp_onchain_transport_lib::HeliusTransactionNotification::Full(value) => value, + _ => { + fault = std::option::Option::Some(crate::runtime_error("source.helius_transaction_notification_unqualified")); + break; + }, + }; + let signal = match project_helius_transaction_signal(&self, &full) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(error) => { + fault = std::option::Option::Some(error); + break; + }, + }; + if let std::result::Result::Err(error) = coordinator.queue_signal(&hydration, signal, received_at, &mut processing_frontier) { + fault = std::option::Option::Some(error); + break; + } + } + } + } + coordinator.abort_all(&mut processing_frontier).await; + processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closing); + let closed = session.close().await; + if let std::option::Option::Some(error) = fault { + processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed); + return std::result::Result::Err(error); + } + return match closed { + std::result::Result::Ok(()) => { + processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closed); + std::result::Result::Ok(()) + }, + std::result::Result::Err(error) => { + processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed); + std::result::Result::Err(source_transport_error(error.code())) + }, + }; + } +} + +impl std::fmt::Debug for crate::RawTransactionIngestHeliusTransactionSource { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let http_snapshot = self.http_pool.snapshot(); + return formatter + .debug_struct("RawTransactionIngestHeliusTransactionSource") + .field("ws_endpoint_name", &self.ws_endpoint.name()) + .field("ws_provider", &self.ws_endpoint.provider().as_str()) + .field("network", &self.network.as_str()) + .field("commitment", &self.commitment) + .field("filter", &self.filter) + .field("filter_fingerprint_bytes", &self.filter_fingerprint.len()) + .field("hydration_role", &self.hydration_role.as_str()) + .field("http_endpoint_count", &http_snapshot.endpoint_count()) + .field("source_key_bytes", &self.source_key.len()) + .finish(); + } +} + /// Validated standard Solana `blockSubscribe` direct RAW source owned by the continuous RAW transaction ingest Worker. /// /// The caller provides one Transport-owned standard WebSocket endpoint, one block filter and a Confirmed/Finalized commitment. The source requests Full/Base64 @@ -1055,8 +1332,8 @@ impl std::fmt::Debug for crate::RawTransactionIngestYellowstoneSource { /// Caller-composed runtime resources accepted by the continuous RAW transaction ingest Worker. /// -/// The aggregate owns a bounded collection of capability-specific live-source contracts. `pre.004` supports Yellowstone, standard Solana logs and standard -/// Solana block sources while deliberately keeping simultaneous multi-source supervision gated until the dedicated supervisor tranche. +/// The aggregate owns a bounded collection of capability-specific live-source contracts. `pre.005` supports Yellowstone, standard Solana logs/block and +/// Helius transaction sources while deliberately keeping simultaneous multi-source supervision gated until the dedicated supervisor tranche. pub struct RawTransactionIngestRuntimeResources { sources: std::vec::Vec, } @@ -1068,6 +1345,12 @@ impl crate::RawTransactionIngestRuntimeResources { return Self { sources: std::vec![RawTransactionIngestLiveSource::Yellowstone(yellowstone_source)] }; } + /// Creates one runtime-resource aggregate from one Helius `transactionSubscribe` plus HTTP hydration source contract. + #[must_use] + pub fn from_helius_transaction_source(source: crate::RawTransactionIngestHeliusTransactionSource) -> Self { + return Self { sources: std::vec![RawTransactionIngestLiveSource::HeliusTransaction(source)] }; + } + /// Creates one runtime-resource aggregate from one standard Solana `blockSubscribe` direct RAW source contract. #[must_use] pub fn from_standard_block_source(source: crate::RawTransactionIngestStandardBlockSource) -> Self { @@ -1107,6 +1390,27 @@ impl crate::RawTransactionIngestRuntimeResources { return std::result::Result::Ok(()); } + /// Adds one validated Helius transaction source while preserving the global 1..32 bound, one-network invariant and unique logical source identity. + pub fn try_push_helius_transaction_source(&mut self, source: crate::RawTransactionIngestHeliusTransactionSource) -> ksp_core_lib::Result<()> { + if self.sources.len() >= crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES { + return std::result::Result::Err(crate::runtime_error("runtime_resources.source_count_exceeded")); + } + let candidate = RawTransactionIngestLiveSource::HeliusTransaction(source); + let expected_network = match self.sources.first() { + std::option::Option::Some(source) => source.network(), + std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty")), + }; + if candidate.network() != expected_network { + return std::result::Result::Err(crate::runtime_error("runtime_resources.source_network_mismatch")); + } + let source_key = candidate.source_key(); + if self.sources.iter().any(|source| return source.source_key() == source_key) { + return std::result::Result::Err(crate::runtime_error("runtime_resources.duplicate_source_identity")); + } + self.sources.push(candidate); + return std::result::Result::Ok(()); + } + /// Adds one validated standard Solana block source while preserving the global 1..32 bound, one-network invariant and unique logical source identity. pub fn try_push_standard_block_source(&mut self, source: crate::RawTransactionIngestStandardBlockSource) -> ksp_core_lib::Result<()> { if self.sources.len() >= crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES { @@ -1218,6 +1522,93 @@ impl std::hash::Hasher for RawTransactionIngestSourceKeyHashWriter<'_> { } } +fn helius_transaction_filter_fingerprint(filter: &ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter) -> [u8; 32] { + let mut hasher = sha2::Sha256::new(); + hasher.update(RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_FILTER_FINGERPRINT_DOMAIN); + hash_helius_optional_bool(&mut hasher, b"failed", filter.failed()); + hash_helius_optional_bool(&mut hasher, b"vote", filter.vote()); + hash_helius_optional_pubkey_list(&mut hasher, b"account_exclude", filter.account_exclude()); + hash_helius_optional_pubkey_list(&mut hasher, b"account_include", filter.account_include()); + hash_helius_optional_pubkey_list(&mut hasher, b"account_required", filter.account_required()); + hash_live_source_key_component(&mut hasher, b"signature"); + match filter.signature() { + std::option::Option::Some(value) => { + hash_live_source_key_component(&mut hasher, b"value"); + hash_live_source_key_component(&mut hasher, value.as_bytes()); + }, + std::option::Option::None => hash_live_source_key_component(&mut hasher, b"omitted"), + } + hash_live_source_key_component(&mut hasher, b"token_accounts"); + match filter.token_accounts() { + std::option::Option::Some(value) => { + hash_live_source_key_component(&mut hasher, b"value"); + hash_live_source_key_component(&mut hasher, value.as_str().as_bytes()); + }, + std::option::Option::None => hash_live_source_key_component(&mut hasher, b"omitted"), + } + return hasher.finalize().into(); +} + +fn helius_transaction_live_source_key( + network: &ksp_store_lib::RawNetworkId, + route: &RawTransactionIngestSourceRoute, + commitment: ksp_onchain_transport_lib::SolanaCommitment, + filter_fingerprint: &[u8; 32], +) -> [u8; 32] { + let mut hasher = sha2::Sha256::new(); + hasher.update(RAW_TRANSACTION_INGEST_LIVE_SOURCE_KEY_DOMAIN); + hash_live_source_key_component(&mut hasher, b"helius_transaction"); + hash_live_source_key_component(&mut hasher, network.as_str().as_bytes()); + hash_live_source_key_component(&mut hasher, route.provider.as_str().as_bytes()); + hash_live_source_key_component(&mut hasher, route.endpoint_id.as_str().as_bytes()); + hash_live_source_key_component(&mut hasher, commitment.as_str().as_bytes()); + hash_live_source_key_component(&mut hasher, filter_fingerprint); + return hasher.finalize().into(); +} + +fn helius_transaction_request( + filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter, + commitment: ksp_onchain_transport_lib::SolanaCommitment, +) -> ksp_onchain_transport_lib::HeliusTransactionSubscribeRequest { + let options = ksp_onchain_transport_lib::HeliusTransactionSubscribeOptions::new( + std::option::Option::Some(commitment), + std::option::Option::Some(ksp_onchain_transport_lib::HeliusTransactionSubscribeEncoding::Base64), + std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full), + std::option::Option::Some(false), + std::option::Option::Some(1), + ); + return ksp_onchain_transport_lib::HeliusTransactionSubscribeRequest::new(filter, std::option::Option::Some(options)); +} + +fn hash_helius_optional_bool(hasher: &mut sha2::Sha256, label: &[u8], value: std::option::Option) { + hash_live_source_key_component(hasher, label); + match value { + std::option::Option::Some(true) => hash_live_source_key_component(hasher, b"true"), + std::option::Option::Some(false) => hash_live_source_key_component(hasher, b"false"), + std::option::Option::None => hash_live_source_key_component(hasher, b"omitted"), + } + return; +} + +fn hash_helius_optional_pubkey_list(hasher: &mut sha2::Sha256, label: &[u8], values: std::option::Option<&[ksp_core_lib::Pubkey]>) { + hash_live_source_key_component(hasher, label); + let values = match values { + std::option::Option::Some(values) => values, + std::option::Option::None => { + hash_live_source_key_component(hasher, b"omitted"); + return; + }, + }; + hash_live_source_key_component(hasher, b"value"); + let mut normalized = values.iter().map(|value| return value.to_bytes()).collect::>(); + normalized.sort_unstable(); + hash_live_source_key_component(hasher, &(normalized.len() as u64).to_be_bytes()); + for value in normalized { + hash_live_source_key_component(hasher, &value); + } + return; +} + fn standard_block_filter_fingerprint(filter: &ksp_onchain_transport_lib::SolanaBlockSubscribeFilter) -> [u8; 32] { let mut hasher = sha2::Sha256::new(); hasher.update(RAW_TRANSACTION_INGEST_STANDARD_BLOCK_FILTER_FINGERPRINT_DOMAIN); @@ -1666,6 +2057,51 @@ fn build_standard_block_provenance( ); } +trait RawTransactionIngestHeliusTransactionView { + fn signature(&self) -> &str; + + fn slot(&self) -> u64; + + fn transaction_index(&self) -> u64; +} + +impl RawTransactionIngestHeliusTransactionView for ksp_onchain_transport_lib::HeliusFullTransactionNotification { + fn signature(&self) -> &str { + return ksp_onchain_transport_lib::HeliusFullTransactionNotification::signature(self); + } + + fn slot(&self) -> u64 { + return ksp_onchain_transport_lib::HeliusFullTransactionNotification::slot(self); + } + + fn transaction_index(&self) -> u64 { + return ksp_onchain_transport_lib::HeliusFullTransactionNotification::transaction_index(self); + } +} + +fn project_helius_transaction_signal( + source: &crate::RawTransactionIngestHeliusTransactionSource, + response: &T, +) -> ksp_core_lib::Result { + let signature = ksp_raw_transaction_lib::parse_raw_transaction_signature(response.signature()); + let signature = match signature { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.helius_transaction_signature_invalid")), + }; + return std::result::Result::Ok(RawTransactionIngestSourceSignal { + created_at: std::option::Option::None, + family: RawTransactionIngestSourceFamily::Transaction, + matched_filter_count: 1, + matched_filter_fingerprint: source.filter_fingerprint, + matched_filter_id: std::option::Option::None, + network: source.network.clone(), + route: source.route.clone(), + signature, + slot: response.slot(), + transaction_index: std::option::Option::Some(response.transaction_index()), + }); +} + trait RawTransactionIngestStandardLogsView { fn signature(&self) -> &str; diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs b/crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs index 6e88d1b..140cdaa 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs +++ b/crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs @@ -1,5 +1,5 @@ // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs -// version: 20 +// version: 21 //! Dependency firewall canaries for the RAW transaction ingest Worker foundation. @@ -296,6 +296,44 @@ fn v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only return; } +#[test] +fn v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only() { + let root = include_str!("../src/lib.rs"); + let runtime = include_str!("../src/runtime.rs"); + let resources = include_str!("../src/runtime_resources.rs"); + for required in [ + "RawTransactionIngestHeliusTransactionSource", + "HeliusLaserStreamWsSession::connect", + ".transaction_subscribe(", + "HeliusTransactionSubscribeOptions::new", + "HeliusTransactionSubscribeEncoding::Base64", + "SolanaTransactionDetails::Full", + "std::option::Option::Some(1)", + "project_helius_transaction_signal", + "helius_ws_http", + "get_transaction_observed", + "max_pending_signals: settings.admission_queue_capacity()", + ] { + assert!(resources.contains(required), "required pre.005 Helius transaction contract missing: {required}"); + } + assert_eq!(resources.matches("get_transaction_observed(").count(), 1); + for forbidden in [ + "ksp_config_lib::", + "ksp_job_backfill_lib::", + "ksp_store_postgres_lib::", + "reqwest::", + "tokio_tungstenite::", + "helius_sdk", + "unbounded_channel", + ] { + assert!( + !runtime.contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden), + "pre.005 crossed a forbidden Helius transaction boundary: {forbidden}", + ); + } + return; +} + fn runtime_contains(value: &str) -> bool { return include_str!("../src/runtime.rs").contains(value); } diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs b/crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs index 7d1431c..5d82ead 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs +++ b/crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs @@ -1,5 +1,5 @@ // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs -// version: 13 +// version: 14 //! External public, security, redaction and release-boundary hardening canaries for `pre.010`. @@ -612,6 +612,46 @@ fn v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit return; } +#[test] +fn v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit() { + let resources = include_str!("../src/runtime_resources.rs"); + for required in [ + "RawTransactionIngestHeliusTransactionSource", + ".field(\"filter\"", + ".field(\"filter_fingerprint_bytes\"", + ".field(\"source_key_bytes\"", + "HeliusTransactionNotification::Full(value)", + "source.helius_transaction_notification_unqualified", + "project_helius_transaction_signal", + "transaction_index: std::option::Option::Some(response.transaction_index())", + "RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL", + ] { + assert!(resources.contains(required), "required pre.005 Helius hardening guard missing: {required}"); + } + let source_struct = match resources.split_once("pub struct RawTransactionIngestHeliusTransactionSource {") { + std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestHeliusTransactionSource") { + std::option::Option::Some((value, _)) => value, + std::option::Option::None => "", + }, + std::option::Option::None => "", + }; + for forbidden in ["url:", "api_key", "credential", "transaction:", "payload:", "tier:"] { + assert!(!source_struct.contains(forbidden), "sensitive/provider material stored in Helius transaction source: {forbidden}"); + } + let projection = match resources.split_once("fn project_helius_transaction_signal") { + std::option::Option::Some((_, tail)) => match tail.split_once("trait RawTransactionIngestStandardLogsView") { + std::option::Option::Some((value, _)) => value, + std::option::Option::None => "", + }, + std::option::Option::None => "", + }; + assert!(!projection.contains(".transaction()"), "Helius nested full payload must not enter Worker projection"); + for forbidden in ["Developer", "Business", "Professional", "paid_tier", "provider_tier"] { + assert!(!resources.contains(forbidden), "provider tier must not be coded in Worker: {forbidden}"); + } + return; +} + #[test] fn v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit() { let resources = include_str!("../src/runtime_resources.rs"); diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs b/crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs index 3c06a45..0313952 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs +++ b/crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs @@ -1,5 +1,5 @@ // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs -// version: 14 +// version: 15 //! External public-surface proofs for the RAW transaction ingest Worker foundation. @@ -224,6 +224,33 @@ fn v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport return; } +#[test] +fn v0_3_13_pre_005_helius_transaction_runtime_resource_surface_is_typed_and_transport_owned() { + let _source_new: fn( + ksp_onchain_transport_lib::WsEndpointSettings, + ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter, + ksp_onchain_transport_lib::SolanaCommitment, + ksp_onchain_transport_lib::HttpTransportPool, + ksp_onchain_transport_lib::HttpRoleName, + ) -> ksp_core_lib::Result = + ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new; + let _resources_from_helius: fn( + ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource, + ) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources = + ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source; + let _push_helius: fn( + &mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources, + ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource, + ) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_helius_transaction_source; + let root = include_str!("../src/lib.rs"); + let resources = include_str!("../src/runtime_resources.rs"); + assert!(root.contains("RawTransactionIngestHeliusTransactionSource")); + for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn http_pool(", "pub fn source_key("] { + assert!(!resources.contains(forbidden), "Helius transaction source implementation escape hatch present: {forbidden}"); + } + return; +} + #[test] fn v0_3_13_pre_004_standard_block_runtime_resource_surface_is_typed_and_transport_owned() { let _source_new: fn( diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs b/crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs index 83bd92a..447a512 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs +++ b/crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs @@ -1,5 +1,5 @@ // file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs -// version: 12 +// version: 13 //! Release-completeness canaries through the `pre.010` public/release/security hardening tranche. @@ -78,6 +78,7 @@ fn pre_010_public_root_export_inventory_is_exact() { "MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT", "RAW_TRANSACTION_INGEST_WORKER_KIND_CODE", "RawTransactionIngestHandle", + "RawTransactionIngestHeliusTransactionSource", "RawTransactionIngestRuntimeResources", "RawTransactionIngestSettings", "RawTransactionIngestSnapshot", @@ -117,6 +118,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() { "v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths", "v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit", "v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit", + "v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit", ] { assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}"); } @@ -132,6 +134,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() { assert!(dependency_boundary.contains("v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap")); assert!(dependency_boundary.contains("v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only")); assert!(dependency_boundary.contains("v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only")); + assert!(dependency_boundary.contains("v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only")); let public_api = include_str!("public_api.rs"); assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root")); assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle")); diff --git a/crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs b/crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs index 6e72d03..bb54aa5 100644 --- a/crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs +++ b/crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs @@ -1,5 +1,5 @@ // file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs -// version: 16 +// version: 17 fn grpc_endpoint(cluster: &str) -> std::option::Option { return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider"); @@ -126,6 +126,65 @@ fn standard_logs_source( }; } +fn helius_transaction_filter(failed: bool) -> ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter { + return ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new( + std::option::Option::Some(false), + std::option::Option::Some(failed), + std::option::Option::None, + std::option::Option::None, + std::option::Option::None, + std::option::Option::None, + std::option::Option::None, + ); +} + +fn helius_transaction_source( + cluster: &str, + endpoint_name: &str, + filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter, + commitment: ksp_onchain_transport_lib::SolanaCommitment, + hydration_role: &str, +) -> std::option::Option { + let endpoint = match ws_endpoint(cluster, endpoint_name, "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream) { + std::option::Option::Some(value) => value, + std::option::Option::None => return std::option::Option::None, + }; + let pool = match http_pool(cluster, hydration_role, "get_transaction") { + std::option::Option::Some(value) => value, + std::option::Option::None => return std::option::Option::None, + }; + return match crate::RawTransactionIngestHeliusTransactionSource::new( + endpoint, + filter, + commitment, + pool, + ksp_onchain_transport_lib::HttpRoleName::new(hydration_role), + ) { + std::result::Result::Ok(value) => std::option::Option::Some(value), + std::result::Result::Err(_) => std::option::Option::None, + }; +} + +struct HeliusTransactionViewFixture { + signature: std::string::String, + slot: u64, + transaction_index: u64, +} + +impl super::RawTransactionIngestHeliusTransactionView for HeliusTransactionViewFixture { + fn signature(&self) -> &str { + return self.signature.as_str(); + } + + fn slot(&self) -> u64 { + return self.slot; + } + + fn transaction_index(&self) -> u64 { + return self.transaction_index; + } +} + struct StandardLogsViewFixture { signature: std::string::String, slot: u64, @@ -434,6 +493,28 @@ fn standard_logs_source_for_http_url( }; } +fn helius_transaction_source_for_http_url(url: &str) -> std::option::Option { + let endpoint = + match ws_endpoint("devnet", "helius-transaction-fixture", "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream) { + std::option::Option::Some(value) => value, + std::option::Option::None => return std::option::Option::None, + }; + let pool = match http_pool_for_url(url, "devnet", "http-hydration-fixture", "fixture-http-provider") { + std::option::Option::Some(value) => value, + std::option::Option::None => return std::option::Option::None, + }; + return match crate::RawTransactionIngestHeliusTransactionSource::new( + endpoint, + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + pool, + ksp_onchain_transport_lib::HttpRoleName::new("hydration"), + ) { + std::result::Result::Ok(value) => std::option::Option::Some(value), + std::result::Result::Err(_) => std::option::Option::None, + }; +} + fn pre_004_settings() -> std::option::Option { let network = match ksp_store_lib::RawNetworkId::new("devnet") { std::result::Result::Ok(value) => value, @@ -2073,6 +2154,309 @@ fn v0_3_13_pre_004_runtime_resources_reject_duplicate_standard_block_identity() return; } +#[test] +fn v0_3_13_pre_005_helius_transaction_source_validates_protocol_commitment_request_and_hydration_route() { + let source = match helius_transaction_source( + "devnet", + "helius-transaction-a", + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + "hydration", + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let request = super::helius_transaction_request(source.filter.clone(), source.commitment); + let options = match request.options() { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + assert_eq!(options.commitment(), std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed)); + assert_eq!(options.encoding(), std::option::Option::Some(ksp_onchain_transport_lib::HeliusTransactionSubscribeEncoding::Base64)); + assert_eq!(options.transaction_details(), std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full)); + assert_eq!(options.show_rewards(), std::option::Option::Some(false)); + assert_eq!(options.max_supported_transaction_version(), std::option::Option::Some(1)); + assert!(request.validate().is_ok()); + let resources = crate::RawTransactionIngestRuntimeResources::from_helius_transaction_source(source); + assert_eq!(resources.source_count(), 1); + let standard_endpoint = + match ws_endpoint("devnet", "helius-transaction-standard", "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let pool = match http_pool("devnet", "hydration", "get_transaction") { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let protocol_error = crate::RawTransactionIngestHeliusTransactionSource::new( + standard_endpoint, + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + pool.clone(), + ksp_onchain_transport_lib::HttpRoleName::new("hydration"), + ); + let protocol_error = match protocol_error { + std::result::Result::Ok(_) => return, + std::result::Result::Err(value) => value, + }; + assert!(protocol_error.context().iter().any(|context| return context.value() == "runtime_resources.helius_transaction_protocol_invalid")); + let helius_endpoint = + match ws_endpoint("devnet", "helius-transaction-processed", "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let commitment_error = crate::RawTransactionIngestHeliusTransactionSource::new( + helius_endpoint, + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Processed, + pool, + ksp_onchain_transport_lib::HttpRoleName::new("hydration"), + ); + let commitment_error = match commitment_error { + std::result::Result::Ok(_) => return, + std::result::Result::Err(value) => value, + }; + assert!(commitment_error.context().iter().any(|context| return context.value() == "runtime_resources.hydration_commitment_invalid")); + let cross_network_endpoint = match ws_endpoint( + "devnet", + "helius-transaction-cross-network", + "fixture-helius-provider", + ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream, + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let cross_network_pool = match http_pool("testnet", "hydration", "get_transaction") { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let cross_network = crate::RawTransactionIngestHeliusTransactionSource::new( + cross_network_endpoint, + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + cross_network_pool, + ksp_onchain_transport_lib::HttpRoleName::new("hydration"), + ); + let cross_network = match cross_network { + std::result::Result::Ok(_) => return, + std::result::Result::Err(value) => value, + }; + assert!(cross_network.context().iter().any(|context| return context.value() == "runtime_resources.transport_network_mismatch")); + return; +} + +#[test] +fn v0_3_13_pre_005_helius_transaction_source_key_is_filter_sensitive_order_normalized_hydration_role_neutral_and_redacted() { + let first_pubkey = ksp_core_lib::Pubkey::new_from_array([0x31_u8; 32]); + let second_pubkey = ksp_core_lib::Pubkey::new_from_array([0x32_u8; 32]); + let signature_canary = "5HueCGU8rMjxEXxiPuD5BDuRaH7RkZBzQ2hL5M5WnQp9"; + let first_filter = ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new( + std::option::Option::Some(false), + std::option::Option::Some(false), + std::option::Option::Some(signature_canary.to_owned()), + std::option::Option::Some(std::vec![first_pubkey, second_pubkey]), + std::option::Option::None, + std::option::Option::None, + std::option::Option::Some(ksp_onchain_transport_lib::HeliusTokenAccountsFilter::BalanceChanged), + ); + let reordered_filter = ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new( + std::option::Option::Some(false), + std::option::Option::Some(false), + std::option::Option::Some(signature_canary.to_owned()), + std::option::Option::Some(std::vec![second_pubkey, first_pubkey]), + std::option::Option::None, + std::option::Option::None, + std::option::Option::Some(ksp_onchain_transport_lib::HeliusTokenAccountsFilter::BalanceChanged), + ); + let changed_filter = ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new( + std::option::Option::Some(false), + std::option::Option::Some(true), + std::option::Option::Some(signature_canary.to_owned()), + std::option::Option::Some(std::vec![first_pubkey, second_pubkey]), + std::option::Option::None, + std::option::Option::None, + std::option::Option::Some(ksp_onchain_transport_lib::HeliusTokenAccountsFilter::BalanceChanged), + ); + let first = match helius_transaction_source( + "devnet", + "helius-transaction-key", + first_filter, + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + "hydration", + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let reordered = match helius_transaction_source( + "devnet", + "helius-transaction-key", + reordered_filter, + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + "secondary_hydration", + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let changed = match helius_transaction_source( + "devnet", + "helius-transaction-key", + changed_filter, + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + "hydration", + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + assert_eq!(first.source_key, reordered.source_key); + assert_ne!(first.source_key, changed.source_key); + let first_pubkey_text = first_pubkey.to_string(); + let debug = std::format!("{first:?}"); + assert!(debug.contains("RawTransactionIngestHeliusTransactionSource")); + assert!(debug.contains("signature_present")); + assert!(!debug.contains(signature_canary)); + assert!(!debug.contains(first_pubkey_text.as_str())); + assert!(!debug.contains("WS-SECRET-CANARY")); + assert!(!debug.contains("api-key")); + assert!(!debug.contains("source_key: [")); + return; +} + +#[test] +fn v0_3_13_pre_005_helius_full_projection_keeps_only_reference_material() { + let source = match helius_transaction_source( + "devnet", + "helius-transaction-projection", + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Finalized, + "hydration", + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let fixture = HeliusTransactionViewFixture { signature: PRE_004_ZERO_SIGNATURE_TEXT.to_owned(), slot: 81, transaction_index: 9 }; + let signal = match super::project_helius_transaction_signal(&source, &fixture) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return, + }; + assert_eq!(signal.family, super::RawTransactionIngestSourceFamily::Transaction); + assert_eq!(signal.slot, 81); + assert_eq!(signal.transaction_index, std::option::Option::Some(9)); + assert_eq!(signal.created_at, std::option::Option::None); + assert_eq!(signal.matched_filter_count, 1); + assert_eq!(signal.matched_filter_id, std::option::Option::None); + assert_eq!(signal.signature, ksp_store_lib::RawTransactionSignature::new([0_u8; 64])); + let debug = std::format!("{signal:?}"); + assert!(!debug.contains(PRE_004_ZERO_SIGNATURE_TEXT)); + return; +} + +#[tokio::test(flavor = "current_thread")] +async fn v0_3_13_pre_005_helius_full_reference_hydrates_once_to_exact_common_raw() { + let body = std::format!( + "{{\"jsonrpc\":\"2.0\",\"result\":{{\"slot\":42,\"blockTime\":1760000120,\"transaction\":[\"{}\",\"base64\"],\"meta\":null,\"version\":1,\"transactionIndex\":7}},\"id\":1}}", + PRE_004_ZERO_TRANSACTION_BASE64, + ); + let (url, server) = match serve_http_once(body) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return, + }; + let source = match helius_transaction_source_for_http_url(url.as_str()) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let settings = match pre_004_settings() { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let fixture = HeliusTransactionViewFixture { signature: PRE_004_ZERO_SIGNATURE_TEXT.to_owned(), slot: 42, transaction_index: 7 }; + let signal = match super::project_helius_transaction_signal(&source, &fixture) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return, + }; + let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_760_000_200_000) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return, + }; + let hydration = source.hydration_context(); + let key = match super::hydration_key(&hydration, &signal) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return, + }; + let fetched = + super::fetch_hydration(hydration.http_pool.clone(), hydration.hydration_role.clone(), hydration.network.clone(), key, hydration.commitment).await; + let fetched = match fetched { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return, + }; + let ingress = super::finalize_hydration(&hydration, &settings, signal, received_at, &fetched.observed); + let ingress = match ingress { + std::result::Result::Ok(std::option::Option::Some(value)) => value, + std::result::Result::Ok(std::option::Option::None) | std::result::Result::Err(_) => return, + }; + assert_eq!(ingress.provenance.protocol().as_str(), "helius_ws_http"); + assert_eq!(ingress.provenance.acquisition_method().as_str(), "transaction_get_transaction"); + let (mut admission, sender) = crate::RawTransactionAdmission::new(1); + if sender.send(ingress).await.is_err() { + return; + } + std::mem::drop(sender); + let acquisition = match admission.receive(settings.network()).await { + std::result::Result::Ok(std::option::Option::Some(value)) => value, + _ => return, + }; + assert_eq!(acquisition.transaction().reference().network().as_str(), "devnet"); + assert_eq!(acquisition.transaction().reference().signature().as_bytes(), &[0_u8; 64]); + assert_eq!(acquisition.transaction().slot(), 42); + assert_eq!(acquisition.transaction().block_time().map(|value| return value.unix_millis()), std::option::Option::Some(1_760_000_120_000)); + let payload = match std::str::from_utf8(acquisition.transaction().payload().bytes()) { + std::result::Result::Ok(value) => value, + std::result::Result::Err(_) => return, + }; + assert!(payload.contains("\"version\":1")); + assert!(payload.contains("\"transactionIndex\":7")); + let request_body = match server.join() { + std::result::Result::Ok(std::result::Result::Ok(value)) => value, + std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => return, + }; + assert!(request_body.contains("\"method\":\"getTransaction\"")); + assert!(request_body.contains("\"encoding\":\"base64\"")); + assert!(request_body.contains("\"maxSupportedTransactionVersion\":1")); + return; +} + +#[test] +fn v0_3_13_pre_005_runtime_resources_reject_duplicate_helius_transaction_identity() { + let first = match helius_transaction_source( + "devnet", + "helius-transaction-duplicate", + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + "hydration", + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let duplicate = match helius_transaction_source( + "devnet", + "helius-transaction-duplicate", + helius_transaction_filter(false), + ksp_onchain_transport_lib::SolanaCommitment::Confirmed, + "secondary_hydration", + ) { + std::option::Option::Some(value) => value, + std::option::Option::None => return, + }; + let mut resources = crate::RawTransactionIngestRuntimeResources::from_helius_transaction_source(first); + let result = resources.try_push_helius_transaction_source(duplicate); + let error = match result { + std::result::Result::Ok(()) => return, + std::result::Result::Err(value) => value, + }; + assert!(error.context().iter().any(|context| return context.value() == "runtime_resources.duplicate_source_identity")); + assert_eq!(resources.source_count(), 1); + return; +} + #[test] fn pre_007_processing_frontier_never_advances_through_pending_work() { let mut frontier = super::RawTransactionIngestProcessingFrontier::new(); diff --git a/deltas/0.3.13/pre.005.md b/deltas/0.3.13/pre.005.md new file mode 100644 index 0000000..9977af7 --- /dev/null +++ b/deltas/0.3.13/pre.005.md @@ -0,0 +1,268 @@ + + + +# Delta 0.3.13-pre.005 — Helius transactionSubscribe + hydration Common RAW + +## Base requise + +```text +livraison précédente : 0.3.13-pre.004-fix.001 +Cargo base : 0.3.13-pre.4.fix.1 +delta base : deltas/0.3.13/pre.004-fix.001.md +archive delta base : ksp-general-0.3.13-pre.004-fix.001.zip +``` + +Le gate opérateur reçu le 10 septembre 2026 sur cette base est vert pour toutes les commandes exécutées : `cargo fmt`, `cargo fmt --check`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et toutes les suites de `ksp-worker-raw-transaction-ingest-lib` (`79` unit, `4` cross-layer, `11` dependency-boundary, `18` hardening, `12` public-api, `4` release-completeness, `0` doc-test). Cette exécution ne contenait pas le test Transport ni les commandes `cargo tree`; ils ne sont donc pas déclarés PASS pour ce gate précis. + +## Objectif + +Ajouter la source Helius WebSocket prévue par la tranche sans dupliquer le chemin Common RAW : + +```text +Helius LaserStream transactionSubscribe + -> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false + -> projection privée signature + slot + transactionIndex + -> coordinateur d'hydration source-neutral existant + -> HTTP getTransaction observed + -> Common RAW canonique + -> admission centrale Worker + -> Store +``` + +Le nested payload transaction Helius n'est pas utilisé comme RAW-direct. Il reste possédé par Transport et n'est jamais copié dans le signal Worker. + +## Version + +```text +livraison : 0.3.13-pre.005 +workspace.package.version : 0.3.13-pre.5 +archive : ksp-general-0.3.13-pre.005.zip +``` + +`Cargo.toml` passe de la version de fichier `547` à `548`. + +## Source Helius Transaction + +Nouvelle surface publique Worker : + +```text +RawTransactionIngestHeliusTransactionSource +RawTransactionIngestRuntimeResources::from_helius_transaction_source +RawTransactionIngestRuntimeResources::try_push_helius_transaction_source +``` + +Construction caller-composed : + +```text +WsEndpointSettings kind helius_laserstream +HeliusTransactionSubscribeFilter +SolanaCommitment Confirmed ou Finalized +HttpTransportPool +HttpRoleName d'hydration +``` + +La construction est sans I/O. Elle refuse un endpoint invalide ou non Helius LaserStream, `Processed`, un réseau/provenance non représentable et l'absence d'une route HTTP `getTransaction` compatible sur le même réseau. + +Au runtime, la source réutilise exclusivement les façades Transport existantes : + +```text +HeliusLaserStreamWsSession::connect +HeliusLaserStreamWsSession::transaction_subscribe +HttpTransportPool::get_transaction_observed via le coordinateur commun +``` + +Aucun client WebSocket direct, SDK Helius, `reqwest`, `tokio-tungstenite`, backend Store ou accès Config n'est ajouté au Worker. + +## Requête Helius qualifiée + +La requête Worker est fixée à : + +```text +commitment = Confirmed | Finalized +encoding = Base64 +transactionDetails = Full +showRewards = false +maxSupportedTransactionVersion = 1 +``` + +Une notification productive doit être `HeliusTransactionNotification::Full`. Une forme `Signature`, `Unknown` ou future malgré la requête `Full` devient `source.helius_transaction_notification_unqualified`; elle n'est jamais convertie en succès silencieux. + +## Projection et hydration + +Le signal privé issu d'une notification `Full` conserve seulement : + +```text +network +signature +slot +transactionIndex +safe route identity +private filter fingerprint +commitment via hydration context +``` + +Le nested champ `transaction` de la notification Helius n'est pas lu par `project_helius_transaction_signal` et n'est pas copié dans Worker. La construction Common RAW reste donc unique : `getTransaction observed` en Base64 avec `maxSupportedTransactionVersion = 1`, puis canonicalisation/admission existantes. + +Le coordinateur d'hydration reste borné et source-neutral : + +```text +in-flight hydration <= persistence_concurrency +pending source signals <= admission_queue_capacity +coalescence key = network + signature + commitment +``` + +Cette voie converge avec Yellowstone et Standard Logs au lieu d'introduire un second moteur HTTP. + +## Identité logique, redaction et secrets + +Le `source_key` Helius est dérivé de : + +```text +domain KSP live-source v1 +family = helius_transaction +network +safe provider identity +safe endpoint identity +commitment +private normalized Helius filter fingerprint +SHA-256 -> [u8; 32] +``` + +L'empreinte privée couvre `vote`, `failed`, signature exacte, listes `accountInclude`/`accountExclude`/`accountRequired` et `tokenAccounts`. Les listes de pubkeys sont triées avant hash afin que leur ordre ne crée pas artificiellement deux identités logiques. Le rôle/pool HTTP d'hydration reste exclu du `source_key`. + +Les URLs et credentials restent encapsulés par les settings Transport privés utilisés pour ouvrir les connexions ; ils ne sont exposés ni par l'API publique ni par `Debug`. Les valeurs de signature/pubkeys du filtre et les octets de `source_key` ne sont pas rendus dans les diagnostics Worker. + +Les profils Config Helius et `KSP_SECRET_HELIUS_API_KEY` existaient déjà avant cette tranche. Aucun changement Config, `.env`, entitlement, quota, prix ou tier provider n'est codé dans Worker. + +## Activation runtime + +Le contrat reste volontairement mono-source avant `pre.007` : + +```text +1 source Yellowstone : productif +1 source Standard Logs : productif +1 source Standard Block : productif +1 source Helius Transaction : productif +2..32 sources : composition validable +start 2..32 : runtime_resources.multi_source_activation_pending avant spawn +``` + +Aucune source configurée n'est silencieusement ignorée. + +## Preuves ajoutées + +Unit tests Worker : + +```text +validation protocole/commitment/route HTTP Helius +requête fixe Full/Base64/maxV1/showRewards=false +source_key filter-sensitive, order-normalized et hydration-role-neutral +Debug redacted : signature/pubkeys/URL/API key/source-key absents +projection Full -> signature/slot/index uniquement +Helius reference -> un getTransaction observed -> Common RAW exact +provenance helius_ws_http + transaction_get_transaction +rejet transactionnel d'une identité Helius dupliquée +``` + +Canaris externes : + +```text +public_api : construction/from/push depuis crate root sans escape hatch des ressources internes +dependency_boundary : façades Transport Helius + un seul getTransaction commun +hardening : nested payload non copié, secrets/redaction et absence de tier provider +release_completeness : export et canaris pre.005 obligatoires +``` + +## Documentation corrigée + +La documentation Worker est mise à jour pour inclure la source Helius et la convergence d'hydration. Une mention héritée de `README.md` qui classait encore `blockSubscribe` et Helius parmi les sources hors périmètre est supprimée ; `USAGE.md` reste volontairement version-neutral. + +## Fichiers modifiés + +```text +Cargo.toml +crates/ksp-worker-raw-transaction-ingest-lib/README.md +crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md +crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs +crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs +crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs +crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs +crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs +crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs +crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs +docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md +docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md +``` + +## Fichiers ajoutés + +```text +deltas/0.3.13/pre.005.md +``` + +## Fichiers supprimés + +```text +aucun +``` + +## Non-claims + +`pre.005` ne prétend pas : + +```text +supervision simultanée de plusieurs sources +HTTP live block polling +RAW-direct depuis le nested payload Helius +SDK Helius dans Worker +secret/API key lu directement par Worker +nouveau profil Config Helius +entitlement/tier/quota/prix provider codé +modification de ksp-onchain-transport-lib +nouvelle dépendance externe +``` + +## Validation dans l'environnement d'assemblage + +Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-`pre.005` restent donc `NON EXÉCUTÉ LOCAL`. Les audits statiques KSP, le scan des identifiants normatifs et le contrôle exhaustif du diff sont exécutés avant packaging. + +## Gate opérateur requis avant pre.006 + +```bash +cargo fmt --all +cargo fmt --all -- --check +python3 scripts/audit_rust_workspace_rules.py +python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas +cargo check --workspace +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test -p ksp-onchain-transport-lib +cargo test -p ksp-worker-raw-transaction-ingest-lib +cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal +cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features +cargo tree --duplicates +``` + +`pre.006` ne doit pas commencer si ce gate révèle une anomalie. + +## Résultats statiques finaux de l'assemblage + +```text +General Rust rule audit: clean +Rust export completeness audit: 0 candidate(s) +KSP workspace Rust rule audit: clean +Markdown table audit: clean (340 tables, 837 files) +Normative rule definitions: 489 +Unique normative IDs: 489 +Duplicates: 0 +``` + +## Contrôle exhaustif avant packaging + +```text +12 fichiers existants modifiés +1 fichier ajouté +0 fichier supprimé +12/12 headers de fichiers existants incrémentés exactement de +1 +Cargo.toml : version workspace uniquement hors header +aucune dépendance ajoutée/modifiée/supprimée +``` diff --git a/docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md b/docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md index 113c057..de2335e 100644 --- a/docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md +++ b/docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md @@ -1,5 +1,5 @@ - + # Plan v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source RawTransaction @@ -827,7 +827,7 @@ Source block, projection full/base64, qualification Legacy/V0/V1, `block:null` e ### pre.005 — Helius transactionSubscribe -Source Helius, max version 1, hydration, Config uniquement si réellement requis, tests redaction/tier non codé. +Source Helius `transactionSubscribe` Full/Base64, max version 1, projection référence signature/slot/index, hydration commune, Config uniquement si réellement requis, tests redaction/tier non codé. ### pre.006 — HTTP live block polling @@ -903,3 +903,143 @@ historical replay/backfill EARLY feeds consumer Desk 0.3.15 ``` + +## 43. Gate opérateur pre.004-fix.001 reçu avant pre.005 + +Le gate opérateur communiqué le 10 septembre 2026 est vert après `pre.004-fix.001` : + +```text +cargo fmt --all + --check : PASS +audit Rust général : clean +export completeness : 0 candidate +workspace Rust rules : clean +Markdown : clean, 340 tables / 836 files +cargo check --workspace : PASS +cargo clippy --workspace --all-targets --all-features -- -D warnings : PASS +worker unit : 79/79 PASS +cross_layer_completeness : 4/4 PASS +dependency_boundary : 11/11 PASS +hardening : 18/18 PASS +public_api : 12/12 PASS +release_completeness : 4/4 PASS +doc-tests Worker : 0/0 PASS +``` + +Cette preuve ferme `pre.004-fix.001` et autorise la tranche Helius. + +## 44. Implémentation pre.005 — Helius transactionSubscribe + hydration + +`pre.005` ajoute `RawTransactionIngestHeliusTransactionSource` à l'agrégat borné existant. La construction sans I/O reçoit : + +```text +WsEndpointSettings kind helius_laserstream +HeliusTransactionSubscribeFilter +SolanaCommitment Confirmed ou Finalized +HttpTransportPool +HttpRoleName hydration +``` + +Le Worker construit la requête provider à partir des types Transport existants avec : + +```text +encoding = Base64 +transactionDetails = Full +showRewards = false +maxSupportedTransactionVersion = 1 +``` + +`HeliusLaserStreamWsSession::connect` et `transaction_subscribe` restent les seules primitives WebSocket utilisées ; aucun SDK provider, socket parallèle ou accès Config n'est ajouté. + +## 45. Projection Helius et convergence Common RAW + +Bien que la notification `Full` contienne un nested payload transaction provider, `pre.005` ne le traite pas comme Common RAW directement. Le signal privé Worker conserve seulement : + +```text +network +signature +slot +transaction_index +commitment via hydration context +safe route identity +private filter fingerprint +``` + +Puis le coordinateur source-neutral existant exécute `getTransaction observed` en `Base64` et `maxSupportedTransactionVersion = 1`. Le même pipeline de canonicalisation/admission est donc utilisé par Yellowstone, Standard Logs et Helius. + +Une notification Helius qui n'est pas `Full` malgré la requête `Full` devient `source.helius_transaction_notification_unqualified`. Aucun `Signature`/`Unknown` inattendu n'est transformé en succès silencieux. + +## 46. Identité, secrets, Config et tier Helius + +La clé live logique Helius est dérivée de : + +```text +domain KSP live source key +family = helius_transaction +network +safe provider identity +safe endpoint identity +commitment +private normalized filter fingerprint +SHA-256 -> [u8; 32] +``` + +Les listes de pubkeys Helius sont normalisées avant hash afin qu'une permutation de listes sémantiquement ensemblistes ne contourne pas la détection de doublon. Signature exacte, pubkeys et mode tokenAccounts participent à l'empreinte privée mais ne sont pas rendus par `Debug`. + +Le rôle/pool HTTP reste exclu de l'identité live comme pour Standard Logs et Yellowstone. Les profils Config Helius Mainnet/Devnet et `KSP_SECRET_HELIUS_API_KEY` existent déjà dans la base ; aucun changement Config ni `.env` n'est requis pour cette tranche. Aucun tier `Developer`, `Business`, quota ou politique payante n'est codé dans le Worker. + +## 47. Preuves déterministes pre.005 + +Les preuves ajoutées couvrent : + +```text +validation endpoint Helius / commitment / route HTTP +requête Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false +source_key filter-sensitive, order-normalized et hydration-role-neutral +Debug redacted : signature/pubkeys/URL/API key/source key absents +projection Full -> signature/slot/index uniquement +Helius reference -> one getTransaction observed -> Common RAW exact +provenance protocol helius_ws_http + transaction_get_transaction +rejet duplicate Helius source identity +public API root + composition aggregate +firewall Transport-only + un seul get_transaction_observed +hardening payload nested non copié + tier provider non codé +``` + +La supervision simultanée reste volontairement différée à `pre.007` : une collection de plusieurs sources est composable/validable mais échoue fermée avant spawn. + +## 48. Non-claims pre.005 + +`pre.005` ne prétend pas : + +```text +avoir activé plusieurs sources simultanément +avoir ajouté HTTP live polling +avoir utilisé le nested payload Helius comme Common RAW direct +avoir ajouté un SDK Helius +avoir ajouté ou lu un secret dans le Worker +avoir codé un tier/quota/prix provider +avoir ajouté un nouveau profil Config +avoir modifié Transport +avoir exécuté Cargo localement après modification dans l'environnement d'assemblage +``` + +## 49. Gate opérateur requis avant pre.006 + +Après application du delta : + +```bash +cargo fmt --all +cargo fmt --all -- --check +python3 scripts/audit_rust_workspace_rules.py +python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas +cargo check --workspace +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test -p ksp-onchain-transport-lib +cargo test -p ksp-worker-raw-transaction-ingest-lib +cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal +cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features +cargo tree --duplicates +``` + +`pre.006` ne commence pas tant qu'une anomalie de format, règle, compilation, test ou frontière de dépendances subsiste. + diff --git a/docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md b/docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md index bdc5186..31be913 100644 --- a/docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md +++ b/docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md @@ -1,11 +1,11 @@ - + # Validation v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source ## 1. Rôle -Ce document enregistre les preuves réellement exécutées et les non-claims de `0.3.13`. `pre.001` ferme l'audit/planification, `pre.002` matérialise la composition runtime multi-source, `pre.003` ajoute Standard Solana `logsSubscribe` + hydration HTTP observée et `pre.004` ajoute Standard Solana `blockSubscribe` RAW-direct qualifié Legacy/V0/V1, sans activer encore la supervision simultanée. +Ce document enregistre les preuves réellement exécutées et les non-claims de `0.3.13`. `pre.001` ferme l'audit/planification, `pre.002` matérialise la composition runtime multi-source, `pre.003` ajoute Standard Solana `logsSubscribe` + hydration HTTP observée, `pre.004` ajoute Standard Solana `blockSubscribe` RAW-direct qualifié Legacy/V0/V1 et `pre.005` ajoute Helius `transactionSubscribe` + hydration commune, sans activer encore la supervision simultanée. ## 2. Archive stable contrôlée @@ -965,3 +965,106 @@ cargo tree --duplicates ``` `pre.005` ne doit pas commencer si une anomalie de format, règle, compilation, API, dépendance ou test apparaît. + +## 43. Gate opérateur pre.004-fix.001 + +Preuve reçue le 10 septembre 2026 : + +```text +fmt : PASS +audits Rust : clean / export completeness 0 +Markdown : clean, 340 tables / 836 files +cargo check --workspace : PASS +clippy workspace all-targets all-features -D warnings : PASS +worker unit : 79/79 PASS +cross-layer : 4/4 PASS +dependency boundary : 11/11 PASS +hardening : 18/18 PASS +public API : 12/12 PASS +release completeness : 4/4 PASS +``` + +## 44. Surface pre.005 matérialisée + +```text +RawTransactionIngestHeliusTransactionSource +RawTransactionIngestRuntimeResources::from_helius_transaction_source +RawTransactionIngestRuntimeResources::try_push_helius_transaction_source +HeliusLaserStreamWsSession::connect +HeliusLaserStreamWsSession::transaction_subscribe +HeliusTransactionSubscribeOptions Full/Base64/maxV1/showRewards=false +project_helius_transaction_signal +hydration source-neutral get_transaction_observed +``` + +Aucun changement Config/Transport/manifeste de crate n'est requis. + +## 45. Contrats Helius vérifiés statiquement + +La tranche exige : + +```text +protocol = helius_laserstream +commitment = confirmed | finalized +HTTP hydration route = get_transaction sur même réseau +notification productive = Full uniquement +signal Worker = signature + slot + transactionIndex + route/filter safe identity +nested transaction payload = non copié +protocol de provenance convergée = helius_ws_http +acquisition method = transaction_get_transaction +``` + +La source key inclut le filtre complet sous empreinte SHA-256 privée, avec normalisation de l'ordre des listes de pubkeys. Le rôle d'hydration n'entre pas dans l'identité live. + +## 46. Config, secret et tier + +Les profils Helius LaserStream Mainnet/Devnet et `KSP_SECRET_HELIUS_API_KEY` existaient avant `pre.005`. Le Worker reçoit seulement `WsEndpointSettings` déjà résolu et ne dépend jamais de Config. Aucun tier provider, prix, quota ou entitlement n'est codé ; l'éligibilité réelle d'un compte Helius reste une propriété externe/live. + +## 47. Preuves unitaires et externes attendues + +```text +unit : request fixed Full/Base64/maxV1/showRewards=false +unit : protocol/commitment/cross-network hydration rejection +unit : source key stable/filter-sensitive/order-normalized/redacted +unit : Full reference projection signature/slot/index only +unit : reference -> HTTP getTransaction -> Common RAW exact +unit : duplicate Helius source rejected transactionally +public_api : root constructor/from/push typed +boundary : facade Helius + one common getTransaction only +hardening : nested payload/secret/tier absent +release : export inventory + canaries pre.005 +``` + +## 48. Validation locale d'assemblage pre.005 + +Le toolchain Cargo/Rustfmt n'est pas disponible dans l'environnement d'assemblage. Les commandes Cargo post-modification restent `NON EXÉCUTÉ LOCAL`; seuls les audits statiques effectivement exécutés peuvent être déclarés PASS dans le delta. + +## 49. Non-claims pre.005 + +```text +pas de multi-source simultané avant pre.007 +pas de HTTP live polling avant pre.006 +pas de RAW-direct depuis le nested payload Helius +pas de SDK Helius +pas de secret/Config dans le Worker +pas de tier/quota provider codé +pas de nouveau profil Config +pas de changement Transport +``` + +## 50. Gate opérateur requis avant pre.006 + +```bash +cargo fmt --all +cargo fmt --all -- --check +python3 scripts/audit_rust_workspace_rules.py +python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas +cargo check --workspace +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test -p ksp-onchain-transport-lib +cargo test -p ksp-worker-raw-transaction-ingest-lib +cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal +cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features +cargo tree --duplicates +``` +