v0.3.13-pre.005

This commit is contained in:
2026-09-10 15:01:00 +02:00
parent 27ab84d198
commit 11ecdccd65
13 changed files with 1524 additions and 27 deletions

View File

@@ -1,5 +1,5 @@
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
<!-- version: 4 -->
<!-- version: 5 -->
# ksp-worker-raw-transaction-ingest-lib
@@ -13,7 +13,7 @@ RawTransactionIngestWorker::start
RawTransactionIngestWorker::start_with_runtime_resources
-> même runtime + une source productive supervisée
Yellowstone, WS standard logsSubscribe ou WS standard blockSubscribe
Yellowstone, WS standard logsSubscribe, WS standard blockSubscribe ou Helius transactionSubscribe
+ hydration HTTP getTransaction lorsque la source produit une référence
```
@@ -39,6 +39,12 @@ Solana standard WS blockSubscribe
-> qualification explicite Legacy / V0 / V1
-> matériau Common RAW direct par transaction du bloc
Helius transactionSubscribe
-> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false
-> signature + slot + transactionIndex uniquement dans le signal Worker
-> coalescence bornée par (network, signature, commitment)
-> HTTP getTransaction observed avant Common RAW
les chemins productifs
-> ksp-raw-transaction-lib
-> admission centrale bornée
@@ -70,7 +76,7 @@ La construction est sans I/O et refuse notamment :
- une provenance provider/endpoint non représentable ;
- l'absence d'une route HTTP compatible pour `getTransaction` sur le même réseau.
Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques. `pre.004` sait exécuter une source unique Yellowstone, Standard Logs ou Standard Block ; une activation simultanée de plusieurs sources reste rejetée avant spawn jusqu'au supervisor dédié. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés.
Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques. `pre.005` sait exécuter une source unique Yellowstone, Standard Logs, Standard Block ou Helius Transaction ; une activation simultanée de plusieurs sources reste rejetée avant spawn jusqu'au supervisor dédié. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés.
## Contrat de source Standard Logs + HTTP
@@ -104,6 +110,24 @@ La qualification est fermée : une version omise/nulle ou supérieure à `1`, un
Pour un bloc multi-transaction, le slot n'est projeté settled qu'après l'admission réussie de toutes ses transactions. Cette voie RAW-direct n'incrémente pas `hydration_pending`; un stop ou une erreur au milieu du bloc ne produit aucune progression artificielle. Reconnect, resubscribe et backpressure WebSocket restent possédés par Transport.
## Contrat de source Helius Transaction + HTTP
`RawTransactionIngestHeliusTransactionSource::new` reçoit :
```text
WsEndpointSettings kind helius_laserstream
HeliusTransactionSubscribeFilter
SolanaCommitment Confirmed ou Finalized
HttpTransportPool
HttpRoleName d'hydration
```
La construction est sans I/O. Elle réutilise le contrat Transport existant et impose `Full`, `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`. Le Worker ne lit pas Config, ne lit pas `KSP_SECRET_HELIUS_API_KEY` et ne code aucun tier provider ; l'URL résolue et le credential restent dans l'endpoint Transport fourni par le caller.
Au runtime, `HeliusLaserStreamWsSession::connect` puis `transaction_subscribe` sont utilisés. Seule une notification `Full` conforme au mode demandé est admise ; une forme `Signature`, `Unknown` ou future devient une faute source sûre. Le payload Helius `transaction` n'est jamais copié dans l'état Worker : la projection conserve uniquement signature, slot et transaction index, puis réutilise le coordinateur d'hydration commun `getTransaction observed`.
La clé logique Helius inclut réseau, identités provider/endpoint sûres, commitment et empreinte privée du filtre. Les listes de pubkeys du filtre sont normalisées avant hash afin que leur ordre ne crée pas artificiellement deux sources logiques ; le rôle HTTP d'hydration reste exclu de l'identité live.
## Runtime et lifecycle
Le Worker s'exécute sur le runtime Tokio courant du caller. Il ne crée pas de runtime global et n'expose aucun `JoinHandle` public.
@@ -121,7 +145,7 @@ Le shutdown est borné par `shutdown_drain_timeout`. Les tâches source, hydrati
La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes subissent la backpressure ; aucune queue non bornée ni silent drop n'est autorisé.
Le Worker possède un coordinateur d'hydration source-neutral borné, réutilisé par Yellowstone et Standard Logs. Standard Block n'entre pas dans ce coordinateur lorsqu'une transaction est direct-qualified :
Le Worker possède un coordinateur d'hydration source-neutral borné, réutilisé par Yellowstone, Standard Logs et Helius Transaction. Standard Block n'entre pas dans ce coordinateur lorsqu'une transaction est direct-qualified :
```text
in-flight hydration <= persistence_concurrency
@@ -240,7 +264,7 @@ La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Sto
La verticale actuelle ne possède pas :
- plusieurs sources productives simultanées dans `RawTransactionIngestRuntimeResources` ;
- source `blockSubscribe`, Helius `transactionSubscribe` ou HTTP polling Worker ;
- source HTTP live polling Worker ;
- sélection Config interne au Worker ;
- checkpoint persistent de processing frontier ;
- campagne de réparation historique automatique ;

View File

@@ -1,5 +1,5 @@
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
<!-- version: 4 -->
<!-- version: 5 -->
# Utilisation de ksp-worker-raw-transaction-ingest-lib
@@ -101,7 +101,7 @@ let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWo
};
```
Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. Une exécution productive accepte une source unique Yellowstone, Standard Logs ou Standard Block ; une collection de plusieurs sources est validable à la composition mais reste rejetée avant spawn tant que la supervision simultanée n'est pas disponible.
Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. Une exécution productive accepte une source unique Yellowstone, Standard Logs, Standard Block ou Helius Transaction ; une collection de plusieurs sources est validable à la composition mais reste rejetée avant spawn tant que la supervision simultanée n'est pas disponible.
### Source Standard Logs productive
@@ -151,6 +151,36 @@ fn standard_block_runtime_resources(
Le Worker demande `Base64`, `Full`, `maxSupportedTransactionVersion = 1` et `showRewards = false`. Une transaction n'est RAW-direct que si sa version est explicitement `Legacy`, `0` ou `1`. Une version omise/nulle ou supérieure, `block: null`, une erreur de bloc, un champ transactions absent/nul ou une transaction non Base64 provoque une faute sûre ; ces cas ne sont jamais assimilés à une progression vide.
### Source Helius Transaction productive
Une source Helius LaserStream `transactionSubscribe` se compose avec le même pattern caller-owned :
```rust
fn helius_transaction_runtime_resources(
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
commitment: ksp_onchain_transport_lib::SolanaCommitment,
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new(
ws_endpoint,
filter,
commitment,
http_pool,
hydration_role,
) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source(source));
}
```
`ws_endpoint` doit être un endpoint Transport de kind `helius_laserstream`. Le caller supérieur résout éventuellement `KSP_SECRET_HELIUS_API_KEY` via Config avant de construire l'endpoint ; le Worker ne lit jamais l'environnement ni Config. Le commitment est limité à `Confirmed`/`Finalized` et la route HTTP doit supporter `getTransaction` sur le même réseau.
Le Worker demande la forme Helius `Full` avec `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`, mais ne fait pas confiance au nested payload pour construire directement le Common RAW. Il conserve seulement signature/slot/index et hydrate par `getTransaction observed`. Une notification d'une autre forme est fail-closed.
## Préparer la source Yellowstone
La `YellowstoneSubscribeRequest` doit :
@@ -174,6 +204,7 @@ Yellowstone TransactionStatus -> signal -> HTTP getTransaction -> Common RAW ->
Yellowstone Block -> un signal par transaction -> HTTP getTransaction -> Common RAW -> admission
Standard WS logsSubscribe -> context.slot + signature -> HTTP getTransaction -> Common RAW -> admission
Standard WS blockSubscribe -> Full/Base64 Legacy|V0|V1 -> Common RAW direct par transaction -> admission
Helius transactionSubscribe -> Full envelope -> signature/slot/index -> HTTP getTransaction -> Common RAW -> admission
Yellowstone BlockMeta -> continuity-only
Yellowstone Slot -> continuity-only
Yellowstone Account/Ping/Pong/Entry -> sans RAW Transaction dans cette verticale
@@ -316,7 +347,7 @@ Le pattern attendu est :
```text
Config / application / service owner
-> résout endpoints, credentials et rôles
-> construit une source Transport Yellowstone, Standard Logs ou Standard Block
-> construit une source Transport Yellowstone, Standard Logs, Standard Block ou Helius Transaction
-> construit HttpTransportPool + hydration role seulement pour les sources hydratées
-> construit le Store
-> construit RawTransactionIngestRuntimeResources

View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
// version: 21
// version: 22
#![warn(missing_docs)]
#![deny(unreachable_pub)]
@@ -10,10 +10,11 @@
//! This tranche owns the concrete Worker family identity, validated technical settings
//! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also
//! owns bounded source-neutral admission, common RAW canonicalization/assembly and backend-neutral
//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. `pre.004` keeps the bounded 1..32 caller-composed
//! aggregate and adds a productive standard Solana `blockSubscribe` direct RAW source beside Yellowstone and Standard Logs while simultaneous multi-source
//! activation remains gated until the dedicated supervisor tranche. Standard Logs and Yellowstone reference paths converge into one source-neutral hydration
//! coordinator contract; qualified Standard Block Legacy/V0/V1 transactions enter the existing central admission path directly. Yellowstone
//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. `pre.005` keeps the bounded 1..32 caller-composed
//! aggregate and adds a productive Helius `transactionSubscribe` + HTTP hydration source beside Yellowstone, Standard Logs and Standard Block while
//! simultaneous multi-source activation remains gated until the dedicated supervisor tranche. Helius Full notifications, Standard Logs and Yellowstone
//! reference paths converge into one source-neutral hydration coordinator contract; qualified Standard Block Legacy/V0/V1 transactions enter the existing
//! central admission path directly. Yellowstone
//! Transaction/TransactionStatus/Block and standard logs notifications become signature/slot references; BlockMeta/Slot remain continuity-only signals.
//! Hydration is coalesced by network/signature/commitment under bounded in-flight and pending budgets. A bounded run-local processing frontier projects
//! hydration pending, oldest pending slot and highest unblocked actually observed slot. The productive source also projects safe Transport reconnect/replay
@@ -52,6 +53,8 @@ pub use self::runtime::RawTransactionIngestTerminalFuture;
pub use self::runtime::RawTransactionIngestWorker;
/// Maximum number of logical live sources accepted by one runtime-resource aggregate.
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES;
/// Validated Helius `transactionSubscribe` + HTTP hydration source contract owned by the continuous RAW transaction ingest Worker.
pub use self::runtime_resources::RawTransactionIngestHeliusTransactionSource;
/// Caller-composed bounded runtime resources for supported continuous RAW transaction live-source families.
pub use self::runtime_resources::RawTransactionIngestRuntimeResources;
/// Validated standard Solana `blockSubscribe` direct RAW source contract owned by the continuous RAW transaction ingest Worker.

View File

@@ -1,11 +1,14 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
// version: 16
// version: 17
use sha2::Digest; // rust-rules: trait-import
/// Maximum number of logical live sources accepted in one RAW transaction ingest runtime-resource aggregate.
pub const MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES: usize = 32;
const RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.helius_transaction.filter.v1\0";
const RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL: &str = "helius_ws_http";
const RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.helius_transaction_http.source_key.v1\0";
const RAW_TRANSACTION_INGEST_LIVE_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.live_source.source_key.v1\0";
const RAW_TRANSACTION_INGEST_STANDARD_BLOCK_ACQUISITION_METHOD: &str = "block_subscribe";
const RAW_TRANSACTION_INGEST_STANDARD_BLOCK_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.standard_block.filter.v1\0";
@@ -26,6 +29,7 @@ enum RawTransactionIngestSourceFamily {
}
enum RawTransactionIngestLiveSource {
HeliusTransaction(crate::RawTransactionIngestHeliusTransactionSource),
StandardBlock(crate::RawTransactionIngestStandardBlockSource),
StandardLogs(crate::RawTransactionIngestStandardLogsSource),
Yellowstone(crate::RawTransactionIngestYellowstoneSource),
@@ -34,6 +38,7 @@ enum RawTransactionIngestLiveSource {
impl RawTransactionIngestLiveSource {
fn network(&self) -> &ksp_store_lib::RawNetworkId {
return match self {
Self::HeliusTransaction(source) => &source.network,
Self::StandardBlock(source) => &source.network,
Self::StandardLogs(source) => &source.network,
Self::Yellowstone(source) => &source.network,
@@ -42,6 +47,7 @@ impl RawTransactionIngestLiveSource {
fn source_key(&self) -> [u8; 32] {
return match self {
Self::HeliusTransaction(source) => source.source_key,
Self::StandardBlock(source) => source.source_key,
Self::StandardLogs(source) => source.source_key,
Self::Yellowstone(source) => source.source_key,
@@ -56,6 +62,7 @@ impl RawTransactionIngestLiveSource {
processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
) -> ksp_core_lib::Result<()> {
return match self {
Self::HeliusTransaction(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
Self::StandardBlock(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
Self::StandardLogs(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
Self::Yellowstone(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
@@ -578,6 +585,276 @@ impl crate::RawTransactionIngestYellowstoneSource {
}
}
/// Validated Helius `transactionSubscribe` plus HTTP hydration source owned by the continuous RAW transaction ingest Worker.
///
/// The caller provides one Transport-owned Helius LaserStream WebSocket endpoint, one Helius transaction filter, a Confirmed/Finalized commitment and one HTTP
/// hydration pool/role. The Worker always requests Full/Base64 notifications with `maxSupportedTransactionVersion = 1` but projects only the provider
/// signature/slot/index reference into the source-neutral hydration coordinator; the nested Helius payload is never copied into Worker state, while endpoint
/// URLs and credentials remain encapsulated by Transport-owned settings and are never exposed by Worker APIs or diagnostics.
pub struct RawTransactionIngestHeliusTransactionSource {
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
commitment: ksp_onchain_transport_lib::SolanaCommitment,
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
network: ksp_store_lib::RawNetworkId,
route: RawTransactionIngestSourceRoute,
filter_fingerprint: [u8; 32],
source_key: [u8; 32],
}
impl crate::RawTransactionIngestHeliusTransactionSource {
/// Creates one validated Helius transaction source without opening WebSocket or HTTP network I/O.
pub fn new(
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
commitment: ksp_onchain_transport_lib::SolanaCommitment,
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
) -> ksp_core_lib::Result<Self> {
let ws_settings = ksp_onchain_transport_lib::WsTransportSettings::new(std::vec![ws_endpoint.clone()]);
if ws_settings.validate().is_err() {
return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_endpoint_invalid"));
}
if ws_endpoint.protocol() != ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream {
return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_protocol_invalid"));
}
match commitment {
ksp_onchain_transport_lib::SolanaCommitment::Confirmed | ksp_onchain_transport_lib::SolanaCommitment::Finalized => {},
ksp_onchain_transport_lib::SolanaCommitment::Processed => {
return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_commitment_invalid"));
},
}
let request = helius_transaction_request(filter.clone(), commitment);
if request.validate().is_err() {
return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_request_invalid"));
}
let network = match ksp_store_lib::RawNetworkId::new(ws_endpoint.cluster().as_str()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => {
return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_network_unrepresentable"));
},
};
let provider = match ksp_store_lib::RawProvenanceCode::new(ws_endpoint.provider().as_str()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => {
return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_provider_unrepresentable"));
},
};
let endpoint_id = match ksp_store_lib::RawProvenanceCode::new(ws_endpoint.name()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => {
return std::result::Result::Err(crate::runtime_error("runtime_resources.helius_transaction_endpoint_unrepresentable"));
},
};
let route = RawTransactionIngestSourceRoute { endpoint_id, provider };
let method = match ksp_onchain_transport_lib::find_http_rpc_method("getTransaction") {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_method_missing")),
};
let compatible_http_routes = compatible_http_route_count(&http_pool, &hydration_role, method.request_kind(), network.as_str(), &route, "hx");
let compatible_http_routes = match compatible_http_routes {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(error),
};
if compatible_http_routes == 0 {
return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_role_unsupported"));
}
let filter_fingerprint = helius_transaction_filter_fingerprint(&filter);
let source_key = helius_transaction_live_source_key(&network, &route, commitment, &filter_fingerprint);
return std::result::Result::Ok(Self {
ws_endpoint,
filter,
commitment,
http_pool,
hydration_role,
network,
route,
filter_fingerprint,
source_key,
});
}
fn hydration_context(&self) -> RawTransactionIngestHydrationContext {
return RawTransactionIngestHydrationContext {
commitment: self.commitment,
http_pool: self.http_pool.clone(),
hydration_role: self.hydration_role.clone(),
network: self.network.clone(),
protocol: RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL,
route: self.route.clone(),
route_prefix: "hx",
source_key_domain: RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_SOURCE_KEY_DOMAIN,
};
}
/// Runs one productive Helius `transactionSubscribe` source until cooperative stop or one safe terminal source failure.
pub(crate) async fn run(
self,
settings: crate::RawTransactionIngestSettings,
mut stop_receiver: tokio::sync::watch::Receiver<bool>,
admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
) -> ksp_core_lib::Result<()> {
let connected = tokio::select! {
biased;
_ = stop_receiver.changed() => {
return std::result::Result::Ok(());
}
result = ksp_onchain_transport_lib::HeliusLaserStreamWsSession::connect(self.ws_endpoint.clone()) => result,
};
let session = match connected {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())),
};
let request = helius_transaction_request(self.filter.clone(), self.commitment);
let subscribed = tokio::select! {
biased;
_ = stop_receiver.changed() => {
let _closed = session.close().await;
return std::result::Result::Ok(());
}
result = session.transaction_subscribe(&request) => result,
};
let mut subscription = match subscribed {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
let _closed = session.close().await;
return std::result::Result::Err(source_transport_error(error.code()));
},
};
let hydration = self.hydration_context();
let mut coordinator = RawTransactionIngestHydrationCoordinator::new(&settings);
let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Active);
let mut fault = std::option::Option::None;
loop {
if *stop_receiver.borrow() {
break;
}
if let std::result::Result::Err(error) = coordinator.start_hydrations(&hydration) {
fault = std::option::Option::Some(error);
break;
}
let can_receive = coordinator.can_receive();
if !can_receive && coordinator.tasks.is_empty() {
fault = std::option::Option::Some(crate::runtime_error("source.hydration_stalled"));
break;
}
tokio::select! {
biased;
_ = stop_receiver.changed() => {
break;
}
joined = coordinator.tasks.join_next(), if !coordinator.tasks.is_empty() => {
let joined = match joined {
std::option::Option::Some(value) => value,
std::option::Option::None => {
fault = std::option::Option::Some(crate::runtime_error("source.hydration_join_missing"));
break;
},
};
let handled = coordinator
.handle_joined(
joined,
&hydration,
&settings,
&admission_sender,
&mut stop_receiver,
&mut processing_frontier,
)
.await;
match handled {
std::result::Result::Ok(true) => {},
std::result::Result::Ok(false) => break,
std::result::Result::Err(error) => {
fault = std::option::Option::Some(error);
break;
},
}
}
notification = subscription.recv(), if can_receive => {
let notification = match notification {
std::option::Option::Some(std::result::Result::Ok(value)) => value,
std::option::Option::Some(std::result::Result::Err(error)) => {
fault = std::option::Option::Some(source_transport_error(error.code()));
break;
},
std::option::Option::None => {
let error = match subscription.terminal_error_code() {
std::option::Option::Some(code) => source_transport_error(code),
std::option::Option::None => crate::runtime_error("source.helius_transaction_subscription_closed"),
};
fault = std::option::Option::Some(error);
break;
},
};
let received_at = match current_raw_timestamp() {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
fault = std::option::Option::Some(error);
break;
},
};
let full = match notification {
ksp_onchain_transport_lib::HeliusTransactionNotification::Full(value) => value,
_ => {
fault = std::option::Option::Some(crate::runtime_error("source.helius_transaction_notification_unqualified"));
break;
},
};
let signal = match project_helius_transaction_signal(&self, &full) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(error) => {
fault = std::option::Option::Some(error);
break;
},
};
if let std::result::Result::Err(error) = coordinator.queue_signal(&hydration, signal, received_at, &mut processing_frontier) {
fault = std::option::Option::Some(error);
break;
}
}
}
}
coordinator.abort_all(&mut processing_frontier).await;
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closing);
let closed = session.close().await;
if let std::option::Option::Some(error) = fault {
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed);
return std::result::Result::Err(error);
}
return match closed {
std::result::Result::Ok(()) => {
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closed);
std::result::Result::Ok(())
},
std::result::Result::Err(error) => {
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed);
std::result::Result::Err(source_transport_error(error.code()))
},
};
}
}
impl std::fmt::Debug for crate::RawTransactionIngestHeliusTransactionSource {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let http_snapshot = self.http_pool.snapshot();
return formatter
.debug_struct("RawTransactionIngestHeliusTransactionSource")
.field("ws_endpoint_name", &self.ws_endpoint.name())
.field("ws_provider", &self.ws_endpoint.provider().as_str())
.field("network", &self.network.as_str())
.field("commitment", &self.commitment)
.field("filter", &self.filter)
.field("filter_fingerprint_bytes", &self.filter_fingerprint.len())
.field("hydration_role", &self.hydration_role.as_str())
.field("http_endpoint_count", &http_snapshot.endpoint_count())
.field("source_key_bytes", &self.source_key.len())
.finish();
}
}
/// Validated standard Solana `blockSubscribe` direct RAW source owned by the continuous RAW transaction ingest Worker.
///
/// The caller provides one Transport-owned standard WebSocket endpoint, one block filter and a Confirmed/Finalized commitment. The source requests Full/Base64
@@ -1055,8 +1332,8 @@ impl std::fmt::Debug for crate::RawTransactionIngestYellowstoneSource {
/// Caller-composed runtime resources accepted by the continuous RAW transaction ingest Worker.
///
/// The aggregate owns a bounded collection of capability-specific live-source contracts. `pre.004` supports Yellowstone, standard Solana logs and standard
/// Solana block sources while deliberately keeping simultaneous multi-source supervision gated until the dedicated supervisor tranche.
/// The aggregate owns a bounded collection of capability-specific live-source contracts. `pre.005` supports Yellowstone, standard Solana logs/block and
/// Helius transaction sources while deliberately keeping simultaneous multi-source supervision gated until the dedicated supervisor tranche.
pub struct RawTransactionIngestRuntimeResources {
sources: std::vec::Vec<RawTransactionIngestLiveSource>,
}
@@ -1068,6 +1345,12 @@ impl crate::RawTransactionIngestRuntimeResources {
return Self { sources: std::vec![RawTransactionIngestLiveSource::Yellowstone(yellowstone_source)] };
}
/// Creates one runtime-resource aggregate from one Helius `transactionSubscribe` plus HTTP hydration source contract.
#[must_use]
pub fn from_helius_transaction_source(source: crate::RawTransactionIngestHeliusTransactionSource) -> Self {
return Self { sources: std::vec![RawTransactionIngestLiveSource::HeliusTransaction(source)] };
}
/// Creates one runtime-resource aggregate from one standard Solana `blockSubscribe` direct RAW source contract.
#[must_use]
pub fn from_standard_block_source(source: crate::RawTransactionIngestStandardBlockSource) -> Self {
@@ -1107,6 +1390,27 @@ impl crate::RawTransactionIngestRuntimeResources {
return std::result::Result::Ok(());
}
/// Adds one validated Helius transaction source while preserving the global 1..32 bound, one-network invariant and unique logical source identity.
pub fn try_push_helius_transaction_source(&mut self, source: crate::RawTransactionIngestHeliusTransactionSource) -> ksp_core_lib::Result<()> {
if self.sources.len() >= crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES {
return std::result::Result::Err(crate::runtime_error("runtime_resources.source_count_exceeded"));
}
let candidate = RawTransactionIngestLiveSource::HeliusTransaction(source);
let expected_network = match self.sources.first() {
std::option::Option::Some(source) => source.network(),
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty")),
};
if candidate.network() != expected_network {
return std::result::Result::Err(crate::runtime_error("runtime_resources.source_network_mismatch"));
}
let source_key = candidate.source_key();
if self.sources.iter().any(|source| return source.source_key() == source_key) {
return std::result::Result::Err(crate::runtime_error("runtime_resources.duplicate_source_identity"));
}
self.sources.push(candidate);
return std::result::Result::Ok(());
}
/// Adds one validated standard Solana block source while preserving the global 1..32 bound, one-network invariant and unique logical source identity.
pub fn try_push_standard_block_source(&mut self, source: crate::RawTransactionIngestStandardBlockSource) -> ksp_core_lib::Result<()> {
if self.sources.len() >= crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES {
@@ -1218,6 +1522,93 @@ impl std::hash::Hasher for RawTransactionIngestSourceKeyHashWriter<'_> {
}
}
fn helius_transaction_filter_fingerprint(filter: &ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter) -> [u8; 32] {
let mut hasher = sha2::Sha256::new();
hasher.update(RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_FILTER_FINGERPRINT_DOMAIN);
hash_helius_optional_bool(&mut hasher, b"failed", filter.failed());
hash_helius_optional_bool(&mut hasher, b"vote", filter.vote());
hash_helius_optional_pubkey_list(&mut hasher, b"account_exclude", filter.account_exclude());
hash_helius_optional_pubkey_list(&mut hasher, b"account_include", filter.account_include());
hash_helius_optional_pubkey_list(&mut hasher, b"account_required", filter.account_required());
hash_live_source_key_component(&mut hasher, b"signature");
match filter.signature() {
std::option::Option::Some(value) => {
hash_live_source_key_component(&mut hasher, b"value");
hash_live_source_key_component(&mut hasher, value.as_bytes());
},
std::option::Option::None => hash_live_source_key_component(&mut hasher, b"omitted"),
}
hash_live_source_key_component(&mut hasher, b"token_accounts");
match filter.token_accounts() {
std::option::Option::Some(value) => {
hash_live_source_key_component(&mut hasher, b"value");
hash_live_source_key_component(&mut hasher, value.as_str().as_bytes());
},
std::option::Option::None => hash_live_source_key_component(&mut hasher, b"omitted"),
}
return hasher.finalize().into();
}
fn helius_transaction_live_source_key(
network: &ksp_store_lib::RawNetworkId,
route: &RawTransactionIngestSourceRoute,
commitment: ksp_onchain_transport_lib::SolanaCommitment,
filter_fingerprint: &[u8; 32],
) -> [u8; 32] {
let mut hasher = sha2::Sha256::new();
hasher.update(RAW_TRANSACTION_INGEST_LIVE_SOURCE_KEY_DOMAIN);
hash_live_source_key_component(&mut hasher, b"helius_transaction");
hash_live_source_key_component(&mut hasher, network.as_str().as_bytes());
hash_live_source_key_component(&mut hasher, route.provider.as_str().as_bytes());
hash_live_source_key_component(&mut hasher, route.endpoint_id.as_str().as_bytes());
hash_live_source_key_component(&mut hasher, commitment.as_str().as_bytes());
hash_live_source_key_component(&mut hasher, filter_fingerprint);
return hasher.finalize().into();
}
fn helius_transaction_request(
filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
commitment: ksp_onchain_transport_lib::SolanaCommitment,
) -> ksp_onchain_transport_lib::HeliusTransactionSubscribeRequest {
let options = ksp_onchain_transport_lib::HeliusTransactionSubscribeOptions::new(
std::option::Option::Some(commitment),
std::option::Option::Some(ksp_onchain_transport_lib::HeliusTransactionSubscribeEncoding::Base64),
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full),
std::option::Option::Some(false),
std::option::Option::Some(1),
);
return ksp_onchain_transport_lib::HeliusTransactionSubscribeRequest::new(filter, std::option::Option::Some(options));
}
fn hash_helius_optional_bool(hasher: &mut sha2::Sha256, label: &[u8], value: std::option::Option<bool>) {
hash_live_source_key_component(hasher, label);
match value {
std::option::Option::Some(true) => hash_live_source_key_component(hasher, b"true"),
std::option::Option::Some(false) => hash_live_source_key_component(hasher, b"false"),
std::option::Option::None => hash_live_source_key_component(hasher, b"omitted"),
}
return;
}
fn hash_helius_optional_pubkey_list(hasher: &mut sha2::Sha256, label: &[u8], values: std::option::Option<&[ksp_core_lib::Pubkey]>) {
hash_live_source_key_component(hasher, label);
let values = match values {
std::option::Option::Some(values) => values,
std::option::Option::None => {
hash_live_source_key_component(hasher, b"omitted");
return;
},
};
hash_live_source_key_component(hasher, b"value");
let mut normalized = values.iter().map(|value| return value.to_bytes()).collect::<std::vec::Vec<_>>();
normalized.sort_unstable();
hash_live_source_key_component(hasher, &(normalized.len() as u64).to_be_bytes());
for value in normalized {
hash_live_source_key_component(hasher, &value);
}
return;
}
fn standard_block_filter_fingerprint(filter: &ksp_onchain_transport_lib::SolanaBlockSubscribeFilter) -> [u8; 32] {
let mut hasher = sha2::Sha256::new();
hasher.update(RAW_TRANSACTION_INGEST_STANDARD_BLOCK_FILTER_FINGERPRINT_DOMAIN);
@@ -1666,6 +2057,51 @@ fn build_standard_block_provenance(
);
}
trait RawTransactionIngestHeliusTransactionView {
fn signature(&self) -> &str;
fn slot(&self) -> u64;
fn transaction_index(&self) -> u64;
}
impl RawTransactionIngestHeliusTransactionView for ksp_onchain_transport_lib::HeliusFullTransactionNotification {
fn signature(&self) -> &str {
return ksp_onchain_transport_lib::HeliusFullTransactionNotification::signature(self);
}
fn slot(&self) -> u64 {
return ksp_onchain_transport_lib::HeliusFullTransactionNotification::slot(self);
}
fn transaction_index(&self) -> u64 {
return ksp_onchain_transport_lib::HeliusFullTransactionNotification::transaction_index(self);
}
}
fn project_helius_transaction_signal<T: RawTransactionIngestHeliusTransactionView>(
source: &crate::RawTransactionIngestHeliusTransactionSource,
response: &T,
) -> ksp_core_lib::Result<RawTransactionIngestSourceSignal> {
let signature = ksp_raw_transaction_lib::parse_raw_transaction_signature(response.signature());
let signature = match signature {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.helius_transaction_signature_invalid")),
};
return std::result::Result::Ok(RawTransactionIngestSourceSignal {
created_at: std::option::Option::None,
family: RawTransactionIngestSourceFamily::Transaction,
matched_filter_count: 1,
matched_filter_fingerprint: source.filter_fingerprint,
matched_filter_id: std::option::Option::None,
network: source.network.clone(),
route: source.route.clone(),
signature,
slot: response.slot(),
transaction_index: std::option::Option::Some(response.transaction_index()),
});
}
trait RawTransactionIngestStandardLogsView {
fn signature(&self) -> &str;

View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
// version: 20
// version: 21
//! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
@@ -296,6 +296,44 @@ fn v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only
return;
}
#[test]
fn v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only() {
let root = include_str!("../src/lib.rs");
let runtime = include_str!("../src/runtime.rs");
let resources = include_str!("../src/runtime_resources.rs");
for required in [
"RawTransactionIngestHeliusTransactionSource",
"HeliusLaserStreamWsSession::connect",
".transaction_subscribe(",
"HeliusTransactionSubscribeOptions::new",
"HeliusTransactionSubscribeEncoding::Base64",
"SolanaTransactionDetails::Full",
"std::option::Option::Some(1)",
"project_helius_transaction_signal",
"helius_ws_http",
"get_transaction_observed",
"max_pending_signals: settings.admission_queue_capacity()",
] {
assert!(resources.contains(required), "required pre.005 Helius transaction contract missing: {required}");
}
assert_eq!(resources.matches("get_transaction_observed(").count(), 1);
for forbidden in [
"ksp_config_lib::",
"ksp_job_backfill_lib::",
"ksp_store_postgres_lib::",
"reqwest::",
"tokio_tungstenite::",
"helius_sdk",
"unbounded_channel",
] {
assert!(
!runtime.contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
"pre.005 crossed a forbidden Helius transaction boundary: {forbidden}",
);
}
return;
}
fn runtime_contains(value: &str) -> bool {
return include_str!("../src/runtime.rs").contains(value);
}

View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
// version: 13
// version: 14
//! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
@@ -612,6 +612,46 @@ fn v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit
return;
}
#[test]
fn v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit() {
let resources = include_str!("../src/runtime_resources.rs");
for required in [
"RawTransactionIngestHeliusTransactionSource",
".field(\"filter\"",
".field(\"filter_fingerprint_bytes\"",
".field(\"source_key_bytes\"",
"HeliusTransactionNotification::Full(value)",
"source.helius_transaction_notification_unqualified",
"project_helius_transaction_signal",
"transaction_index: std::option::Option::Some(response.transaction_index())",
"RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL",
] {
assert!(resources.contains(required), "required pre.005 Helius hardening guard missing: {required}");
}
let source_struct = match resources.split_once("pub struct RawTransactionIngestHeliusTransactionSource {") {
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestHeliusTransactionSource") {
std::option::Option::Some((value, _)) => value,
std::option::Option::None => "",
},
std::option::Option::None => "",
};
for forbidden in ["url:", "api_key", "credential", "transaction:", "payload:", "tier:"] {
assert!(!source_struct.contains(forbidden), "sensitive/provider material stored in Helius transaction source: {forbidden}");
}
let projection = match resources.split_once("fn project_helius_transaction_signal") {
std::option::Option::Some((_, tail)) => match tail.split_once("trait RawTransactionIngestStandardLogsView") {
std::option::Option::Some((value, _)) => value,
std::option::Option::None => "",
},
std::option::Option::None => "",
};
assert!(!projection.contains(".transaction()"), "Helius nested full payload must not enter Worker projection");
for forbidden in ["Developer", "Business", "Professional", "paid_tier", "provider_tier"] {
assert!(!resources.contains(forbidden), "provider tier must not be coded in Worker: {forbidden}");
}
return;
}
#[test]
fn v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit() {
let resources = include_str!("../src/runtime_resources.rs");

View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
// version: 14
// version: 15
//! External public-surface proofs for the RAW transaction ingest Worker foundation.
@@ -224,6 +224,33 @@ fn v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport
return;
}
#[test]
fn v0_3_13_pre_005_helius_transaction_runtime_resource_surface_is_typed_and_transport_owned() {
let _source_new: fn(
ksp_onchain_transport_lib::WsEndpointSettings,
ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
ksp_onchain_transport_lib::SolanaCommitment,
ksp_onchain_transport_lib::HttpTransportPool,
ksp_onchain_transport_lib::HttpRoleName,
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource> =
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new;
let _resources_from_helius: fn(
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource,
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source;
let _push_helius: fn(
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource,
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_helius_transaction_source;
let root = include_str!("../src/lib.rs");
let resources = include_str!("../src/runtime_resources.rs");
assert!(root.contains("RawTransactionIngestHeliusTransactionSource"));
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn http_pool(", "pub fn source_key("] {
assert!(!resources.contains(forbidden), "Helius transaction source implementation escape hatch present: {forbidden}");
}
return;
}
#[test]
fn v0_3_13_pre_004_standard_block_runtime_resource_surface_is_typed_and_transport_owned() {
let _source_new: fn(

View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
// version: 12
// version: 13
//! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
@@ -78,6 +78,7 @@ fn pre_010_public_root_export_inventory_is_exact() {
"MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
"RAW_TRANSACTION_INGEST_WORKER_KIND_CODE",
"RawTransactionIngestHandle",
"RawTransactionIngestHeliusTransactionSource",
"RawTransactionIngestRuntimeResources",
"RawTransactionIngestSettings",
"RawTransactionIngestSnapshot",
@@ -117,6 +118,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
"v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths",
"v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit",
"v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit",
"v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit",
] {
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
}
@@ -132,6 +134,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
assert!(dependency_boundary.contains("v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap"));
assert!(dependency_boundary.contains("v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only"));
assert!(dependency_boundary.contains("v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only"));
assert!(dependency_boundary.contains("v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only"));
let public_api = include_str!("public_api.rs");
assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));

View File

@@ -1,5 +1,5 @@
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
// version: 16
// version: 17
fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
@@ -126,6 +126,65 @@ fn standard_logs_source(
};
}
fn helius_transaction_filter(failed: bool) -> ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter {
return ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new(
std::option::Option::Some(false),
std::option::Option::Some(failed),
std::option::Option::None,
std::option::Option::None,
std::option::Option::None,
std::option::Option::None,
std::option::Option::None,
);
}
fn helius_transaction_source(
cluster: &str,
endpoint_name: &str,
filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
commitment: ksp_onchain_transport_lib::SolanaCommitment,
hydration_role: &str,
) -> std::option::Option<crate::RawTransactionIngestHeliusTransactionSource> {
let endpoint = match ws_endpoint(cluster, endpoint_name, "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream) {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::option::Option::None,
};
let pool = match http_pool(cluster, hydration_role, "get_transaction") {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::option::Option::None,
};
return match crate::RawTransactionIngestHeliusTransactionSource::new(
endpoint,
filter,
commitment,
pool,
ksp_onchain_transport_lib::HttpRoleName::new(hydration_role),
) {
std::result::Result::Ok(value) => std::option::Option::Some(value),
std::result::Result::Err(_) => std::option::Option::None,
};
}
struct HeliusTransactionViewFixture {
signature: std::string::String,
slot: u64,
transaction_index: u64,
}
impl super::RawTransactionIngestHeliusTransactionView for HeliusTransactionViewFixture {
fn signature(&self) -> &str {
return self.signature.as_str();
}
fn slot(&self) -> u64 {
return self.slot;
}
fn transaction_index(&self) -> u64 {
return self.transaction_index;
}
}
struct StandardLogsViewFixture {
signature: std::string::String,
slot: u64,
@@ -434,6 +493,28 @@ fn standard_logs_source_for_http_url(
};
}
fn helius_transaction_source_for_http_url(url: &str) -> std::option::Option<crate::RawTransactionIngestHeliusTransactionSource> {
let endpoint =
match ws_endpoint("devnet", "helius-transaction-fixture", "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream) {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::option::Option::None,
};
let pool = match http_pool_for_url(url, "devnet", "http-hydration-fixture", "fixture-http-provider") {
std::option::Option::Some(value) => value,
std::option::Option::None => return std::option::Option::None,
};
return match crate::RawTransactionIngestHeliusTransactionSource::new(
endpoint,
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
pool,
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
) {
std::result::Result::Ok(value) => std::option::Option::Some(value),
std::result::Result::Err(_) => std::option::Option::None,
};
}
fn pre_004_settings() -> std::option::Option<crate::RawTransactionIngestSettings> {
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
std::result::Result::Ok(value) => value,
@@ -2073,6 +2154,309 @@ fn v0_3_13_pre_004_runtime_resources_reject_duplicate_standard_block_identity()
return;
}
#[test]
fn v0_3_13_pre_005_helius_transaction_source_validates_protocol_commitment_request_and_hydration_route() {
let source = match helius_transaction_source(
"devnet",
"helius-transaction-a",
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
"hydration",
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let request = super::helius_transaction_request(source.filter.clone(), source.commitment);
let options = match request.options() {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
assert_eq!(options.commitment(), std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed));
assert_eq!(options.encoding(), std::option::Option::Some(ksp_onchain_transport_lib::HeliusTransactionSubscribeEncoding::Base64));
assert_eq!(options.transaction_details(), std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full));
assert_eq!(options.show_rewards(), std::option::Option::Some(false));
assert_eq!(options.max_supported_transaction_version(), std::option::Option::Some(1));
assert!(request.validate().is_ok());
let resources = crate::RawTransactionIngestRuntimeResources::from_helius_transaction_source(source);
assert_eq!(resources.source_count(), 1);
let standard_endpoint =
match ws_endpoint("devnet", "helius-transaction-standard", "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let pool = match http_pool("devnet", "hydration", "get_transaction") {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let protocol_error = crate::RawTransactionIngestHeliusTransactionSource::new(
standard_endpoint,
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
pool.clone(),
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
);
let protocol_error = match protocol_error {
std::result::Result::Ok(_) => return,
std::result::Result::Err(value) => value,
};
assert!(protocol_error.context().iter().any(|context| return context.value() == "runtime_resources.helius_transaction_protocol_invalid"));
let helius_endpoint =
match ws_endpoint("devnet", "helius-transaction-processed", "fixture-helius-provider", ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let commitment_error = crate::RawTransactionIngestHeliusTransactionSource::new(
helius_endpoint,
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Processed,
pool,
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
);
let commitment_error = match commitment_error {
std::result::Result::Ok(_) => return,
std::result::Result::Err(value) => value,
};
assert!(commitment_error.context().iter().any(|context| return context.value() == "runtime_resources.hydration_commitment_invalid"));
let cross_network_endpoint = match ws_endpoint(
"devnet",
"helius-transaction-cross-network",
"fixture-helius-provider",
ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream,
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let cross_network_pool = match http_pool("testnet", "hydration", "get_transaction") {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let cross_network = crate::RawTransactionIngestHeliusTransactionSource::new(
cross_network_endpoint,
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
cross_network_pool,
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
);
let cross_network = match cross_network {
std::result::Result::Ok(_) => return,
std::result::Result::Err(value) => value,
};
assert!(cross_network.context().iter().any(|context| return context.value() == "runtime_resources.transport_network_mismatch"));
return;
}
#[test]
fn v0_3_13_pre_005_helius_transaction_source_key_is_filter_sensitive_order_normalized_hydration_role_neutral_and_redacted() {
let first_pubkey = ksp_core_lib::Pubkey::new_from_array([0x31_u8; 32]);
let second_pubkey = ksp_core_lib::Pubkey::new_from_array([0x32_u8; 32]);
let signature_canary = "5HueCGU8rMjxEXxiPuD5BDuRaH7RkZBzQ2hL5M5WnQp9";
let first_filter = ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new(
std::option::Option::Some(false),
std::option::Option::Some(false),
std::option::Option::Some(signature_canary.to_owned()),
std::option::Option::Some(std::vec![first_pubkey, second_pubkey]),
std::option::Option::None,
std::option::Option::None,
std::option::Option::Some(ksp_onchain_transport_lib::HeliusTokenAccountsFilter::BalanceChanged),
);
let reordered_filter = ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new(
std::option::Option::Some(false),
std::option::Option::Some(false),
std::option::Option::Some(signature_canary.to_owned()),
std::option::Option::Some(std::vec![second_pubkey, first_pubkey]),
std::option::Option::None,
std::option::Option::None,
std::option::Option::Some(ksp_onchain_transport_lib::HeliusTokenAccountsFilter::BalanceChanged),
);
let changed_filter = ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::new(
std::option::Option::Some(false),
std::option::Option::Some(true),
std::option::Option::Some(signature_canary.to_owned()),
std::option::Option::Some(std::vec![first_pubkey, second_pubkey]),
std::option::Option::None,
std::option::Option::None,
std::option::Option::Some(ksp_onchain_transport_lib::HeliusTokenAccountsFilter::BalanceChanged),
);
let first = match helius_transaction_source(
"devnet",
"helius-transaction-key",
first_filter,
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
"hydration",
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let reordered = match helius_transaction_source(
"devnet",
"helius-transaction-key",
reordered_filter,
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
"secondary_hydration",
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let changed = match helius_transaction_source(
"devnet",
"helius-transaction-key",
changed_filter,
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
"hydration",
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
assert_eq!(first.source_key, reordered.source_key);
assert_ne!(first.source_key, changed.source_key);
let first_pubkey_text = first_pubkey.to_string();
let debug = std::format!("{first:?}");
assert!(debug.contains("RawTransactionIngestHeliusTransactionSource"));
assert!(debug.contains("signature_present"));
assert!(!debug.contains(signature_canary));
assert!(!debug.contains(first_pubkey_text.as_str()));
assert!(!debug.contains("WS-SECRET-CANARY"));
assert!(!debug.contains("api-key"));
assert!(!debug.contains("source_key: ["));
return;
}
#[test]
fn v0_3_13_pre_005_helius_full_projection_keeps_only_reference_material() {
let source = match helius_transaction_source(
"devnet",
"helius-transaction-projection",
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Finalized,
"hydration",
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let fixture = HeliusTransactionViewFixture { signature: PRE_004_ZERO_SIGNATURE_TEXT.to_owned(), slot: 81, transaction_index: 9 };
let signal = match super::project_helius_transaction_signal(&source, &fixture) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
assert_eq!(signal.family, super::RawTransactionIngestSourceFamily::Transaction);
assert_eq!(signal.slot, 81);
assert_eq!(signal.transaction_index, std::option::Option::Some(9));
assert_eq!(signal.created_at, std::option::Option::None);
assert_eq!(signal.matched_filter_count, 1);
assert_eq!(signal.matched_filter_id, std::option::Option::None);
assert_eq!(signal.signature, ksp_store_lib::RawTransactionSignature::new([0_u8; 64]));
let debug = std::format!("{signal:?}");
assert!(!debug.contains(PRE_004_ZERO_SIGNATURE_TEXT));
return;
}
#[tokio::test(flavor = "current_thread")]
async fn v0_3_13_pre_005_helius_full_reference_hydrates_once_to_exact_common_raw() {
let body = std::format!(
"{{\"jsonrpc\":\"2.0\",\"result\":{{\"slot\":42,\"blockTime\":1760000120,\"transaction\":[\"{}\",\"base64\"],\"meta\":null,\"version\":1,\"transactionIndex\":7}},\"id\":1}}",
PRE_004_ZERO_TRANSACTION_BASE64,
);
let (url, server) = match serve_http_once(body) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
let source = match helius_transaction_source_for_http_url(url.as_str()) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let settings = match pre_004_settings() {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let fixture = HeliusTransactionViewFixture { signature: PRE_004_ZERO_SIGNATURE_TEXT.to_owned(), slot: 42, transaction_index: 7 };
let signal = match super::project_helius_transaction_signal(&source, &fixture) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_760_000_200_000) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
let hydration = source.hydration_context();
let key = match super::hydration_key(&hydration, &signal) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
let fetched =
super::fetch_hydration(hydration.http_pool.clone(), hydration.hydration_role.clone(), hydration.network.clone(), key, hydration.commitment).await;
let fetched = match fetched {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
let ingress = super::finalize_hydration(&hydration, &settings, signal, received_at, &fetched.observed);
let ingress = match ingress {
std::result::Result::Ok(std::option::Option::Some(value)) => value,
std::result::Result::Ok(std::option::Option::None) | std::result::Result::Err(_) => return,
};
assert_eq!(ingress.provenance.protocol().as_str(), "helius_ws_http");
assert_eq!(ingress.provenance.acquisition_method().as_str(), "transaction_get_transaction");
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
if sender.send(ingress).await.is_err() {
return;
}
std::mem::drop(sender);
let acquisition = match admission.receive(settings.network()).await {
std::result::Result::Ok(std::option::Option::Some(value)) => value,
_ => return,
};
assert_eq!(acquisition.transaction().reference().network().as_str(), "devnet");
assert_eq!(acquisition.transaction().reference().signature().as_bytes(), &[0_u8; 64]);
assert_eq!(acquisition.transaction().slot(), 42);
assert_eq!(acquisition.transaction().block_time().map(|value| return value.unix_millis()), std::option::Option::Some(1_760_000_120_000));
let payload = match std::str::from_utf8(acquisition.transaction().payload().bytes()) {
std::result::Result::Ok(value) => value,
std::result::Result::Err(_) => return,
};
assert!(payload.contains("\"version\":1"));
assert!(payload.contains("\"transactionIndex\":7"));
let request_body = match server.join() {
std::result::Result::Ok(std::result::Result::Ok(value)) => value,
std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => return,
};
assert!(request_body.contains("\"method\":\"getTransaction\""));
assert!(request_body.contains("\"encoding\":\"base64\""));
assert!(request_body.contains("\"maxSupportedTransactionVersion\":1"));
return;
}
#[test]
fn v0_3_13_pre_005_runtime_resources_reject_duplicate_helius_transaction_identity() {
let first = match helius_transaction_source(
"devnet",
"helius-transaction-duplicate",
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
"hydration",
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let duplicate = match helius_transaction_source(
"devnet",
"helius-transaction-duplicate",
helius_transaction_filter(false),
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
"secondary_hydration",
) {
std::option::Option::Some(value) => value,
std::option::Option::None => return,
};
let mut resources = crate::RawTransactionIngestRuntimeResources::from_helius_transaction_source(first);
let result = resources.try_push_helius_transaction_source(duplicate);
let error = match result {
std::result::Result::Ok(()) => return,
std::result::Result::Err(value) => value,
};
assert!(error.context().iter().any(|context| return context.value() == "runtime_resources.duplicate_source_identity"));
assert_eq!(resources.source_count(), 1);
return;
}
#[test]
fn pre_007_processing_frontier_never_advances_through_pending_work() {
let mut frontier = super::RawTransactionIngestProcessingFrontier::new();