v0.3.13-pre.003
This commit is contained in:
@@ -1,12 +1,12 @@
|
|||||||
# file: Cargo.toml
|
# file: Cargo.toml
|
||||||
# version: 541
|
# version: 542
|
||||||
|
|
||||||
[workspace]
|
[workspace]
|
||||||
resolver = "3"
|
resolver = "3"
|
||||||
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api", "crates/ksp-worker-raw-transaction-ingest-lib"]
|
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api", "crates/ksp-worker-raw-transaction-ingest-lib"]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
version = "0.3.13-pre.2.fix.1"
|
version = "0.3.13-pre.3"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# ksp-worker-raw-transaction-ingest-lib
|
# ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
@@ -12,7 +12,8 @@ RawTransactionIngestWorker::start
|
|||||||
-> fondation source-neutral, sans source productive
|
-> fondation source-neutral, sans source productive
|
||||||
|
|
||||||
RawTransactionIngestWorker::start_with_runtime_resources
|
RawTransactionIngestWorker::start_with_runtime_resources
|
||||||
-> même runtime + une source Yellowstone productive supervisée
|
-> même runtime + une source productive supervisée
|
||||||
|
Yellowstone ou WS standard logsSubscribe
|
||||||
+ hydration HTTP getTransaction
|
+ hydration HTTP getTransaction
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -20,14 +21,20 @@ Le Worker reste indépendant de Config et de tout backend Store physique. Le cal
|
|||||||
|
|
||||||
## Pipeline productif actuel
|
## Pipeline productif actuel
|
||||||
|
|
||||||
La première verticale live est :
|
Les verticales live productives sont :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Yellowstone standard subscribe
|
Yellowstone standard subscribe
|
||||||
-> Transaction / TransactionStatus / Block
|
-> Transaction / TransactionStatus / Block
|
||||||
-> signal source-neutral (network, signature, slot, commitment, provenance sûre)
|
-> signal source-neutral (network, signature, slot, commitment, provenance sûre)
|
||||||
|
|
||||||
|
Solana standard WS logsSubscribe
|
||||||
|
-> context.slot + signature
|
||||||
|
-> signal source-neutral (network, signature, slot, commitment, provenance sûre)
|
||||||
|
|
||||||
|
les deux chemins
|
||||||
-> coalescence bornée par (network, signature, commitment)
|
-> coalescence bornée par (network, signature, commitment)
|
||||||
-> HTTP getTransaction observed
|
-> HTTP getTransaction observed, Base64, maxSupportedTransactionVersion=1
|
||||||
-> ksp-raw-transaction-lib
|
-> ksp-raw-transaction-lib
|
||||||
-> admission centrale bornée
|
-> admission centrale bornée
|
||||||
-> ksp-store-lib
|
-> ksp-store-lib
|
||||||
@@ -58,7 +65,23 @@ La construction est sans I/O et refuse notamment :
|
|||||||
- une provenance provider/endpoint non représentable ;
|
- une provenance provider/endpoint non représentable ;
|
||||||
- l'absence d'une route HTTP compatible pour `getTransaction` sur le même réseau.
|
- l'absence d'une route HTTP compatible pour `getTransaction` sur le même réseau.
|
||||||
|
|
||||||
Le runtime-resource aggregate public contient exactement une source Yellowstone validée. Il n'expose ni enum provider, ni collection de sources, ni callback, ni queue d'enqueue, ni client inférieur.
|
Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques. `pre.003` sait exécuter une source unique Yellowstone ou une source unique Standard Logs ; une activation simultanée de plusieurs sources reste rejetée avant spawn jusqu'au supervisor dédié. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés.
|
||||||
|
|
||||||
|
## Contrat de source Standard Logs + HTTP
|
||||||
|
|
||||||
|
`RawTransactionIngestStandardLogsSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaLogsSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O. Elle refuse un endpoint WS invalide ou non standard, `Processed`, un réseau/provenance non représentable et l'absence de route HTTP `getTransaction` compatible sur le même réseau. Le filtre `All`, `AllWithVotes` ou `Mentions(pubkey)` participe uniquement à une empreinte privée ; le pubkey d'un filtre `Mentions` n'est pas recopié dans `Debug`, snapshot ou provenance textuelle.
|
||||||
|
|
||||||
|
Au runtime, le Worker ouvre `SolanaStandardWsSession::connect`, puis `logs_subscribe`. Les lignes de logs et `err` restent dans Transport et ne sont jamais stockées dans le signal Worker. Seuls `context.slot` et `signature` sont projetés vers l'hydration commune. Reconnect, resubscribe et backpressure de la subscription restent possédés par Transport.
|
||||||
|
|
||||||
## Runtime et lifecycle
|
## Runtime et lifecycle
|
||||||
|
|
||||||
@@ -77,11 +100,11 @@ Le shutdown est borné par `shutdown_drain_timeout`. Les tâches source, hydrati
|
|||||||
|
|
||||||
La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes subissent la backpressure ; aucune queue non bornée ni silent drop n'est autorisé.
|
La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes subissent la backpressure ; aucune queue non bornée ni silent drop n'est autorisé.
|
||||||
|
|
||||||
La source Yellowstone possède également un coordinateur d'hydration borné :
|
Le Worker possède un coordinateur d'hydration source-neutral borné, réutilisé par Yellowstone et Standard Logs :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
in-flight hydration <= persistence_concurrency
|
in-flight hydration <= persistence_concurrency
|
||||||
pending source signals <= 65_536
|
pending source signals <= admission_queue_capacity
|
||||||
```
|
```
|
||||||
|
|
||||||
Les signaux partageant le même `(network, signature, commitment)` sont coalescés avant le fan-out HTTP. Les provenances utiles restent néanmoins conservées pour les ingress produits après hydration.
|
Les signaux partageant le même `(network, signature, commitment)` sont coalescés avant le fan-out HTTP. Les provenances utiles restent néanmoins conservées pour les ingress produits après hydration.
|
||||||
@@ -196,7 +219,7 @@ La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Sto
|
|||||||
La verticale actuelle ne possède pas :
|
La verticale actuelle ne possède pas :
|
||||||
|
|
||||||
- plusieurs sources productives simultanées dans `RawTransactionIngestRuntimeResources` ;
|
- plusieurs sources productives simultanées dans `RawTransactionIngestRuntimeResources` ;
|
||||||
- source WS/Helius-specific ou HTTP polling Worker ;
|
- source `blockSubscribe`, Helius `transactionSubscribe` ou HTTP polling Worker ;
|
||||||
- sélection Config interne au Worker ;
|
- sélection Config interne au Worker ;
|
||||||
- checkpoint persistent de processing frontier ;
|
- checkpoint persistent de processing frontier ;
|
||||||
- campagne de réparation historique automatique ;
|
- campagne de réparation historique automatique ;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# Utilisation de ksp-worker-raw-transaction-ingest-lib
|
# Utilisation de ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
@@ -101,7 +101,35 @@ let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWo
|
|||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que la source Yellowstone/HTTP cible ce même réseau.
|
Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. Une exécution productive accepte une source unique Yellowstone ou Standard Logs ; une collection de plusieurs sources est validable à la composition mais reste rejetée avant spawn tant que la supervision simultanée n'est pas disponible.
|
||||||
|
|
||||||
|
### Source Standard Logs productive
|
||||||
|
|
||||||
|
Une source standard Solana WS se compose ainsi :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn standard_logs_runtime_resources(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
ws_endpoint,
|
||||||
|
filter,
|
||||||
|
commitment,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_logs_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`ws_endpoint` doit être un endpoint Transport valide de kind `solana_standard`. Le commitment doit être explicitement `Confirmed` ou `Finalized`. Le pool HTTP doit exposer `getTransaction` via le rôle indiqué sur le même réseau. `All`, `AllWithVotes` et `Mentions(pubkey)` sont acceptés par le contrat Transport ; la valeur du filtre reste privée dans le Worker.
|
||||||
|
|
||||||
## Préparer la source Yellowstone
|
## Préparer la source Yellowstone
|
||||||
|
|
||||||
@@ -118,15 +146,16 @@ Le caller ne passe pas de signature, `program_id`, plage de slots ou limite hist
|
|||||||
|
|
||||||
## Comprendre la pipeline live
|
## Comprendre la pipeline live
|
||||||
|
|
||||||
Les familles Yellowstone sont traitées ainsi :
|
Les familles productives sont traitées ainsi :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Transaction -> signal -> HTTP getTransaction -> Common RAW -> admission
|
Yellowstone Transaction -> signal -> HTTP getTransaction -> Common RAW -> admission
|
||||||
TransactionStatus -> signal -> HTTP getTransaction -> Common RAW -> admission
|
Yellowstone TransactionStatus -> signal -> HTTP getTransaction -> Common RAW -> admission
|
||||||
Block -> un signal par transaction -> HTTP getTransaction -> Common RAW -> admission
|
Yellowstone Block -> un signal par transaction -> HTTP getTransaction -> Common RAW -> admission
|
||||||
BlockMeta -> continuity-only
|
Standard WS logsSubscribe -> context.slot + signature -> HTTP getTransaction -> Common RAW -> admission
|
||||||
Slot -> continuity-only
|
Yellowstone BlockMeta -> continuity-only
|
||||||
Account/Ping/Pong/Entry -> sans RAW Transaction dans cette verticale
|
Yellowstone Slot -> continuity-only
|
||||||
|
Yellowstone Account/Ping/Pong/Entry -> sans RAW Transaction dans cette verticale
|
||||||
```
|
```
|
||||||
|
|
||||||
Les signaux de même `(network, signature, commitment)` sont coalescés avant l'hydration HTTP. Le Worker ne possède pas une boucle de retry HTTP : reroutage/retry/backoff restent dans `ksp-onchain-transport-lib`.
|
Les signaux de même `(network, signature, commitment)` sont coalescés avant l'hydration HTTP. Le Worker ne possède pas une boucle de retry HTTP : reroutage/retry/backoff restent dans `ksp-onchain-transport-lib`.
|
||||||
@@ -266,7 +295,7 @@ Le pattern attendu est :
|
|||||||
```text
|
```text
|
||||||
Config / application / service owner
|
Config / application / service owner
|
||||||
-> résout endpoints, credentials et rôles
|
-> résout endpoints, credentials et rôles
|
||||||
-> construit YellowstoneGrpcChannel + YellowstoneSubscribeRequest
|
-> construit une source Transport Yellowstone ou Standard Logs
|
||||||
-> construit HttpTransportPool + hydration role
|
-> construit HttpTransportPool + hydration role
|
||||||
-> construit le Store
|
-> construit le Store
|
||||||
-> construit RawTransactionIngestRuntimeResources
|
-> construit RawTransactionIngestRuntimeResources
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
// version: 19
|
// version: 20
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
#![deny(unreachable_pub)]
|
#![deny(unreachable_pub)]
|
||||||
@@ -10,12 +10,11 @@
|
|||||||
//! This tranche owns the concrete Worker family identity, validated technical settings
|
//! This tranche owns the concrete Worker family identity, validated technical settings
|
||||||
//! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also
|
//! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also
|
||||||
//! owns bounded source-neutral admission, common RAW canonicalization/assembly and backend-neutral
|
//! owns bounded source-neutral admission, common RAW canonicalization/assembly and backend-neutral
|
||||||
//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. `pre.002` adds a bounded 1..32 caller-composed live-source
|
//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. `pre.003` keeps the bounded 1..32 caller-composed
|
||||||
//! aggregate with deterministic logical source identities while deliberately keeping productive multi-source activation gated until the supervisor tranche; the
|
//! aggregate and adds a productive standard Solana `logsSubscribe` + observed HTTP `getTransaction` source beside Yellowstone while simultaneous multi-source
|
||||||
//! validated Yellowstone/HTTP runtime-resource contract still drives the only productive supervised source task.
|
//! activation remains gated until the dedicated supervisor tranche. Both live source families converge into one source-neutral hydration coordinator contract and
|
||||||
//! Transaction/TransactionStatus/Block updates feed bounded HTTP `getTransaction` hydration and the existing central
|
//! the existing central admission path. Yellowstone Transaction/TransactionStatus/Block and standard logs notifications become signature/slot references;
|
||||||
//! admission path; BlockMeta/Slot remain continuity-only signals. Hydration is coalesced by network/signature/commitment
|
//! BlockMeta/Slot remain continuity-only signals. Hydration is coalesced by network/signature/commitment under bounded in-flight and pending budgets. A bounded run-local processing frontier projects hydration pending,
|
||||||
//! under bounded in-flight and pending budgets. A bounded run-local processing frontier projects hydration pending,
|
|
||||||
//! oldest pending slot and highest unblocked actually observed slot. The productive source now also projects safe Transport reconnect/replay state and faults
|
//! oldest pending slot and highest unblocked actually observed slot. The productive source now also projects safe Transport reconnect/replay state and faults
|
||||||
//! conservatively when Transport proves a replay-retention continuity gap; history remediation remains outside this crate.
|
//! conservatively when Transport proves a replay-retention continuity gap; history remediation remains outside this crate.
|
||||||
|
|
||||||
@@ -52,8 +51,10 @@ pub use self::runtime::RawTransactionIngestTerminalFuture;
|
|||||||
pub use self::runtime::RawTransactionIngestWorker;
|
pub use self::runtime::RawTransactionIngestWorker;
|
||||||
/// Maximum number of logical live sources accepted by one runtime-resource aggregate.
|
/// Maximum number of logical live sources accepted by one runtime-resource aggregate.
|
||||||
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES;
|
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES;
|
||||||
/// Caller-composed runtime resources for the first Yellowstone + HTTP source family.
|
/// Caller-composed bounded runtime resources for supported continuous RAW transaction live-source families.
|
||||||
pub use self::runtime_resources::RawTransactionIngestRuntimeResources;
|
pub use self::runtime_resources::RawTransactionIngestRuntimeResources;
|
||||||
|
/// Validated standard Solana `logsSubscribe` + HTTP hydration source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestStandardLogsSource;
|
||||||
/// Validated Yellowstone + HTTP source contract owned by the continuous RAW transaction ingest Worker.
|
/// Validated Yellowstone + HTTP source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
pub use self::runtime_resources::RawTransactionIngestYellowstoneSource;
|
pub use self::runtime_resources::RawTransactionIngestYellowstoneSource;
|
||||||
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
// version: 12
|
// version: 13
|
||||||
|
|
||||||
type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
|
type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
|
||||||
type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
|
type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
|
||||||
@@ -99,8 +99,8 @@ impl crate::RawTransactionIngestWorker {
|
|||||||
|
|
||||||
/// Starts one Worker with caller-composed runtime resources.
|
/// Starts one Worker with caller-composed runtime resources.
|
||||||
///
|
///
|
||||||
/// `pre.002` accepts a validated multi-source aggregate but intentionally activates only the existing single-Yellowstone supervisor path; aggregates with
|
/// `pre.003` activates either the existing Yellowstone source or one standard Solana `logsSubscribe` source through the same bounded Worker foundation.
|
||||||
/// more than one source fail closed before any source task is spawned.
|
/// Aggregates with more than one source still fail closed before any source task is spawned until the dedicated simultaneous supervisor tranche.
|
||||||
pub fn start_with_runtime_resources(
|
pub fn start_with_runtime_resources(
|
||||||
settings: crate::RawTransactionIngestSettings,
|
settings: crate::RawTransactionIngestSettings,
|
||||||
store: std::sync::Arc<ksp_store_lib::Store>,
|
store: std::sync::Arc<ksp_store_lib::Store>,
|
||||||
@@ -117,10 +117,9 @@ impl crate::RawTransactionIngestWorker {
|
|||||||
if let std::result::Result::Err(error) = runtime_resources.validate_network(settings.network()) {
|
if let std::result::Result::Err(error) = runtime_resources.validate_network(settings.network()) {
|
||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
}
|
}
|
||||||
let source = match runtime_resources.into_yellowstone_source() {
|
if let std::result::Result::Err(error) = runtime_resources.validate_single_source_activation() {
|
||||||
std::result::Result::Ok(value) => value,
|
return std::result::Result::Err(error);
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
}
|
||||||
};
|
|
||||||
let source_settings = settings.clone();
|
let source_settings = settings.clone();
|
||||||
let (processing_frontier_sender, processing_frontier_receiver) =
|
let (processing_frontier_sender, processing_frontier_receiver) =
|
||||||
tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||||
@@ -132,7 +131,7 @@ impl crate::RawTransactionIngestWorker {
|
|||||||
std::option::Option::Some(processing_frontier_receiver),
|
std::option::Option::Some(processing_frontier_receiver),
|
||||||
move |children, stop_receiver, admission_sender| {
|
move |children, stop_receiver, admission_sender| {
|
||||||
let _abort_handle = children.spawn(async move {
|
let _abort_handle = children.spawn(async move {
|
||||||
return source.run(source_settings, stop_receiver, admission_sender, processing_frontier_sender).await;
|
return runtime_resources.run_single_live_source(source_settings, stop_receiver, admission_sender, processing_frontier_sender).await;
|
||||||
});
|
});
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
// version: 13
|
// version: 14
|
||||||
|
|
||||||
use sha2::Digest; // rust-rules: trait-import
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
@@ -7,6 +7,9 @@ use sha2::Digest; // rust-rules: trait-import
|
|||||||
pub const MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES: usize = 32;
|
pub const MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES: usize = 32;
|
||||||
|
|
||||||
const RAW_TRANSACTION_INGEST_LIVE_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.live_source.source_key.v1\0";
|
const RAW_TRANSACTION_INGEST_LIVE_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.live_source.source_key.v1\0";
|
||||||
|
const RAW_TRANSACTION_INGEST_STANDARD_LOGS_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.standard_logs.filter.v1\0";
|
||||||
|
const RAW_TRANSACTION_INGEST_STANDARD_LOGS_HTTP_PROTOCOL: &str = "solana_ws_http";
|
||||||
|
const RAW_TRANSACTION_INGEST_STANDARD_LOGS_HTTP_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.standard_logs_http.source_key.v1\0";
|
||||||
const RAW_TRANSACTION_INGEST_YELLOWSTONE_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.yellowstone.filters.v1\0";
|
const RAW_TRANSACTION_INGEST_YELLOWSTONE_FILTER_FINGERPRINT_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.yellowstone.filters.v1\0";
|
||||||
const RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_PROTOCOL: &str = "yellowstone_http";
|
const RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_PROTOCOL: &str = "yellowstone_http";
|
||||||
const RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.yellowstone_http.source_key.v1\0";
|
const RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.yellowstone_http.source_key.v1\0";
|
||||||
@@ -14,26 +17,43 @@ const RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_SOURCE_KEY_DOMAIN: &[u8] = b"ksp.r
|
|||||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
enum RawTransactionIngestSourceFamily {
|
enum RawTransactionIngestSourceFamily {
|
||||||
Block,
|
Block,
|
||||||
|
Logs,
|
||||||
Transaction,
|
Transaction,
|
||||||
TransactionStatus,
|
TransactionStatus,
|
||||||
}
|
}
|
||||||
|
|
||||||
enum RawTransactionIngestLiveSource {
|
enum RawTransactionIngestLiveSource {
|
||||||
|
StandardLogs(crate::RawTransactionIngestStandardLogsSource),
|
||||||
Yellowstone(crate::RawTransactionIngestYellowstoneSource),
|
Yellowstone(crate::RawTransactionIngestYellowstoneSource),
|
||||||
}
|
}
|
||||||
|
|
||||||
impl RawTransactionIngestLiveSource {
|
impl RawTransactionIngestLiveSource {
|
||||||
fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
||||||
return match self {
|
return match self {
|
||||||
|
Self::StandardLogs(source) => &source.network,
|
||||||
Self::Yellowstone(source) => &source.network,
|
Self::Yellowstone(source) => &source.network,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
fn source_key(&self) -> [u8; 32] {
|
fn source_key(&self) -> [u8; 32] {
|
||||||
return match self {
|
return match self {
|
||||||
|
Self::StandardLogs(source) => source.source_key,
|
||||||
Self::Yellowstone(source) => source.source_key,
|
Self::Yellowstone(source) => source.source_key,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn run(
|
||||||
|
self,
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
stop_receiver: tokio::sync::watch::Receiver<bool>,
|
||||||
|
admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
|
||||||
|
processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
return match self {
|
||||||
|
Self::StandardLogs(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
|
||||||
|
Self::Yellowstone(source) => source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await,
|
||||||
|
};
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
@@ -321,6 +341,18 @@ impl std::convert::From<(&crate::RawTransactionIngestYellowstoneSource, &ksp_onc
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Clone)]
|
||||||
|
struct RawTransactionIngestHydrationContext {
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
protocol: &'static str,
|
||||||
|
route: RawTransactionIngestSourceRoute,
|
||||||
|
route_prefix: &'static str,
|
||||||
|
source_key_domain: &'static [u8],
|
||||||
|
}
|
||||||
|
|
||||||
/// Validated Yellowstone plus HTTP runtime source owned by the continuous RAW transaction ingest Worker.
|
/// Validated Yellowstone plus HTTP runtime source owned by the continuous RAW transaction ingest Worker.
|
||||||
///
|
///
|
||||||
/// The Transport-owned channel, subscribe request, HTTP pool and hydration role remain private. Construction validates deterministic source-composition
|
/// The Transport-owned channel, subscribe request, HTTP pool and hydration role remain private. Construction validates deterministic source-composition
|
||||||
@@ -373,7 +405,7 @@ impl crate::RawTransactionIngestYellowstoneSource {
|
|||||||
std::option::Option::Some(value) => value,
|
std::option::Option::Some(value) => value,
|
||||||
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_method_missing")),
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_method_missing")),
|
||||||
};
|
};
|
||||||
let compatible_http_routes = compatible_http_route_count(&http_pool, &hydration_role, method.request_kind(), network.as_str(), &route);
|
let compatible_http_routes = compatible_http_route_count(&http_pool, &hydration_role, method.request_kind(), network.as_str(), &route, "ys");
|
||||||
let compatible_http_routes = match compatible_http_routes {
|
let compatible_http_routes = match compatible_http_routes {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
@@ -389,6 +421,23 @@ impl crate::RawTransactionIngestYellowstoneSource {
|
|||||||
return std::result::Result::Ok(Self { yellowstone_channel, subscribe_request, http_pool, hydration_role, network, route, source_key });
|
return std::result::Result::Ok(Self { yellowstone_channel, subscribe_request, http_pool, hydration_role, network, route, source_key });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn hydration_context(&self) -> RawTransactionIngestHydrationContext {
|
||||||
|
let commitment = match self.subscribe_request.commitment() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
};
|
||||||
|
return RawTransactionIngestHydrationContext {
|
||||||
|
commitment,
|
||||||
|
http_pool: self.http_pool.clone(),
|
||||||
|
hydration_role: self.hydration_role.clone(),
|
||||||
|
network: self.network.clone(),
|
||||||
|
protocol: RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_PROTOCOL,
|
||||||
|
route: self.route.clone(),
|
||||||
|
route_prefix: "ys",
|
||||||
|
source_key_domain: RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_SOURCE_KEY_DOMAIN,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/// Runs the productive Yellowstone source task until cooperative stop or one safe terminal source failure.
|
/// Runs the productive Yellowstone source task until cooperative stop or one safe terminal source failure.
|
||||||
pub(crate) async fn run(
|
pub(crate) async fn run(
|
||||||
self,
|
self,
|
||||||
@@ -408,6 +457,7 @@ impl crate::RawTransactionIngestYellowstoneSource {
|
|||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())),
|
std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())),
|
||||||
};
|
};
|
||||||
|
let hydration = self.hydration_context();
|
||||||
let mut coordinator = RawTransactionIngestHydrationCoordinator::new(&settings);
|
let mut coordinator = RawTransactionIngestHydrationCoordinator::new(&settings);
|
||||||
let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
|
let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
|
||||||
let snapshot_source = session.snapshot_source();
|
let snapshot_source = session.snapshot_source();
|
||||||
@@ -422,7 +472,7 @@ impl crate::RawTransactionIngestYellowstoneSource {
|
|||||||
if *stop_receiver.borrow() {
|
if *stop_receiver.borrow() {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
if let std::result::Result::Err(error) = coordinator.start_hydrations(&self) {
|
if let std::result::Result::Err(error) = coordinator.start_hydrations(&hydration) {
|
||||||
fault = std::option::Option::Some(error);
|
fault = std::option::Option::Some(error);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -460,7 +510,7 @@ impl crate::RawTransactionIngestYellowstoneSource {
|
|||||||
let handled = coordinator
|
let handled = coordinator
|
||||||
.handle_joined(
|
.handle_joined(
|
||||||
joined,
|
joined,
|
||||||
&self,
|
&hydration,
|
||||||
&settings,
|
&settings,
|
||||||
&admission_sender,
|
&admission_sender,
|
||||||
&mut stop_receiver,
|
&mut stop_receiver,
|
||||||
@@ -490,6 +540,7 @@ impl crate::RawTransactionIngestYellowstoneSource {
|
|||||||
};
|
};
|
||||||
if let std::result::Result::Err(error) = route_yellowstone_update(
|
if let std::result::Result::Err(error) = route_yellowstone_update(
|
||||||
&self,
|
&self,
|
||||||
|
&hydration,
|
||||||
&mut coordinator,
|
&mut coordinator,
|
||||||
&mut processing_frontier,
|
&mut processing_frontier,
|
||||||
update,
|
update,
|
||||||
@@ -520,6 +571,258 @@ impl crate::RawTransactionIngestYellowstoneSource {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Validated standard Solana `logsSubscribe` plus HTTP hydration source owned by the continuous RAW transaction ingest Worker.
|
||||||
|
///
|
||||||
|
/// The caller provides one Transport-owned standard WebSocket endpoint, one standard logs filter, a Confirmed/Finalized commitment and one HTTP hydration
|
||||||
|
/// pool/role. The source keeps the WebSocket URL and filter private, never copies remote logs/errors into Worker state, and emits only signature/slot
|
||||||
|
/// references into the source-neutral hydration coordinator.
|
||||||
|
pub struct RawTransactionIngestStandardLogsSource {
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
route: RawTransactionIngestSourceRoute,
|
||||||
|
filter_fingerprint: [u8; 32],
|
||||||
|
source_key: [u8; 32],
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestStandardLogsSource {
|
||||||
|
/// Creates one validated standard Solana logs source without opening WebSocket or HTTP network I/O.
|
||||||
|
pub fn new(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<Self> {
|
||||||
|
let ws_settings = ksp_onchain_transport_lib::WsTransportSettings::new(std::vec![ws_endpoint.clone()]);
|
||||||
|
if ws_settings.validate().is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.standard_logs_endpoint_invalid"));
|
||||||
|
}
|
||||||
|
if ws_endpoint.protocol() != ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.standard_logs_protocol_invalid"));
|
||||||
|
}
|
||||||
|
match commitment {
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed | ksp_onchain_transport_lib::SolanaCommitment::Finalized => {},
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Processed => {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_commitment_invalid"));
|
||||||
|
},
|
||||||
|
}
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(ws_endpoint.cluster().as_str()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("runtime_resources.standard_logs_network_unrepresentable")),
|
||||||
|
};
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new(ws_endpoint.provider().as_str()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("runtime_resources.standard_logs_provider_unrepresentable")),
|
||||||
|
};
|
||||||
|
let endpoint_id = match ksp_store_lib::RawProvenanceCode::new(ws_endpoint.name()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("runtime_resources.standard_logs_endpoint_unrepresentable")),
|
||||||
|
};
|
||||||
|
let route = RawTransactionIngestSourceRoute { endpoint_id, provider };
|
||||||
|
let method = match ksp_onchain_transport_lib::find_http_rpc_method("getTransaction") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_method_missing")),
|
||||||
|
};
|
||||||
|
let compatible_http_routes = compatible_http_route_count(&http_pool, &hydration_role, method.request_kind(), network.as_str(), &route, "ws");
|
||||||
|
let compatible_http_routes = match compatible_http_routes {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if compatible_http_routes == 0 {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.hydration_role_unsupported"));
|
||||||
|
}
|
||||||
|
let filter_fingerprint = standard_logs_filter_fingerprint(&filter);
|
||||||
|
let source_key = standard_logs_live_source_key(&network, &route, commitment, &filter_fingerprint);
|
||||||
|
return std::result::Result::Ok(Self {
|
||||||
|
ws_endpoint,
|
||||||
|
filter,
|
||||||
|
commitment,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
network,
|
||||||
|
route,
|
||||||
|
filter_fingerprint,
|
||||||
|
source_key,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hydration_context(&self) -> RawTransactionIngestHydrationContext {
|
||||||
|
return RawTransactionIngestHydrationContext {
|
||||||
|
commitment: self.commitment,
|
||||||
|
http_pool: self.http_pool.clone(),
|
||||||
|
hydration_role: self.hydration_role.clone(),
|
||||||
|
network: self.network.clone(),
|
||||||
|
protocol: RAW_TRANSACTION_INGEST_STANDARD_LOGS_HTTP_PROTOCOL,
|
||||||
|
route: self.route.clone(),
|
||||||
|
route_prefix: "ws",
|
||||||
|
source_key_domain: RAW_TRANSACTION_INGEST_STANDARD_LOGS_HTTP_SOURCE_KEY_DOMAIN,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs one productive standard `logsSubscribe` source until cooperative stop or one safe terminal source failure.
|
||||||
|
pub(crate) async fn run(
|
||||||
|
self,
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
mut stop_receiver: tokio::sync::watch::Receiver<bool>,
|
||||||
|
admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
|
||||||
|
processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let connected = tokio::select! {
|
||||||
|
biased;
|
||||||
|
_ = stop_receiver.changed() => {
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
result = ksp_onchain_transport_lib::SolanaStandardWsSession::connect(self.ws_endpoint.clone()) => result,
|
||||||
|
};
|
||||||
|
let session = match connected {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(source_transport_error(error.code())),
|
||||||
|
};
|
||||||
|
let config = ksp_onchain_transport_lib::SolanaCommitmentConfig::new(std::option::Option::Some(self.commitment));
|
||||||
|
let subscribed = tokio::select! {
|
||||||
|
biased;
|
||||||
|
_ = stop_receiver.changed() => {
|
||||||
|
let _closed = session.close().await;
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
result = session.logs_subscribe(&self.filter, std::option::Option::Some(&config)) => result,
|
||||||
|
};
|
||||||
|
let mut subscription = match subscribed {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
let _closed = session.close().await;
|
||||||
|
return std::result::Result::Err(source_transport_error(error.code()));
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let hydration = self.hydration_context();
|
||||||
|
let mut coordinator = RawTransactionIngestHydrationCoordinator::new(&settings);
|
||||||
|
let mut processing_frontier = RawTransactionIngestProcessingFrontierReporter::new(processing_frontier_sender);
|
||||||
|
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Active);
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
loop {
|
||||||
|
if *stop_receiver.borrow() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = coordinator.start_hydrations(&hydration) {
|
||||||
|
fault = std::option::Option::Some(error);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
let can_receive = coordinator.can_receive();
|
||||||
|
if !can_receive && coordinator.tasks.is_empty() {
|
||||||
|
fault = std::option::Option::Some(crate::runtime_error("source.hydration_stalled"));
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::select! {
|
||||||
|
biased;
|
||||||
|
_ = stop_receiver.changed() => {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
joined = coordinator.tasks.join_next(), if !coordinator.tasks.is_empty() => {
|
||||||
|
let joined = match joined {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => {
|
||||||
|
fault = std::option::Option::Some(crate::runtime_error("source.hydration_join_missing"));
|
||||||
|
break;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let handled = coordinator
|
||||||
|
.handle_joined(
|
||||||
|
joined,
|
||||||
|
&hydration,
|
||||||
|
&settings,
|
||||||
|
&admission_sender,
|
||||||
|
&mut stop_receiver,
|
||||||
|
&mut processing_frontier,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
match handled {
|
||||||
|
std::result::Result::Ok(true) => {},
|
||||||
|
std::result::Result::Ok(false) => break,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fault = std::option::Option::Some(error);
|
||||||
|
break;
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
notification = subscription.recv(), if can_receive => {
|
||||||
|
let notification = match notification {
|
||||||
|
std::option::Option::Some(std::result::Result::Ok(value)) => value,
|
||||||
|
std::option::Option::Some(std::result::Result::Err(error)) => {
|
||||||
|
fault = std::option::Option::Some(source_transport_error(error.code()));
|
||||||
|
break;
|
||||||
|
},
|
||||||
|
std::option::Option::None => {
|
||||||
|
let error = match subscription.terminal_error_code() {
|
||||||
|
std::option::Option::Some(code) => source_transport_error(code),
|
||||||
|
std::option::Option::None => crate::runtime_error("source.standard_logs_subscription_closed"),
|
||||||
|
};
|
||||||
|
fault = std::option::Option::Some(error);
|
||||||
|
break;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let received_at = match current_raw_timestamp() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fault = std::option::Option::Some(error);
|
||||||
|
break;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let signal = match project_standard_logs_signal(&self, ¬ification) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fault = std::option::Option::Some(error);
|
||||||
|
break;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
if let std::result::Result::Err(error) = coordinator.queue_signal(&hydration, signal, received_at, &mut processing_frontier) {
|
||||||
|
fault = std::option::Option::Some(error);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
coordinator.abort_all(&mut processing_frontier).await;
|
||||||
|
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closing);
|
||||||
|
let closed = session.close().await;
|
||||||
|
if let std::option::Option::Some(error) = fault {
|
||||||
|
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed);
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
return match closed {
|
||||||
|
std::result::Result::Ok(()) => {
|
||||||
|
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Closed);
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
processing_frontier.set_source_state(crate::RawTransactionIngestSourceState::Failed);
|
||||||
|
std::result::Result::Err(source_transport_error(error.code()))
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestStandardLogsSource {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let http_snapshot = self.http_pool.snapshot();
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestStandardLogsSource")
|
||||||
|
.field("ws_endpoint_name", &self.ws_endpoint.name())
|
||||||
|
.field("ws_provider", &self.ws_endpoint.provider().as_str())
|
||||||
|
.field("network", &self.network.as_str())
|
||||||
|
.field("commitment", &self.commitment)
|
||||||
|
.field("filter_kind", &standard_logs_filter_kind(&self.filter))
|
||||||
|
.field("filter_fingerprint_bytes", &self.filter_fingerprint.len())
|
||||||
|
.field("hydration_role", &self.hydration_role.as_str())
|
||||||
|
.field("http_endpoint_count", &http_snapshot.endpoint_count())
|
||||||
|
.field("source_key_bytes", &self.source_key.len())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl std::fmt::Debug for crate::RawTransactionIngestYellowstoneSource {
|
impl std::fmt::Debug for crate::RawTransactionIngestYellowstoneSource {
|
||||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
let http_snapshot = self.http_pool.snapshot();
|
let http_snapshot = self.http_pool.snapshot();
|
||||||
@@ -545,8 +848,8 @@ impl std::fmt::Debug for crate::RawTransactionIngestYellowstoneSource {
|
|||||||
|
|
||||||
/// Caller-composed runtime resources accepted by the continuous RAW transaction ingest Worker.
|
/// Caller-composed runtime resources accepted by the continuous RAW transaction ingest Worker.
|
||||||
///
|
///
|
||||||
/// The aggregate owns a bounded collection of capability-specific live-source contracts. `pre.002` validates composition and stable logical identities but
|
/// The aggregate owns a bounded collection of capability-specific live-source contracts. `pre.003` supports Yellowstone and standard Solana logs sources while
|
||||||
/// deliberately keeps productive multi-source supervision gated until the dedicated supervisor tranche.
|
/// deliberately keeping simultaneous multi-source supervision gated until the dedicated supervisor tranche.
|
||||||
pub struct RawTransactionIngestRuntimeResources {
|
pub struct RawTransactionIngestRuntimeResources {
|
||||||
sources: std::vec::Vec<RawTransactionIngestLiveSource>,
|
sources: std::vec::Vec<RawTransactionIngestLiveSource>,
|
||||||
}
|
}
|
||||||
@@ -558,6 +861,12 @@ impl crate::RawTransactionIngestRuntimeResources {
|
|||||||
return Self { sources: std::vec![RawTransactionIngestLiveSource::Yellowstone(yellowstone_source)] };
|
return Self { sources: std::vec![RawTransactionIngestLiveSource::Yellowstone(yellowstone_source)] };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Creates one runtime-resource aggregate from one standard Solana `logsSubscribe` plus HTTP hydration source contract.
|
||||||
|
#[must_use]
|
||||||
|
pub fn from_standard_logs_source(source: crate::RawTransactionIngestStandardLogsSource) -> Self {
|
||||||
|
return Self { sources: std::vec![RawTransactionIngestLiveSource::StandardLogs(source)] };
|
||||||
|
}
|
||||||
|
|
||||||
/// Returns the number of validated logical live sources currently owned by this aggregate.
|
/// Returns the number of validated logical live sources currently owned by this aggregate.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn source_count(&self) -> usize {
|
pub fn source_count(&self) -> usize {
|
||||||
@@ -585,6 +894,27 @@ impl crate::RawTransactionIngestRuntimeResources {
|
|||||||
return std::result::Result::Ok(());
|
return std::result::Result::Ok(());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Adds one validated standard Solana logs source while preserving the global 1..32 bound, one-network invariant and unique logical source identity.
|
||||||
|
pub fn try_push_standard_logs_source(&mut self, source: crate::RawTransactionIngestStandardLogsSource) -> ksp_core_lib::Result<()> {
|
||||||
|
if self.sources.len() >= crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.source_count_exceeded"));
|
||||||
|
}
|
||||||
|
let candidate = RawTransactionIngestLiveSource::StandardLogs(source);
|
||||||
|
let expected_network = match self.sources.first() {
|
||||||
|
std::option::Option::Some(source) => source.network(),
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty")),
|
||||||
|
};
|
||||||
|
if candidate.network() != expected_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.source_network_mismatch"));
|
||||||
|
}
|
||||||
|
let source_key = candidate.source_key();
|
||||||
|
if self.sources.iter().any(|source| return source.source_key() == source_key) {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.duplicate_source_identity"));
|
||||||
|
}
|
||||||
|
self.sources.push(candidate);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
/// Validates that caller-owned Worker settings target the same logical network as every composed live source.
|
/// Validates that caller-owned Worker settings target the same logical network as every composed live source.
|
||||||
pub(crate) fn validate_network(&self, network: &ksp_store_lib::RawNetworkId) -> ksp_core_lib::Result<()> {
|
pub(crate) fn validate_network(&self, network: &ksp_store_lib::RawNetworkId) -> ksp_core_lib::Result<()> {
|
||||||
if self.sources.is_empty() || self.sources.len() > crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES {
|
if self.sources.is_empty() || self.sources.len() > crate::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES {
|
||||||
@@ -602,22 +932,34 @@ impl crate::RawTransactionIngestRuntimeResources {
|
|||||||
return std::result::Result::Ok(());
|
return std::result::Result::Ok(());
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Consumes the aggregate into the single Yellowstone source supported by the current productive supervisor wiring.
|
/// Validates that the current supervisor tranche can activate the aggregate without silently dropping configured sources.
|
||||||
///
|
pub(crate) fn validate_single_source_activation(&self) -> ksp_core_lib::Result<()> {
|
||||||
/// Multi-source activation is intentionally deferred to the dedicated supervisor/source-inventory tranche; a collection larger than one therefore fails
|
|
||||||
/// closed rather than silently dropping sources or starting only a subset.
|
|
||||||
pub(crate) fn into_yellowstone_source(self) -> ksp_core_lib::Result<crate::RawTransactionIngestYellowstoneSource> {
|
|
||||||
if self.sources.is_empty() {
|
if self.sources.is_empty() {
|
||||||
return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty"));
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty"));
|
||||||
}
|
}
|
||||||
if self.sources.len() != 1 {
|
if self.sources.len() != 1 {
|
||||||
return std::result::Result::Err(crate::runtime_error("runtime_resources.multi_source_activation_pending"));
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.multi_source_activation_pending"));
|
||||||
}
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs the already validated single live source through the common Worker source-task contract.
|
||||||
|
pub(crate) async fn run_single_live_source(
|
||||||
|
self,
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
stop_receiver: tokio::sync::watch::Receiver<bool>,
|
||||||
|
admission_sender: tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
|
||||||
|
processing_frontier_sender: tokio::sync::watch::Sender<crate::RawTransactionIngestProcessingFrontierProjection>,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
let mut sources = self.sources.into_iter();
|
let mut sources = self.sources.into_iter();
|
||||||
return match sources.next() {
|
let source = match sources.next() {
|
||||||
std::option::Option::Some(RawTransactionIngestLiveSource::Yellowstone(source)) => std::result::Result::Ok(source),
|
std::option::Option::Some(value) => value,
|
||||||
std::option::Option::None => std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty")),
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("runtime_resources.source_collection_empty")),
|
||||||
};
|
};
|
||||||
|
if sources.next().is_some() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.multi_source_activation_pending"));
|
||||||
|
}
|
||||||
|
return source.run(settings, stop_receiver, admission_sender, processing_frontier_sender).await;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -642,6 +984,45 @@ impl std::hash::Hasher for RawTransactionIngestSourceKeyHashWriter<'_> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn standard_logs_filter_fingerprint(filter: &ksp_onchain_transport_lib::SolanaLogsSubscribeFilter) -> [u8; 32] {
|
||||||
|
let mut hasher = sha2::Sha256::new();
|
||||||
|
hasher.update(RAW_TRANSACTION_INGEST_STANDARD_LOGS_FILTER_FINGERPRINT_DOMAIN);
|
||||||
|
match filter {
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All => hash_live_source_key_component(&mut hasher, b"all"),
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::AllWithVotes => hash_live_source_key_component(&mut hasher, b"allWithVotes"),
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::Mentions(pubkey) => {
|
||||||
|
hash_live_source_key_component(&mut hasher, b"mentions");
|
||||||
|
hash_live_source_key_component(&mut hasher, &pubkey.to_bytes());
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return hasher.finalize().into();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn standard_logs_filter_kind(filter: &ksp_onchain_transport_lib::SolanaLogsSubscribeFilter) -> &'static str {
|
||||||
|
return match filter {
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All => "all",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::AllWithVotes => "all_with_votes",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::Mentions(_) => "mentions",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn standard_logs_live_source_key(
|
||||||
|
network: &ksp_store_lib::RawNetworkId,
|
||||||
|
route: &RawTransactionIngestSourceRoute,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
filter_fingerprint: &[u8; 32],
|
||||||
|
) -> [u8; 32] {
|
||||||
|
let mut hasher = sha2::Sha256::new();
|
||||||
|
hasher.update(RAW_TRANSACTION_INGEST_LIVE_SOURCE_KEY_DOMAIN);
|
||||||
|
hash_live_source_key_component(&mut hasher, b"standard_logs");
|
||||||
|
hash_live_source_key_component(&mut hasher, network.as_str().as_bytes());
|
||||||
|
hash_live_source_key_component(&mut hasher, route.provider.as_str().as_bytes());
|
||||||
|
hash_live_source_key_component(&mut hasher, route.endpoint_id.as_str().as_bytes());
|
||||||
|
hash_live_source_key_component(&mut hasher, commitment.as_str().as_bytes());
|
||||||
|
hash_live_source_key_component(&mut hasher, filter_fingerprint);
|
||||||
|
return hasher.finalize().into();
|
||||||
|
}
|
||||||
|
|
||||||
fn yellowstone_live_source_key(
|
fn yellowstone_live_source_key(
|
||||||
network: &ksp_store_lib::RawNetworkId,
|
network: &ksp_store_lib::RawNetworkId,
|
||||||
route: &RawTransactionIngestSourceRoute,
|
route: &RawTransactionIngestSourceRoute,
|
||||||
@@ -675,7 +1056,8 @@ fn compatible_http_route_count(
|
|||||||
hydration_role: &ksp_onchain_transport_lib::HttpRoleName,
|
hydration_role: &ksp_onchain_transport_lib::HttpRoleName,
|
||||||
request_kind: &str,
|
request_kind: &str,
|
||||||
expected_cluster: &str,
|
expected_cluster: &str,
|
||||||
yellowstone_route: &RawTransactionIngestSourceRoute,
|
source_route: &RawTransactionIngestSourceRoute,
|
||||||
|
route_prefix: &str,
|
||||||
) -> ksp_core_lib::Result<usize> {
|
) -> ksp_core_lib::Result<usize> {
|
||||||
let snapshot = pool.snapshot();
|
let snapshot = pool.snapshot();
|
||||||
let mut compatible = 0_usize;
|
let mut compatible = 0_usize;
|
||||||
@@ -694,7 +1076,7 @@ fn compatible_http_route_count(
|
|||||||
if endpoint.cluster() != expected_cluster {
|
if endpoint.cluster() != expected_cluster {
|
||||||
return std::result::Result::Err(crate::runtime_error("runtime_resources.transport_network_mismatch"));
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.transport_network_mismatch"));
|
||||||
}
|
}
|
||||||
let composite = composite_provenance_codes(yellowstone_route, endpoint.provider(), endpoint.name());
|
let composite = composite_provenance_codes(source_route, route_prefix, endpoint.provider(), endpoint.name());
|
||||||
if let std::result::Result::Err(error) = composite {
|
if let std::result::Result::Err(error) = composite {
|
||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
}
|
}
|
||||||
@@ -705,18 +1087,19 @@ fn compatible_http_route_count(
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn composite_provenance_codes(
|
fn composite_provenance_codes(
|
||||||
yellowstone_route: &RawTransactionIngestSourceRoute,
|
source_route: &RawTransactionIngestSourceRoute,
|
||||||
|
route_prefix: &str,
|
||||||
http_provider: &str,
|
http_provider: &str,
|
||||||
http_endpoint: &str,
|
http_endpoint: &str,
|
||||||
) -> ksp_core_lib::Result<(ksp_store_lib::RawProvenanceCode, ksp_store_lib::RawProvenanceCode)> {
|
) -> ksp_core_lib::Result<(ksp_store_lib::RawProvenanceCode, ksp_store_lib::RawProvenanceCode)> {
|
||||||
let provider = std::format!("ys.{}:http.{}", yellowstone_route.provider.as_str(), http_provider);
|
let provider = std::format!("{}.{}:http.{}", route_prefix, source_route.provider.as_str(), http_provider);
|
||||||
let provider = match ksp_store_lib::RawProvenanceCode::new(provider) {
|
let provider = match ksp_store_lib::RawProvenanceCode::new(provider) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => {
|
std::result::Result::Err(_) => {
|
||||||
return std::result::Result::Err(crate::runtime_error("runtime_resources.composite_provider_unrepresentable"));
|
return std::result::Result::Err(crate::runtime_error("runtime_resources.composite_provider_unrepresentable"));
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
let endpoint = std::format!("ys.{}:http.{}", yellowstone_route.endpoint_id.as_str(), http_endpoint);
|
let endpoint = std::format!("{}.{}:http.{}", route_prefix, source_route.endpoint_id.as_str(), http_endpoint);
|
||||||
let endpoint = match ksp_store_lib::RawProvenanceCode::new(endpoint) {
|
let endpoint = match ksp_store_lib::RawProvenanceCode::new(endpoint) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => {
|
std::result::Result::Err(_) => {
|
||||||
@@ -849,8 +1232,48 @@ fn project_yellowstone_signal<T: RawTransactionIngestYellowstoneSignalView>(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
trait RawTransactionIngestStandardLogsView {
|
||||||
|
fn signature(&self) -> &str;
|
||||||
|
|
||||||
|
fn slot(&self) -> u64;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RawTransactionIngestStandardLogsView for ksp_onchain_transport_lib::SolanaRpcResponse<ksp_onchain_transport_lib::SolanaLogsNotification> {
|
||||||
|
fn signature(&self) -> &str {
|
||||||
|
return self.value().signature();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn slot(&self) -> u64 {
|
||||||
|
return self.context().slot();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn project_standard_logs_signal<T: RawTransactionIngestStandardLogsView>(
|
||||||
|
source: &crate::RawTransactionIngestStandardLogsSource,
|
||||||
|
response: &T,
|
||||||
|
) -> ksp_core_lib::Result<RawTransactionIngestSourceSignal> {
|
||||||
|
let signature = ksp_raw_transaction_lib::parse_raw_transaction_signature(response.signature());
|
||||||
|
let signature = match signature {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("source.standard_logs_signature_invalid")),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(RawTransactionIngestSourceSignal {
|
||||||
|
created_at: std::option::Option::None,
|
||||||
|
family: RawTransactionIngestSourceFamily::Logs,
|
||||||
|
matched_filter_count: 1,
|
||||||
|
matched_filter_fingerprint: source.filter_fingerprint,
|
||||||
|
matched_filter_id: std::option::Option::None,
|
||||||
|
network: source.network.clone(),
|
||||||
|
route: source.route.clone(),
|
||||||
|
signature,
|
||||||
|
slot: response.slot(),
|
||||||
|
transaction_index: std::option::Option::None,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
fn route_yellowstone_update(
|
fn route_yellowstone_update(
|
||||||
source: &crate::RawTransactionIngestYellowstoneSource,
|
source: &crate::RawTransactionIngestYellowstoneSource,
|
||||||
|
hydration: &RawTransactionIngestHydrationContext,
|
||||||
coordinator: &mut RawTransactionIngestHydrationCoordinator,
|
coordinator: &mut RawTransactionIngestHydrationCoordinator,
|
||||||
processing_frontier: &mut RawTransactionIngestProcessingFrontierReporter,
|
processing_frontier: &mut RawTransactionIngestProcessingFrontierReporter,
|
||||||
update: ksp_onchain_transport_lib::YellowstoneSubscribeUpdate,
|
update: ksp_onchain_transport_lib::YellowstoneSubscribeUpdate,
|
||||||
@@ -861,14 +1284,14 @@ fn route_yellowstone_update(
|
|||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
return coordinator.queue_signal(source, RawTransactionIngestSourceSignal::from((source, value.as_ref())), received_at, processing_frontier);
|
return coordinator.queue_signal(hydration, RawTransactionIngestSourceSignal::from((source, value.as_ref())), received_at, processing_frontier);
|
||||||
},
|
},
|
||||||
ksp_onchain_transport_lib::YellowstoneSubscribeUpdate::TransactionStatus(value) => {
|
ksp_onchain_transport_lib::YellowstoneSubscribeUpdate::TransactionStatus(value) => {
|
||||||
let received_at = match current_raw_timestamp() {
|
let received_at = match current_raw_timestamp() {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
return coordinator.queue_signal(source, RawTransactionIngestSourceSignal::from((source, &value)), received_at, processing_frontier);
|
return coordinator.queue_signal(hydration, RawTransactionIngestSourceSignal::from((source, &value)), received_at, processing_frontier);
|
||||||
},
|
},
|
||||||
ksp_onchain_transport_lib::YellowstoneSubscribeUpdate::Block(value) => {
|
ksp_onchain_transport_lib::YellowstoneSubscribeUpdate::Block(value) => {
|
||||||
let received_at = match current_raw_timestamp() {
|
let received_at = match current_raw_timestamp() {
|
||||||
@@ -880,7 +1303,7 @@ fn route_yellowstone_update(
|
|||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
for signal in signals {
|
for signal in signals {
|
||||||
if let std::result::Result::Err(error) = coordinator.queue_signal(source, signal, received_at, processing_frontier) {
|
if let std::result::Result::Err(error) = coordinator.queue_signal(hydration, signal, received_at, processing_frontier) {
|
||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -935,7 +1358,7 @@ async fn wait_yellowstone_session_snapshot(
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn source_transport_error(code: ksp_core_lib::ErrorCode) -> ksp_core_lib::Error {
|
fn source_transport_error(code: ksp_core_lib::ErrorCode) -> ksp_core_lib::Error {
|
||||||
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED, "RAW transaction ingest Yellowstone source transport failed")
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED, "RAW transaction ingest source transport failed")
|
||||||
.with_context("transport_domain", code.domain())
|
.with_context("transport_domain", code.domain())
|
||||||
.with_context("transport_code", code.code());
|
.with_context("transport_code", code.code());
|
||||||
}
|
}
|
||||||
@@ -1251,7 +1674,7 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
fn new(settings: &crate::RawTransactionIngestSettings) -> Self {
|
fn new(settings: &crate::RawTransactionIngestSettings) -> Self {
|
||||||
return Self {
|
return Self {
|
||||||
max_in_flight: settings.persistence_concurrency(),
|
max_in_flight: settings.persistence_concurrency(),
|
||||||
max_pending_signals: crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
max_pending_signals: settings.admission_queue_capacity(),
|
||||||
pending_signal_count: 0,
|
pending_signal_count: 0,
|
||||||
pending: std::collections::BTreeMap::new(),
|
pending: std::collections::BTreeMap::new(),
|
||||||
tasks: RawTransactionIngestHydrationTasks::new(),
|
tasks: RawTransactionIngestHydrationTasks::new(),
|
||||||
@@ -1264,7 +1687,7 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
|
|
||||||
fn queue_signal(
|
fn queue_signal(
|
||||||
&mut self,
|
&mut self,
|
||||||
source: &crate::RawTransactionIngestYellowstoneSource,
|
hydration: &RawTransactionIngestHydrationContext,
|
||||||
signal: RawTransactionIngestSourceSignal,
|
signal: RawTransactionIngestSourceSignal,
|
||||||
received_at: ksp_store_lib::RawTimestamp,
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
processing_frontier: &mut RawTransactionIngestProcessingFrontierReporter,
|
processing_frontier: &mut RawTransactionIngestProcessingFrontierReporter,
|
||||||
@@ -1272,7 +1695,7 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
if self.pending_signal_count >= self.max_pending_signals {
|
if self.pending_signal_count >= self.max_pending_signals {
|
||||||
return std::result::Result::Err(crate::runtime_error("source.hydration_pending_saturated"));
|
return std::result::Result::Err(crate::runtime_error("source.hydration_pending_saturated"));
|
||||||
}
|
}
|
||||||
let key = match hydration_key(source, &signal) {
|
let key = match hydration_key(hydration, &signal) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
@@ -1295,10 +1718,10 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
if let std::result::Result::Err(error) = processing_frontier.observe_pending(signal_slot) {
|
if let std::result::Result::Err(error) = processing_frontier.observe_pending(signal_slot) {
|
||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
}
|
}
|
||||||
return self.start_hydrations(source);
|
return self.start_hydrations(hydration);
|
||||||
}
|
}
|
||||||
|
|
||||||
fn start_hydrations(&mut self, source: &crate::RawTransactionIngestYellowstoneSource) -> ksp_core_lib::Result<()> {
|
fn start_hydrations(&mut self, hydration: &RawTransactionIngestHydrationContext) -> ksp_core_lib::Result<()> {
|
||||||
while self.tasks.len() < self.max_in_flight {
|
while self.tasks.len() < self.max_in_flight {
|
||||||
let key = self.pending.iter().find_map(|(key, pending)| {
|
let key = self.pending.iter().find_map(|(key, pending)| {
|
||||||
if pending.in_flight {
|
if pending.in_flight {
|
||||||
@@ -1310,21 +1733,18 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
std::option::Option::Some(value) => value,
|
std::option::Option::Some(value) => value,
|
||||||
std::option::Option::None => break,
|
std::option::Option::None => break,
|
||||||
};
|
};
|
||||||
let commitment = match hydration_commitment(source) {
|
let commitment = hydration.commitment;
|
||||||
std::result::Result::Ok(value) => value,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
let pending = match self.pending.get_mut(&key) {
|
let pending = match self.pending.get_mut(&key) {
|
||||||
std::option::Option::Some(value) => value,
|
std::option::Option::Some(value) => value,
|
||||||
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.hydration_pending_missing")),
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("source.hydration_pending_missing")),
|
||||||
};
|
};
|
||||||
pending.in_flight = true;
|
pending.in_flight = true;
|
||||||
let pool = source.http_pool.clone();
|
let pool = hydration.http_pool.clone();
|
||||||
let role = source.hydration_role.clone();
|
let role = hydration.hydration_role.clone();
|
||||||
let expected_network = source.network.clone();
|
let expected_network = hydration.network.clone();
|
||||||
let task_key = key.clone();
|
let task_key = key.clone();
|
||||||
let _abort_handle = self.tasks.spawn(async move {
|
let _abort_handle = self.tasks.spawn(async move {
|
||||||
return fetch_yellowstone_hydration(pool, role, expected_network, task_key, commitment).await;
|
return fetch_hydration(pool, role, expected_network, task_key, commitment).await;
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
return std::result::Result::Ok(());
|
return std::result::Result::Ok(());
|
||||||
@@ -1333,7 +1753,7 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
async fn handle_joined(
|
async fn handle_joined(
|
||||||
&mut self,
|
&mut self,
|
||||||
joined: std::result::Result<ksp_core_lib::Result<RawTransactionIngestHydrationFetch>, tokio::task::JoinError>,
|
joined: std::result::Result<ksp_core_lib::Result<RawTransactionIngestHydrationFetch>, tokio::task::JoinError>,
|
||||||
source: &crate::RawTransactionIngestYellowstoneSource,
|
hydration: &RawTransactionIngestHydrationContext,
|
||||||
settings: &crate::RawTransactionIngestSettings,
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
admission_sender: &tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
|
admission_sender: &tokio::sync::mpsc::Sender<crate::RawTransactionIngress>,
|
||||||
stop_receiver: &mut tokio::sync::watch::Receiver<bool>,
|
stop_receiver: &mut tokio::sync::watch::Receiver<bool>,
|
||||||
@@ -1354,7 +1774,7 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
self.pending_signal_count -= pending.signals.len();
|
self.pending_signal_count -= pending.signals.len();
|
||||||
for pending_signal in pending.signals {
|
for pending_signal in pending.signals {
|
||||||
let signal_slot = pending_signal.signal.slot;
|
let signal_slot = pending_signal.signal.slot;
|
||||||
let ingress = finalize_yellowstone_hydration(source, settings, pending_signal.signal, pending_signal.received_at, &fetched.observed);
|
let ingress = finalize_hydration(hydration, settings, pending_signal.signal, pending_signal.received_at, &fetched.observed);
|
||||||
let ingress = match ingress {
|
let ingress = match ingress {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
@@ -1385,7 +1805,7 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if let std::result::Result::Err(error) = self.start_hydrations(source) {
|
if let std::result::Result::Err(error) = self.start_hydrations(hydration) {
|
||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
}
|
}
|
||||||
return std::result::Result::Ok(true);
|
return std::result::Result::Ok(true);
|
||||||
@@ -1401,39 +1821,21 @@ impl RawTransactionIngestHydrationCoordinator {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn hydration_commitment(source: &crate::RawTransactionIngestYellowstoneSource) -> ksp_core_lib::Result<ksp_onchain_transport_lib::SolanaCommitment> {
|
|
||||||
return match source.subscribe_request.commitment() {
|
|
||||||
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed) => {
|
|
||||||
std::result::Result::Ok(ksp_onchain_transport_lib::SolanaCommitment::Confirmed)
|
|
||||||
},
|
|
||||||
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Finalized) => {
|
|
||||||
std::result::Result::Ok(ksp_onchain_transport_lib::SolanaCommitment::Finalized)
|
|
||||||
},
|
|
||||||
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Processed) | std::option::Option::None => {
|
|
||||||
std::result::Result::Err(crate::runtime_error("hydration.commitment_invalid"))
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
fn hydration_key(
|
fn hydration_key(
|
||||||
source: &crate::RawTransactionIngestYellowstoneSource,
|
hydration: &RawTransactionIngestHydrationContext,
|
||||||
signal: &RawTransactionIngestSourceSignal,
|
signal: &RawTransactionIngestSourceSignal,
|
||||||
) -> ksp_core_lib::Result<RawTransactionIngestHydrationKey> {
|
) -> ksp_core_lib::Result<RawTransactionIngestHydrationKey> {
|
||||||
if signal.network != source.network {
|
if signal.network != hydration.network {
|
||||||
return std::result::Result::Err(crate::runtime_error("hydration.network_mismatch"));
|
return std::result::Result::Err(crate::runtime_error("hydration.network_mismatch"));
|
||||||
}
|
}
|
||||||
let commitment = match hydration_commitment(source) {
|
|
||||||
std::result::Result::Ok(value) => value,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
return std::result::Result::Ok(RawTransactionIngestHydrationKey {
|
return std::result::Result::Ok(RawTransactionIngestHydrationKey {
|
||||||
commitment: commitment.as_str(),
|
commitment: hydration.commitment.as_str(),
|
||||||
network: signal.network.clone(),
|
network: signal.network.clone(),
|
||||||
signature: signal.signature,
|
signature: signal.signature,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn fetch_yellowstone_hydration(
|
async fn fetch_hydration(
|
||||||
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
expected_network: ksp_store_lib::RawNetworkId,
|
expected_network: ksp_store_lib::RawNetworkId,
|
||||||
@@ -1449,7 +1851,7 @@ async fn fetch_yellowstone_hydration(
|
|||||||
let config = ksp_onchain_transport_lib::SolanaGetTransactionConfig::new(
|
let config = ksp_onchain_transport_lib::SolanaGetTransactionConfig::new(
|
||||||
std::option::Option::Some(commitment),
|
std::option::Option::Some(commitment),
|
||||||
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
|
||||||
std::option::Option::Some(0),
|
std::option::Option::Some(1),
|
||||||
);
|
);
|
||||||
let signature_text = ksp_raw_transaction_lib::format_raw_transaction_signature(&key.signature);
|
let signature_text = ksp_raw_transaction_lib::format_raw_transaction_signature(&key.signature);
|
||||||
let observed = http_pool.get_transaction_observed(&hydration_role, signature_text.as_str(), std::option::Option::Some(&config)).await;
|
let observed = http_pool.get_transaction_observed(&hydration_role, signature_text.as_str(), std::option::Option::Some(&config)).await;
|
||||||
@@ -1460,23 +1862,20 @@ async fn fetch_yellowstone_hydration(
|
|||||||
return std::result::Result::Ok(RawTransactionIngestHydrationFetch { key, observed });
|
return std::result::Result::Ok(RawTransactionIngestHydrationFetch { key, observed });
|
||||||
}
|
}
|
||||||
|
|
||||||
fn finalize_yellowstone_hydration(
|
fn finalize_hydration(
|
||||||
source: &crate::RawTransactionIngestYellowstoneSource,
|
hydration: &RawTransactionIngestHydrationContext,
|
||||||
settings: &crate::RawTransactionIngestSettings,
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
signal: RawTransactionIngestSourceSignal,
|
signal: RawTransactionIngestSourceSignal,
|
||||||
received_at: ksp_store_lib::RawTimestamp,
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
observed: &RawTransactionIngestObservedTransaction,
|
observed: &RawTransactionIngestObservedTransaction,
|
||||||
) -> ksp_core_lib::Result<std::option::Option<crate::RawTransactionIngress>> {
|
) -> ksp_core_lib::Result<std::option::Option<crate::RawTransactionIngress>> {
|
||||||
if &signal.network != settings.network() || signal.network != source.network {
|
if &signal.network != settings.network() || signal.network != hydration.network {
|
||||||
return std::result::Result::Err(crate::runtime_error("hydration.network_mismatch"));
|
return std::result::Result::Err(crate::runtime_error("hydration.network_mismatch"));
|
||||||
}
|
}
|
||||||
if signal.route != source.route {
|
if signal.route != hydration.route {
|
||||||
return std::result::Result::Err(crate::runtime_error("hydration.source_route_mismatch"));
|
return std::result::Result::Err(crate::runtime_error("hydration.source_route_mismatch"));
|
||||||
}
|
}
|
||||||
let commitment = match hydration_commitment(source) {
|
let commitment = hydration.commitment;
|
||||||
std::result::Result::Ok(value) => value,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
let transaction = match observed.value().as_ref() {
|
let transaction = match observed.value().as_ref() {
|
||||||
std::option::Option::Some(value) => value,
|
std::option::Option::Some(value) => value,
|
||||||
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
||||||
@@ -1510,12 +1909,12 @@ fn finalize_yellowstone_hydration(
|
|||||||
if embedded_signature != signal.signature {
|
if embedded_signature != signal.signature {
|
||||||
return std::result::Result::Err(crate::runtime_error("hydration.signature_mismatch"));
|
return std::result::Result::Err(crate::runtime_error("hydration.signature_mismatch"));
|
||||||
}
|
}
|
||||||
let provenance = build_hydration_provenance(settings, &signal, observed.provider().as_str(), observed.endpoint_name(), commitment, received_at);
|
let provenance = build_hydration_provenance(hydration, settings, &signal, observed.provider().as_str(), observed.endpoint_name(), commitment, received_at);
|
||||||
let provenance = match provenance {
|
let provenance = match provenance {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
let source_key = hydration_source_key(&provenance);
|
let source_key = hydration_source_key(hydration.source_key_domain, &provenance);
|
||||||
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
signal.network.clone(),
|
signal.network.clone(),
|
||||||
signal.signature,
|
signal.signature,
|
||||||
@@ -1548,20 +1947,18 @@ async fn hydrate_yellowstone_signal(
|
|||||||
received_at: ksp_store_lib::RawTimestamp,
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
) -> ksp_core_lib::Result<RawTransactionIngestHydrationOutcome> {
|
) -> ksp_core_lib::Result<RawTransactionIngestHydrationOutcome> {
|
||||||
let reference = ksp_store_lib::RawTransactionReference::new(signal.network.clone(), signal.signature);
|
let reference = ksp_store_lib::RawTransactionReference::new(signal.network.clone(), signal.signature);
|
||||||
let key = match hydration_key(source, &signal) {
|
let hydration = source.hydration_context();
|
||||||
|
let key = match hydration_key(&hydration, &signal) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
let commitment = match hydration_commitment(source) {
|
let commitment = hydration.commitment;
|
||||||
std::result::Result::Ok(value) => value,
|
let fetched = fetch_hydration(hydration.http_pool.clone(), hydration.hydration_role.clone(), hydration.network.clone(), key, commitment).await;
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
let fetched = fetch_yellowstone_hydration(source.http_pool.clone(), source.hydration_role.clone(), source.network.clone(), key, commitment).await;
|
|
||||||
let fetched = match fetched {
|
let fetched = match fetched {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
let ingress = finalize_yellowstone_hydration(source, settings, signal, received_at, &fetched.observed);
|
let ingress = finalize_hydration(&hydration, settings, signal, received_at, &fetched.observed);
|
||||||
return match ingress {
|
return match ingress {
|
||||||
std::result::Result::Ok(std::option::Option::Some(value)) => {
|
std::result::Result::Ok(std::option::Option::Some(value)) => {
|
||||||
std::result::Result::Ok(RawTransactionIngestHydrationOutcome::Available(std::boxed::Box::new(value)))
|
std::result::Result::Ok(RawTransactionIngestHydrationOutcome::Available(std::boxed::Box::new(value)))
|
||||||
@@ -1572,6 +1969,7 @@ async fn hydrate_yellowstone_signal(
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn build_hydration_provenance(
|
fn build_hydration_provenance(
|
||||||
|
hydration: &RawTransactionIngestHydrationContext,
|
||||||
settings: &crate::RawTransactionIngestSettings,
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
signal: &RawTransactionIngestSourceSignal,
|
signal: &RawTransactionIngestSourceSignal,
|
||||||
http_provider: &str,
|
http_provider: &str,
|
||||||
@@ -1579,11 +1977,11 @@ fn build_hydration_provenance(
|
|||||||
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
received_at: ksp_store_lib::RawTimestamp,
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
) -> ksp_core_lib::Result<ksp_store_lib::RawAcquisitionProvenance> {
|
) -> ksp_core_lib::Result<ksp_store_lib::RawAcquisitionProvenance> {
|
||||||
let (provider, endpoint_id) = match composite_provenance_codes(&signal.route, http_provider, http_endpoint) {
|
let (provider, endpoint_id) = match composite_provenance_codes(&signal.route, hydration.route_prefix, http_provider, http_endpoint) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
let protocol = match ksp_store_lib::RawProvenanceCode::new(RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_PROTOCOL) {
|
let protocol = match ksp_store_lib::RawProvenanceCode::new(hydration.protocol) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("hydration.protocol_unrepresentable")),
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("hydration.protocol_unrepresentable")),
|
||||||
};
|
};
|
||||||
@@ -1624,6 +2022,7 @@ fn build_hydration_provenance(
|
|||||||
fn hydration_method_code(family: RawTransactionIngestSourceFamily) -> &'static str {
|
fn hydration_method_code(family: RawTransactionIngestSourceFamily) -> &'static str {
|
||||||
return match family {
|
return match family {
|
||||||
RawTransactionIngestSourceFamily::Block => "block_get_transaction",
|
RawTransactionIngestSourceFamily::Block => "block_get_transaction",
|
||||||
|
RawTransactionIngestSourceFamily::Logs => "logs_get_transaction",
|
||||||
RawTransactionIngestSourceFamily::Transaction => "transaction_get_transaction",
|
RawTransactionIngestSourceFamily::Transaction => "transaction_get_transaction",
|
||||||
RawTransactionIngestSourceFamily::TransactionStatus => "status_get_transaction",
|
RawTransactionIngestSourceFamily::TransactionStatus => "status_get_transaction",
|
||||||
};
|
};
|
||||||
@@ -1669,9 +2068,9 @@ fn representable_observed_at(
|
|||||||
return std::option::Option::Some(observed_at);
|
return std::option::Option::Some(observed_at);
|
||||||
}
|
}
|
||||||
|
|
||||||
fn hydration_source_key(provenance: &ksp_store_lib::RawAcquisitionProvenance) -> [u8; 32] {
|
fn hydration_source_key(source_key_domain: &[u8], provenance: &ksp_store_lib::RawAcquisitionProvenance) -> [u8; 32] {
|
||||||
let mut hasher = sha2::Sha256::new();
|
let mut hasher = sha2::Sha256::new();
|
||||||
hasher.update(RAW_TRANSACTION_INGEST_YELLOWSTONE_HTTP_SOURCE_KEY_DOMAIN);
|
hasher.update(source_key_domain);
|
||||||
hash_hydration_source_key(&mut hasher, provenance.provider().as_str().as_bytes());
|
hash_hydration_source_key(&mut hasher, provenance.provider().as_str().as_bytes());
|
||||||
hash_hydration_source_key(&mut hasher, provenance.protocol().as_str().as_bytes());
|
hash_hydration_source_key(&mut hasher, provenance.protocol().as_str().as_bytes());
|
||||||
hash_hydration_source_key(&mut hasher, provenance.acquisition_method().as_str().as_bytes());
|
hash_hydration_source_key(&mut hasher, provenance.acquisition_method().as_str().as_bytes());
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
// version: 16
|
// version: 17
|
||||||
|
|
||||||
//! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
|
//! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
@@ -176,9 +176,9 @@ fn v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_
|
|||||||
let runtime = include_str!("../src/runtime.rs");
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
let resources = include_str!("../src/runtime_resources.rs");
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
for required in [
|
for required in [
|
||||||
"runtime_resources.into_yellowstone_source()",
|
"runtime_resources.validate_single_source_activation()",
|
||||||
"children.spawn",
|
"children.spawn",
|
||||||
"source.run(source_settings, stop_receiver, admission_sender, processing_frontier_sender)",
|
".run_single_live_source(source_settings, stop_receiver, admission_sender, processing_frontier_sender)",
|
||||||
] {
|
] {
|
||||||
assert!(runtime.contains(required), "required pre.006 supervisor wiring missing: {required}");
|
assert!(runtime.contains(required), "required pre.006 supervisor wiring missing: {required}");
|
||||||
}
|
}
|
||||||
@@ -214,6 +214,47 @@ fn v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
"SolanaStandardWsSession::connect",
|
||||||
|
".logs_subscribe(",
|
||||||
|
"SolanaLogsSubscribeFilter",
|
||||||
|
"SolanaCommitmentConfig::new",
|
||||||
|
"project_standard_logs_signal",
|
||||||
|
"RawTransactionIngestSourceFamily::Logs",
|
||||||
|
"logs_get_transaction",
|
||||||
|
"solana_ws_http",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"max_pending_signals: settings.admission_queue_capacity()",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 standard logs contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("run_single_live_source"));
|
||||||
|
for forbidden in [
|
||||||
|
"get_block_observed",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!runtime.contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
|
||||||
|
"pre.003 crossed a forbidden boundary: {forbidden}",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn v0_3_12_pre_006_source_surface_hardens_shutdown_and_faults_without_backend_edges() {
|
fn v0_3_12_pre_006_source_surface_hardens_shutdown_and_faults_without_backend_edges() {
|
||||||
let root = include_str!("../src/lib.rs");
|
let root = include_str!("../src/lib.rs");
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
// version: 11
|
// version: 12
|
||||||
|
|
||||||
//! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
|
//! External public, security, redaction and release-boundary hardening canaries for `pre.010`.
|
||||||
|
|
||||||
@@ -159,7 +159,10 @@ fn pre_010_source_visibility_contract_uses_crate_root_for_shared_items() {
|
|||||||
(include_str!("../src/runtime.rs"), &["RawTransactionIngestHandle", "RawTransactionIngestWorker"]),
|
(include_str!("../src/runtime.rs"), &["RawTransactionIngestHandle", "RawTransactionIngestWorker"]),
|
||||||
(include_str!("../src/snapshot.rs"), &["RawTransactionIngestSnapshot", "RawTransactionIngestSnapshotSource"]),
|
(include_str!("../src/snapshot.rs"), &["RawTransactionIngestSnapshot", "RawTransactionIngestSnapshotSource"]),
|
||||||
(include_str!("../src/persistence.rs"), &["RawTransactionIngestPersistenceOutcome", "RawTransactionIngestPersistencePort"]),
|
(include_str!("../src/persistence.rs"), &["RawTransactionIngestPersistenceOutcome", "RawTransactionIngestPersistencePort"]),
|
||||||
(include_str!("../src/runtime_resources.rs"), &["RawTransactionIngestYellowstoneSource", "RawTransactionIngestRuntimeResources"]),
|
(
|
||||||
|
include_str!("../src/runtime_resources.rs"),
|
||||||
|
&["RawTransactionIngestStandardLogsSource", "RawTransactionIngestYellowstoneSource", "RawTransactionIngestRuntimeResources"],
|
||||||
|
),
|
||||||
];
|
];
|
||||||
for (source, symbols) in source_contracts {
|
for (source, symbols) in source_contracts {
|
||||||
for symbol in symbols {
|
for symbol in symbols {
|
||||||
@@ -317,7 +320,8 @@ fn v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filter
|
|||||||
fn v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted() {
|
fn v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted() {
|
||||||
let resources = include_str!("../src/runtime_resources.rs");
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
for required in [
|
for required in [
|
||||||
"ys.{}:http.{}",
|
"route_prefix",
|
||||||
|
"{}.{}:http.{}",
|
||||||
"composite_provider_unrepresentable",
|
"composite_provider_unrepresentable",
|
||||||
"composite_endpoint_unrepresentable",
|
"composite_endpoint_unrepresentable",
|
||||||
"RawAcquisitionOrigin::Live",
|
"RawAcquisitionOrigin::Live",
|
||||||
@@ -390,7 +394,7 @@ fn v0_3_12_pre_006_runtime_resource_contract_opens_one_supervised_transport_sour
|
|||||||
assert!(resources.contains(required), "productive runtime-resource source behavior missing: {required}");
|
assert!(resources.contains(required), "productive runtime-resource source behavior missing: {required}");
|
||||||
}
|
}
|
||||||
assert!(runtime.contains("start_with_runtime_resources"));
|
assert!(runtime.contains("start_with_runtime_resources"));
|
||||||
assert!(runtime.contains("source.run(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"));
|
assert!(runtime.contains("run_single_live_source(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"));
|
||||||
for forbidden in ["get_block_observed", "ksp_config_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
for forbidden in ["get_block_observed", "ksp_config_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
assert!(!resources.contains(forbidden) && !runtime.contains(forbidden), "pre.006 runtime source crossed a forbidden boundary: {forbidden}");
|
assert!(!resources.contains(forbidden) && !runtime.contains(forbidden), "pre.006 runtime source crossed a forbidden boundary: {forbidden}");
|
||||||
}
|
}
|
||||||
@@ -402,7 +406,7 @@ fn v0_3_12_pre_006_source_coalescence_is_bounded_stop_preemptible_and_redacted()
|
|||||||
let resources = include_str!("../src/runtime_resources.rs");
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
for required in [
|
for required in [
|
||||||
"max_in_flight: settings.persistence_concurrency()",
|
"max_in_flight: settings.persistence_concurrency()",
|
||||||
"max_pending_signals: crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
"max_pending_signals: settings.admission_queue_capacity()",
|
||||||
"pending_signal_count",
|
"pending_signal_count",
|
||||||
"stop_receiver.changed()",
|
"stop_receiver.changed()",
|
||||||
"tasks.abort_all()",
|
"tasks.abort_all()",
|
||||||
@@ -523,6 +527,45 @@ fn v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_n
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
".field(\"filter_kind\"",
|
||||||
|
".field(\"filter_fingerprint_bytes\"",
|
||||||
|
".field(\"source_key_bytes\"",
|
||||||
|
"project_standard_logs_signal",
|
||||||
|
"matched_filter_count: 1",
|
||||||
|
"transaction_index: std::option::Option::None",
|
||||||
|
"created_at: std::option::Option::None",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 redaction/reference guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestStandardLogsSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestStandardLogsSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(source_struct.contains("filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter"));
|
||||||
|
for forbidden in ["logs:", "err:", "payload:", "body:", "url:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "remote/sensitive material stored in standard logs source: {forbidden}");
|
||||||
|
}
|
||||||
|
let projection = match resources.split_once("fn project_standard_logs_signal") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("fn route_yellowstone_update") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in [".logs()", ".err()"] {
|
||||||
|
assert!(!projection.contains(forbidden), "logs/error material copied into standard logs projection: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit() {
|
fn v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit() {
|
||||||
let resources = include_str!("../src/runtime_resources.rs");
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
// version: 12
|
// version: 13
|
||||||
|
|
||||||
//! External public-surface proofs for the RAW transaction ingest Worker foundation.
|
//! External public-surface proofs for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
@@ -197,6 +197,33 @@ fn v0_3_13_pre_002_multi_source_runtime_resource_surface_is_bounded_and_source_n
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource::new;
|
||||||
|
let _resources_from_logs: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_logs_source;
|
||||||
|
let _push_logs: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_standard_logs_source;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(root.contains("RawTransactionIngestStandardLogsSource"));
|
||||||
|
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn http_pool(", "pub fn source_key("] {
|
||||||
|
assert!(!resources.contains(forbidden), "standard logs source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only() {
|
fn v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only() {
|
||||||
let _hydration_pending: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
let _hydration_pending: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
// version: 10
|
// version: 11
|
||||||
|
|
||||||
//! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
|
//! Release-completeness canaries through the `pre.010` public/release/security hardening tranche.
|
||||||
|
|
||||||
@@ -84,6 +84,7 @@ fn pre_010_public_root_export_inventory_is_exact() {
|
|||||||
"RawTransactionIngestSnapshotFuture",
|
"RawTransactionIngestSnapshotFuture",
|
||||||
"RawTransactionIngestSnapshotSource",
|
"RawTransactionIngestSnapshotSource",
|
||||||
"RawTransactionIngestSourceState",
|
"RawTransactionIngestSourceState",
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
"RawTransactionIngestTerminalFuture",
|
"RawTransactionIngestTerminalFuture",
|
||||||
"RawTransactionIngestWorker",
|
"RawTransactionIngestWorker",
|
||||||
"RawTransactionIngestYellowstoneSource",
|
"RawTransactionIngestYellowstoneSource",
|
||||||
@@ -113,6 +114,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
|||||||
"v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit",
|
"v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit",
|
||||||
"pre_010_lower_layers_have_no_dependency_return_to_concrete_worker",
|
"pre_010_lower_layers_have_no_dependency_return_to_concrete_worker",
|
||||||
"v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths",
|
"v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths",
|
||||||
|
"v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit",
|
||||||
] {
|
] {
|
||||||
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
|
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
|
||||||
}
|
}
|
||||||
@@ -126,6 +128,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
|||||||
assert!(dependency_boundary.contains("v0_3_12_pre_005_block_and_continuity_adapters_remain_private_and_transport_facade_only"));
|
assert!(dependency_boundary.contains("v0_3_12_pre_005_block_and_continuity_adapters_remain_private_and_transport_facade_only"));
|
||||||
assert!(dependency_boundary.contains("v0_3_12_pre_007_processing_frontier_remains_run_local_and_backend_neutral_after_continuity_extension"));
|
assert!(dependency_boundary.contains("v0_3_12_pre_007_processing_frontier_remains_run_local_and_backend_neutral_after_continuity_extension"));
|
||||||
assert!(dependency_boundary.contains("v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap"));
|
assert!(dependency_boundary.contains("v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only"));
|
||||||
let public_api = include_str!("public_api.rs");
|
let public_api = include_str!("public_api.rs");
|
||||||
assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
|
assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
|
||||||
assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
|
assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
|
||||||
@@ -133,6 +136,7 @@ fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
|||||||
assert!(public_api.contains("pre_009_source_and_drain_timeout_error_codes_are_public_and_stable"));
|
assert!(public_api.contains("pre_009_source_and_drain_timeout_error_codes_are_public_and_stable"));
|
||||||
assert!(public_api.contains("v0_3_12_pre_002_runtime_resource_types_are_consumable_without_client_escape_hatch"));
|
assert!(public_api.contains("v0_3_12_pre_002_runtime_resource_types_are_consumable_without_client_escape_hatch"));
|
||||||
assert!(public_api.contains("v0_3_13_pre_002_multi_source_runtime_resource_surface_is_bounded_and_source_neutral"));
|
assert!(public_api.contains("v0_3_13_pre_002_multi_source_runtime_resource_surface_is_bounded_and_source_neutral"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport_owned"));
|
||||||
assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
|
assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
|
||||||
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
// version: 12
|
// version: 13
|
||||||
|
|
||||||
fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
|
fn grpc_endpoint(cluster: &str) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
|
||||||
return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
|
return grpc_endpoint_with_identity(cluster, "yellowstone-fixture", "fixture-provider");
|
||||||
@@ -62,6 +62,69 @@ fn http_pool(cluster: &str, role_name: &str, request_kind: &str) -> std::option:
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn ws_endpoint(
|
||||||
|
cluster: &str,
|
||||||
|
endpoint_name: &str,
|
||||||
|
provider: &str,
|
||||||
|
protocol: ksp_onchain_transport_lib::WsProtocolKind,
|
||||||
|
) -> std::option::Option<ksp_onchain_transport_lib::WsEndpointSettings> {
|
||||||
|
let url = match ksp_onchain_transport_lib::WsEndpointUrl::parse("ws://127.0.0.1:10001/WS-SECRET-CANARY") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_onchain_transport_lib::WsEndpointSettings::new(
|
||||||
|
endpoint_name,
|
||||||
|
true,
|
||||||
|
ksp_onchain_transport_lib::WsProviderName::new(provider),
|
||||||
|
ksp_onchain_transport_lib::WsClusterName::new(cluster),
|
||||||
|
protocol,
|
||||||
|
url,
|
||||||
|
ksp_onchain_transport_lib::WsSessionSettings::default(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn standard_logs_source(
|
||||||
|
cluster: &str,
|
||||||
|
endpoint_name: &str,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
hydration_role: &str,
|
||||||
|
) -> std::option::Option<crate::RawTransactionIngestStandardLogsSource> {
|
||||||
|
let endpoint = match ws_endpoint(cluster, endpoint_name, "fixture-ws-provider", ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let pool = match http_pool(cluster, hydration_role, "get_transaction") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return match crate::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
endpoint,
|
||||||
|
filter,
|
||||||
|
commitment,
|
||||||
|
pool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new(hydration_role),
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
struct StandardLogsViewFixture {
|
||||||
|
signature: std::string::String,
|
||||||
|
slot: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl super::RawTransactionIngestStandardLogsView for StandardLogsViewFixture {
|
||||||
|
fn signature(&self) -> &str {
|
||||||
|
return self.signature.as_str();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn slot(&self) -> u64 {
|
||||||
|
return self.slot;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn transaction_request(
|
fn transaction_request(
|
||||||
commitment: std::option::Option<ksp_onchain_transport_lib::SolanaCommitment>,
|
commitment: std::option::Option<ksp_onchain_transport_lib::SolanaCommitment>,
|
||||||
) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneSubscribeRequest> {
|
) -> std::option::Option<ksp_onchain_transport_lib::YellowstoneSubscribeRequest> {
|
||||||
@@ -331,6 +394,30 @@ fn signal_source_for_http_url_with_commitment(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn standard_logs_source_for_http_url(
|
||||||
|
url: &str,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
) -> std::option::Option<crate::RawTransactionIngestStandardLogsSource> {
|
||||||
|
let endpoint = match ws_endpoint("devnet", "standard-logs-fixture", "fixture-ws-provider", ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let pool = match http_pool_for_url(url, "devnet", "http-hydration-fixture", "fixture-http-provider") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return match crate::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
endpoint,
|
||||||
|
filter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
pool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
fn pre_004_settings() -> std::option::Option<crate::RawTransactionIngestSettings> {
|
fn pre_004_settings() -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
|
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
@@ -1049,7 +1136,7 @@ fn pre_004_composite_provenance_rejects_overflow_without_truncation() {
|
|||||||
std::result::Result::Err(_) => return,
|
std::result::Result::Err(_) => return,
|
||||||
};
|
};
|
||||||
let route = super::RawTransactionIngestSourceRoute { endpoint_id, provider };
|
let route = super::RawTransactionIngestSourceRoute { endpoint_id, provider };
|
||||||
let result = super::composite_provenance_codes(&route, "h".repeat(120).as_str(), "http-endpoint");
|
let result = super::composite_provenance_codes(&route, "ys", "h".repeat(120).as_str(), "http-endpoint");
|
||||||
let error = match result {
|
let error = match result {
|
||||||
std::result::Result::Ok(_) => return,
|
std::result::Result::Ok(_) => return,
|
||||||
std::result::Result::Err(error) => error,
|
std::result::Result::Err(error) => error,
|
||||||
@@ -1100,7 +1187,7 @@ async fn pre_004_observed_get_transaction_closes_signal_to_common_raw_ingress_wi
|
|||||||
assert!(request_body.contains(PRE_004_ZERO_SIGNATURE_TEXT));
|
assert!(request_body.contains(PRE_004_ZERO_SIGNATURE_TEXT));
|
||||||
assert!(request_body.contains("\"commitment\":\"confirmed\""));
|
assert!(request_body.contains("\"commitment\":\"confirmed\""));
|
||||||
assert!(request_body.contains("\"encoding\":\"base64\""));
|
assert!(request_body.contains("\"encoding\":\"base64\""));
|
||||||
assert!(request_body.contains("\"maxSupportedTransactionVersion\":0"));
|
assert!(request_body.contains("\"maxSupportedTransactionVersion\":1"));
|
||||||
let ingress = match outcome {
|
let ingress = match outcome {
|
||||||
super::RawTransactionIngestHydrationOutcome::Available(value) => value,
|
super::RawTransactionIngestHydrationOutcome::Available(value) => value,
|
||||||
super::RawTransactionIngestHydrationOutcome::Missing(_) => return,
|
super::RawTransactionIngestHydrationOutcome::Missing(_) => return,
|
||||||
@@ -1172,7 +1259,7 @@ async fn pre_004_finalized_hydration_preserves_request_and_provenance_commitment
|
|||||||
let request_body = http_request_body(request.as_str());
|
let request_body = http_request_body(request.as_str());
|
||||||
assert!(request_body.contains("\"commitment\":\"finalized\""));
|
assert!(request_body.contains("\"commitment\":\"finalized\""));
|
||||||
assert!(request_body.contains("\"encoding\":\"base64\""));
|
assert!(request_body.contains("\"encoding\":\"base64\""));
|
||||||
assert!(request_body.contains("\"maxSupportedTransactionVersion\":0"));
|
assert!(request_body.contains("\"maxSupportedTransactionVersion\":1"));
|
||||||
let ingress = match outcome {
|
let ingress = match outcome {
|
||||||
super::RawTransactionIngestHydrationOutcome::Available(value) => value,
|
super::RawTransactionIngestHydrationOutcome::Available(value) => value,
|
||||||
super::RawTransactionIngestHydrationOutcome::Missing(_) => return,
|
super::RawTransactionIngestHydrationOutcome::Missing(_) => return,
|
||||||
@@ -1421,11 +1508,12 @@ async fn pre_006_coalescence_key_merges_transaction_and_status_before_http_fanou
|
|||||||
std::option::Option::Some(value) => value,
|
std::option::Option::Some(value) => value,
|
||||||
std::option::Option::None => return,
|
std::option::Option::None => return,
|
||||||
};
|
};
|
||||||
let transaction_key = match super::hydration_key(&source, &transaction) {
|
let hydration = source.hydration_context();
|
||||||
|
let transaction_key = match super::hydration_key(&hydration, &transaction) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => return,
|
std::result::Result::Err(_) => return,
|
||||||
};
|
};
|
||||||
let status_key = match super::hydration_key(&source, &status) {
|
let status_key = match super::hydration_key(&hydration, &status) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => return,
|
std::result::Result::Err(_) => return,
|
||||||
};
|
};
|
||||||
@@ -1437,10 +1525,10 @@ async fn pre_006_coalescence_key_merges_transaction_and_status_before_http_fanou
|
|||||||
let mut coordinator = super::RawTransactionIngestHydrationCoordinator::new(&settings);
|
let mut coordinator = super::RawTransactionIngestHydrationCoordinator::new(&settings);
|
||||||
let (frontier_sender, _frontier_receiver) = tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
let (frontier_sender, _frontier_receiver) = tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||||
let mut processing_frontier = super::RawTransactionIngestProcessingFrontierReporter::new(frontier_sender);
|
let mut processing_frontier = super::RawTransactionIngestProcessingFrontierReporter::new(frontier_sender);
|
||||||
if coordinator.queue_signal(&source, transaction, received_at, &mut processing_frontier).is_err() {
|
if coordinator.queue_signal(&hydration, transaction, received_at, &mut processing_frontier).is_err() {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if coordinator.queue_signal(&source, status, received_at, &mut processing_frontier).is_err() {
|
if coordinator.queue_signal(&hydration, status, received_at, &mut processing_frontier).is_err() {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
assert_eq!(coordinator.pending.len(), 1);
|
assert_eq!(coordinator.pending.len(), 1);
|
||||||
@@ -1467,11 +1555,12 @@ fn pre_006_coalescence_key_separates_network_signature_and_commitment() {
|
|||||||
std::option::Option::Some(value) => value,
|
std::option::Option::Some(value) => value,
|
||||||
std::option::Option::None => return,
|
std::option::Option::None => return,
|
||||||
};
|
};
|
||||||
let first_key = match super::hydration_key(&source, &first) {
|
let hydration = source.hydration_context();
|
||||||
|
let first_key = match super::hydration_key(&hydration, &first) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => return,
|
std::result::Result::Err(_) => return,
|
||||||
};
|
};
|
||||||
let second_key = match super::hydration_key(&source, &second) {
|
let second_key = match super::hydration_key(&hydration, &second) {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => return,
|
std::result::Result::Err(_) => return,
|
||||||
};
|
};
|
||||||
@@ -1481,6 +1570,273 @@ fn pre_006_coalescence_key_separates_network_signature_and_commitment() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_source_validates_protocol_commitment_and_hydration_route() {
|
||||||
|
let accepted = standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-a",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
"hydration",
|
||||||
|
);
|
||||||
|
let accepted = match accepted {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let resources = crate::RawTransactionIngestRuntimeResources::from_standard_logs_source(accepted);
|
||||||
|
assert_eq!(resources.source_count(), 1);
|
||||||
|
let helius_endpoint =
|
||||||
|
match ws_endpoint("devnet", "standard-logs-helius", "fixture-ws-provider", ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let pool = match http_pool("devnet", "hydration", "get_transaction") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let protocol_error = crate::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
helius_endpoint,
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
pool.clone(),
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
|
||||||
|
);
|
||||||
|
let protocol_error = match protocol_error {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert!(protocol_error.context().iter().any(|context| return context.value() == "runtime_resources.standard_logs_protocol_invalid"));
|
||||||
|
let standard_endpoint =
|
||||||
|
match ws_endpoint("devnet", "standard-logs-processed", "fixture-ws-provider", ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let commitment_error = crate::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
standard_endpoint,
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::AllWithVotes,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Processed,
|
||||||
|
pool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
|
||||||
|
);
|
||||||
|
let commitment_error = match commitment_error {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert!(commitment_error.context().iter().any(|context| return context.value() == "runtime_resources.hydration_commitment_invalid"));
|
||||||
|
let cross_network_endpoint =
|
||||||
|
match ws_endpoint("devnet", "standard-logs-cross-network", "fixture-ws-provider", ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let cross_network_pool = match http_pool("testnet", "hydration", "get_transaction") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let cross_network = crate::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
cross_network_endpoint,
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
cross_network_pool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("hydration"),
|
||||||
|
);
|
||||||
|
let cross_network = match cross_network {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert!(cross_network.context().iter().any(|context| return context.value() == "runtime_resources.transport_network_mismatch"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_source_key_is_filter_sensitive_hydration_role_neutral_and_redacted() {
|
||||||
|
let all_a = match standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-key",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
"hydration",
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let all_b = match standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-key",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
"secondary_hydration",
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let votes = match standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-key",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::AllWithVotes,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
"hydration",
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(all_a.source_key, all_b.source_key);
|
||||||
|
assert_ne!(all_a.source_key, votes.source_key);
|
||||||
|
let mention = ksp_core_lib::Pubkey::new_from_array([0x5a_u8; 32]);
|
||||||
|
let mention_text = mention.to_string();
|
||||||
|
let mentioned = match standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-key",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::Mentions(mention),
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
"hydration",
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_ne!(all_a.source_key, mentioned.source_key);
|
||||||
|
let debug = std::format!("{mentioned:?}");
|
||||||
|
assert!(debug.contains("filter_kind"));
|
||||||
|
assert!(debug.contains("mentions"));
|
||||||
|
assert!(!debug.contains(mention_text.as_str()));
|
||||||
|
assert!(!debug.contains("WS-SECRET-CANARY"));
|
||||||
|
assert!(!debug.contains("source_key: ["));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_projection_keeps_only_reference_material() {
|
||||||
|
let source = match standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-projection",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Finalized,
|
||||||
|
"hydration",
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let fixture = StandardLogsViewFixture { signature: PRE_004_ZERO_SIGNATURE_TEXT.to_owned(), slot: 73 };
|
||||||
|
let signal = match super::project_standard_logs_signal(&source, &fixture) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(signal.family, super::RawTransactionIngestSourceFamily::Logs);
|
||||||
|
assert_eq!(signal.slot, 73);
|
||||||
|
assert_eq!(signal.transaction_index, std::option::Option::None);
|
||||||
|
assert_eq!(signal.created_at, std::option::Option::None);
|
||||||
|
assert_eq!(signal.matched_filter_count, 1);
|
||||||
|
assert_eq!(signal.matched_filter_id, std::option::Option::None);
|
||||||
|
assert_eq!(signal.signature, ksp_store_lib::RawTransactionSignature::new([0_u8; 64]));
|
||||||
|
let debug = std::format!("{signal:?}");
|
||||||
|
assert!(!debug.contains(PRE_004_ZERO_SIGNATURE_TEXT));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_003_standard_logs_reference_hydrates_once_to_exact_common_raw() {
|
||||||
|
let body = std::format!(
|
||||||
|
"{{\"jsonrpc\":\"2.0\",\"result\":{{\"slot\":42,\"blockTime\":1760000120,\"transaction\":[\"{}\",\"base64\"],\"meta\":{{\"err\":null,\"fee\":5000}},\"version\":\"legacy\",\"transactionIndex\":7}},\"id\":1}}",
|
||||||
|
PRE_004_ZERO_TRANSACTION_BASE64,
|
||||||
|
);
|
||||||
|
let (url, server) = match serve_http_once(body) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = match standard_logs_source_for_http_url(url.as_str(), ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::All) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = match pre_004_settings() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let fixture = StandardLogsViewFixture { signature: PRE_004_ZERO_SIGNATURE_TEXT.to_owned(), slot: 42 };
|
||||||
|
let signal = match super::project_standard_logs_signal(&source, &fixture) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_760_000_200_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let hydration = source.hydration_context();
|
||||||
|
let key = match super::hydration_key(&hydration, &signal) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let fetched =
|
||||||
|
super::fetch_hydration(hydration.http_pool.clone(), hydration.hydration_role.clone(), hydration.network.clone(), key, hydration.commitment).await;
|
||||||
|
let fetched = match fetched {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let ingress = super::finalize_hydration(&hydration, &settings, signal, received_at, &fetched.observed);
|
||||||
|
let ingress = match ingress {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
std::result::Result::Ok(std::option::Option::None) | std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(ingress.provenance.protocol().as_str(), "solana_ws_http");
|
||||||
|
assert_eq!(ingress.provenance.acquisition_method().as_str(), "logs_get_transaction");
|
||||||
|
assert_eq!(ingress.material.slot(), 42);
|
||||||
|
assert_eq!(ingress.material.signature(), &ksp_store_lib::RawTransactionSignature::new([0_u8; 64]));
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
if sender.send(ingress).await.is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
std::mem::drop(sender);
|
||||||
|
let acquisition = match admission.receive(settings.network()).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(acquisition.transaction().reference().network().as_str(), "devnet");
|
||||||
|
assert_eq!(acquisition.transaction().reference().signature().as_bytes(), &[0_u8; 64]);
|
||||||
|
assert_eq!(acquisition.transaction().slot(), 42);
|
||||||
|
assert_eq!(acquisition.transaction().block_time().map(|value| return value.unix_millis()), std::option::Option::Some(1_760_000_120_000));
|
||||||
|
assert_eq!(acquisition.observation().provenance().protocol().as_str(), "solana_ws_http");
|
||||||
|
assert_eq!(acquisition.observation().provenance().acquisition_method().as_str(), "logs_get_transaction");
|
||||||
|
let request_body = match server.join() {
|
||||||
|
std::result::Result::Ok(std::result::Result::Ok(value)) => value,
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(request_body.contains("\"method\":\"getTransaction\""));
|
||||||
|
assert!(request_body.contains("\"encoding\":\"base64\""));
|
||||||
|
assert!(request_body.contains("\"maxSupportedTransactionVersion\":1"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_runtime_resources_reject_duplicate_standard_logs_identity() {
|
||||||
|
let first = match standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-duplicate",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::AllWithVotes,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
"hydration",
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let duplicate = match standard_logs_source(
|
||||||
|
"devnet",
|
||||||
|
"standard-logs-duplicate",
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter::AllWithVotes,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
"secondary_hydration",
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let mut resources = crate::RawTransactionIngestRuntimeResources::from_standard_logs_source(first);
|
||||||
|
let result = resources.try_push_standard_logs_source(duplicate);
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(()) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert!(error.context().iter().any(|context| return context.value() == "runtime_resources.duplicate_source_identity"));
|
||||||
|
assert_eq!(resources.source_count(), 1);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pre_007_processing_frontier_never_advances_through_pending_work() {
|
fn pre_007_processing_frontier_never_advances_through_pending_work() {
|
||||||
let mut frontier = super::RawTransactionIngestProcessingFrontier::new();
|
let mut frontier = super::RawTransactionIngestProcessingFrontier::new();
|
||||||
@@ -1593,12 +1949,13 @@ async fn pre_009_duplicate_storm_is_bounded_coalesced_and_abort_leaves_no_orphan
|
|||||||
coordinator.max_pending_signals = 3;
|
coordinator.max_pending_signals = 3;
|
||||||
let (frontier_sender, frontier_receiver) = tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
let (frontier_sender, frontier_receiver) = tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||||
let mut processing_frontier = super::RawTransactionIngestProcessingFrontierReporter::new(frontier_sender);
|
let mut processing_frontier = super::RawTransactionIngestProcessingFrontierReporter::new(frontier_sender);
|
||||||
|
let hydration = source.hydration_context();
|
||||||
assert!(processing_frontier.observe_settled(9).is_ok());
|
assert!(processing_frontier.observe_settled(9).is_ok());
|
||||||
assert!(processing_frontier.observe_settled(10).is_ok());
|
assert!(processing_frontier.observe_settled(10).is_ok());
|
||||||
for _ in 0..3 {
|
for _ in 0..3 {
|
||||||
assert!(coordinator.queue_signal(&source, signal.clone(), received_at, &mut processing_frontier).is_ok());
|
assert!(coordinator.queue_signal(&hydration, signal.clone(), received_at, &mut processing_frontier).is_ok());
|
||||||
}
|
}
|
||||||
let saturated = coordinator.queue_signal(&source, signal, received_at, &mut processing_frontier);
|
let saturated = coordinator.queue_signal(&hydration, signal, received_at, &mut processing_frontier);
|
||||||
let saturated = match saturated {
|
let saturated = match saturated {
|
||||||
std::result::Result::Ok(()) => return,
|
std::result::Result::Ok(()) => return,
|
||||||
std::result::Result::Err(value) => value,
|
std::result::Result::Err(value) => value,
|
||||||
@@ -1678,7 +2035,8 @@ async fn pre_009_abort_joins_in_flight_hydration_and_clears_pending_projection()
|
|||||||
let mut coordinator = super::RawTransactionIngestHydrationCoordinator::new(&settings);
|
let mut coordinator = super::RawTransactionIngestHydrationCoordinator::new(&settings);
|
||||||
let (frontier_sender, frontier_receiver) = tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
let (frontier_sender, frontier_receiver) = tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||||
let mut processing_frontier = super::RawTransactionIngestProcessingFrontierReporter::new(frontier_sender);
|
let mut processing_frontier = super::RawTransactionIngestProcessingFrontierReporter::new(frontier_sender);
|
||||||
assert!(coordinator.queue_signal(&source, signal, received_at, &mut processing_frontier).is_ok());
|
let hydration = source.hydration_context();
|
||||||
|
assert!(coordinator.queue_signal(&hydration, signal, received_at, &mut processing_frontier).is_ok());
|
||||||
assert_eq!(coordinator.pending_signal_count, 1);
|
assert_eq!(coordinator.pending_signal_count, 1);
|
||||||
assert_eq!(coordinator.tasks.len(), 1);
|
assert_eq!(coordinator.tasks.len(), 1);
|
||||||
assert_eq!(frontier_receiver.borrow().hydration_pending(), 1);
|
assert_eq!(frontier_receiver.borrow().hydration_pending(), 1);
|
||||||
|
|||||||
260
deltas/0.3.13/pre.003.md
Normal file
260
deltas/0.3.13/pre.003.md
Normal file
@@ -0,0 +1,260 @@
|
|||||||
|
<!-- file: deltas/0.3.13/pre.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.13-pre.003 — Standard Solana logsSubscribe + hydration commune
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
delivery précédente : 0.3.13-pre.002-fix.001
|
||||||
|
Cargo base : 0.3.13-pre.2.fix.1
|
||||||
|
delta base : deltas/0.3.13/pre.002-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de la base a été exécuté après `cargo clean` et est vert : `cargo fmt --all -- --check` silencieux, audits Rust/Markdown propres, `cargo check --workspace`, Clippy strict, 389 tests unitaires Transport, 52 tests public API Transport, 44 tests release Transport, 4 doc-tests Transport, puis 70 tests unitaires Worker et toutes ses suites d'intégration sans échec. Les smokes réseau opt-in sont restés ignorés comme prévu.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Ajouter la première nouvelle famille live de `0.3.13` sans dupliquer Transport ni l'admission RAW :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana standard WS logsSubscribe
|
||||||
|
-> context.slot + signature
|
||||||
|
-> hydration HTTP getTransaction observed
|
||||||
|
-> Common RAW
|
||||||
|
-> admission centrale Worker
|
||||||
|
-> Store
|
||||||
|
```
|
||||||
|
|
||||||
|
La tranche ne démarre toujours qu'une seule source productive à la fois. La supervision simultanée de plusieurs sources reste réservée à `pre.007`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.13-pre.003
|
||||||
|
workspace.package.version : 0.3.13-pre.3
|
||||||
|
archive attendue : ksp-general-0.3.13-pre.003.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
## Source Standard Logs
|
||||||
|
|
||||||
|
Nouvelle surface publique Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestStandardLogsSource
|
||||||
|
```
|
||||||
|
|
||||||
|
Construction caller-composed :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaLogsSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction ne réalise aucune I/O. Elle valide le protocole WebSocket, le commitment, le réseau, la représentation sûre provider/endpoint et l'existence d'une route HTTP `getTransaction` compatible sur le même cluster.
|
||||||
|
|
||||||
|
Au runtime, la source utilise exclusivement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SolanaStandardWsSession::connect
|
||||||
|
logs_subscribe
|
||||||
|
HttpTransportPool::get_transaction_observed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne construit aucun socket/client inférieur et ne dépend directement ni de `reqwest`, ni de `tokio-tungstenite`, ni de `tonic`, ni de `yellowstone-grpc-proto`.
|
||||||
|
|
||||||
|
## Projection et redaction
|
||||||
|
|
||||||
|
Une notification `logsSubscribe` est réduite à son matériau de référence nécessaire :
|
||||||
|
|
||||||
|
```text
|
||||||
|
context.slot
|
||||||
|
signature
|
||||||
|
commitment caller-composed
|
||||||
|
identité source/filter privée
|
||||||
|
```
|
||||||
|
|
||||||
|
Les champs distants `logs` et `err` ne sont pas copiés dans le signal Worker, le snapshot, la provenance textuelle ou `Debug`.
|
||||||
|
|
||||||
|
Les variantes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
All
|
||||||
|
AllWithVotes
|
||||||
|
Mentions(pubkey)
|
||||||
|
```
|
||||||
|
|
||||||
|
participent à une empreinte SHA-256 privée. La valeur du pubkey `Mentions` n'est pas exposée par le Worker.
|
||||||
|
|
||||||
|
## Identité logique
|
||||||
|
|
||||||
|
Le `source_key` Standard Logs est dérivé de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
domain KSP live-source v1
|
||||||
|
family = standard_logs
|
||||||
|
network
|
||||||
|
safe provider identity
|
||||||
|
safe endpoint identity
|
||||||
|
commitment
|
||||||
|
private filter fingerprint
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rôle et le pool HTTP d'hydration n'entrent pas dans cette identité : ils décrivent la stratégie d'enrichissement, pas une nouvelle source live.
|
||||||
|
|
||||||
|
Le rejet transactionnel des doublons et le bound global `1..32` introduits en `pre.002` s'appliquent aussi à Standard Logs.
|
||||||
|
|
||||||
|
## Hydration source-neutral
|
||||||
|
|
||||||
|
Le coordinator privé précédemment utilisé par Yellowstone est généralisé en contrat d'hydration source-neutral et réutilisé par Standard Logs.
|
||||||
|
|
||||||
|
Pour `pre.003`, chaque exécution mono-source possède encore sa propre instance. Le partage réellement global entre plusieurs tâches source sera introduit avec le supervisor `pre.007`.
|
||||||
|
|
||||||
|
Bornes effectives :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration in-flight <= persistence_concurrency
|
||||||
|
pending source signals <= admission_queue_capacity
|
||||||
|
```
|
||||||
|
|
||||||
|
Le plafond pending est donc aligné sur les settings effectifs du Worker plutôt que sur le plafond absolu maximal.
|
||||||
|
|
||||||
|
La requête commune `getTransaction observed` utilise maintenant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
encoding = base64
|
||||||
|
maxSupportedTransactionVersion = 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette adaptation s'applique également à Yellowstone. Elle prépare l'hydration aux transactions Legacy/V0/V1 mais ne qualifie aucun nouveau chemin RAW-direct V1.
|
||||||
|
|
||||||
|
## Activation runtime
|
||||||
|
|
||||||
|
Le runtime-resource aggregate peut composer Yellowstone et Standard Logs dans la collection privée bornée, mais la barrière de sûreté reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
1 source Yellowstone : productif
|
||||||
|
1 source Standard Logs : productif
|
||||||
|
2..32 sources : composition validable
|
||||||
|
start 2..32 : runtime_resources.multi_source_activation_pending avant spawn
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune source configurée n'est silencieusement ignorée.
|
||||||
|
|
||||||
|
## Preuves ajoutées
|
||||||
|
|
||||||
|
Unit tests Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
validation standard WS / commitment / route HTTP
|
||||||
|
source_key stable et filter-sensitive
|
||||||
|
source_key indépendant du hydration role
|
||||||
|
All / AllWithVotes / Mentions distingués
|
||||||
|
Mentions/URL/source-key bytes redacted
|
||||||
|
projection reference-only
|
||||||
|
fixture signature+slot -> une hydration HTTP -> matériau Common RAW exact
|
||||||
|
getTransaction base64 + maxSupportedTransactionVersion=1
|
||||||
|
rejet duplicate Standard Logs identity
|
||||||
|
```
|
||||||
|
|
||||||
|
Canaris externes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
public_api : nouvelle construction/composition accessible depuis crate root
|
||||||
|
dependency_boundary : uniquement façades Transport + Common RAW/Store existants
|
||||||
|
hardening : aucune copie logs/err/payload/url et Debug borné
|
||||||
|
release_completeness : nouvelle surface et canaris pre.003 obligatoires
|
||||||
|
```
|
||||||
|
|
||||||
|
Reconnect, resubscribe, IDs distants et backpressure WebSocket restent couverts et possédés par `ksp-onchain-transport-lib`; le Worker réutilise la façade `SolanaStandardWsSession` au lieu de reproduire ces mécanismes.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/README.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime_resources.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime_resources.rs
|
||||||
|
docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md
|
||||||
|
docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.13/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Non-claims
|
||||||
|
|
||||||
|
`pre.003` ne prétend pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockSubscribe productif
|
||||||
|
Helius transactionSubscribe productif
|
||||||
|
HTTP live block polling
|
||||||
|
supervision simultanée de plusieurs sources
|
||||||
|
fanout Store d'observations multi-source
|
||||||
|
coordinator partagé entre plusieurs tâches source simultanées
|
||||||
|
gap repair/backfill
|
||||||
|
nouvelle dépendance externe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Les contrôles réellement exécutés après finalisation de la tranche sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
PASS : General Rust rule audit clean
|
||||||
|
PASS : Rust export completeness audit 0 candidate
|
||||||
|
PASS : KSP workspace Rust rule audit clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
PASS : Markdown table audit clean (340 tables, 831 files)
|
||||||
|
|
||||||
|
supplemental normative definition-id scan
|
||||||
|
PASS : 489 definitions, 489 unique, 0 duplicate definition id
|
||||||
|
|
||||||
|
pre-packaging exhaustive diff scan
|
||||||
|
PASS : 13 fichiers modifiés, 1 ajouté, 0 supprimé
|
||||||
|
PASS : les 13 headers versionnés des fichiers modifiés sont incrémentés exactement de +1
|
||||||
|
PASS : aucune dépendance Cargo nouvelle
|
||||||
|
```
|
||||||
|
|
||||||
|
Le toolchain Rust/Cargo/Rustfmt n'est pas disponible dans cet environnement. Les commandes Cargo post-modification sont donc explicitement `NON EXÉCUTÉ LOCAL`, jamais déclarées PASS.
|
||||||
|
|
||||||
|
Les caches Python produits par les audits sont supprimés avant packaging et ne font pas partie du delta.
|
||||||
|
|
||||||
|
## Gate opérateur requis avant pre.004
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.004` reste bloquée si ce gate révèle une anomalie de format, règle, compilation, API, dépendance ou test.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md -->
|
<!-- file: docs/plans/034-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE_PLAN.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# Plan v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source RawTransaction
|
# Plan v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source RawTransaction
|
||||||
|
|
||||||
@@ -460,6 +460,26 @@ Hydration retry/rate limit : Transport HTTP.
|
|||||||
|
|
||||||
Preuve : fixture logs notification -> une seule hydration -> exact Common RAW golden ; mentions/all/allWithVotes ; stop/reconnect/backpressure ; redaction des logs.
|
Preuve : fixture logs notification -> une seule hydration -> exact Common RAW golden ; mentions/all/allWithVotes ; stop/reconnect/backpressure ; redaction des logs.
|
||||||
|
|
||||||
|
### 13.1 Décisions matérialisées en pre.003
|
||||||
|
|
||||||
|
La tranche conserve la barrière d’activation simultanée de `pre.002` : une source unique Yellowstone ou Standard Logs est productive ; une collection `2..32` reste rejetée avant spawn jusqu’au supervisor `pre.007`. Il n’est donc pas créé de supervisor provisoire qui ignorerait une partie de la configuration.
|
||||||
|
|
||||||
|
Le coordinateur d’hydration Yellowstone devient un contrat privé source-neutral réutilisé par Standard Logs. Pour cette tranche, chaque exécution mono-source en possède une instance ; la possession réellement globale entre plusieurs sources n’arrive qu’avec le supervisor `pre.007`. Cette distinction corrige le raccourci de wording « coordinator global » du sizing initial sans modifier l’objectif de convergence.
|
||||||
|
|
||||||
|
Bornes retenues :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration in-flight <= persistence_concurrency
|
||||||
|
pending source signals <= admission_queue_capacity
|
||||||
|
aucune queue Worker non bornée
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `getTransaction observed` commun passe à `maxSupportedTransactionVersion = 1` pour Yellowstone comme pour Standard Logs. Cela ne déclare aucun chemin RAW-direct V1 ; il s’agit uniquement de rendre l’hydration HTTP version-aware conformément au contrat `0.3.13`.
|
||||||
|
|
||||||
|
Pour Standard Logs, l’identité logique inclut le réseau, la famille `standard_logs`, l’identité provider/endpoint sûre, le commitment et l’empreinte privée du filtre. Le rôle/pool HTTP d’hydration n’entre pas dans cette identité. Les variantes `All`, `AllWithVotes` et `Mentions(pubkey)` sont distinguées sans recopier le pubkey dans `Debug`, snapshot ou diagnostics.
|
||||||
|
|
||||||
|
Les valeurs `logs` et `err` de la notification restent entièrement dans Transport : le Worker ne projette que `context.slot` et `signature`.
|
||||||
|
|
||||||
## 14. Contrat source WS standard blockSubscribe
|
## 14. Contrat source WS standard blockSubscribe
|
||||||
|
|
||||||
Entrée caller-composed :
|
Entrée caller-composed :
|
||||||
@@ -776,7 +796,7 @@ Collection bornée 1..32, source key, discriminants capability-owned, validation
|
|||||||
|
|
||||||
### pre.003 — WS standard logsSubscribe + hydration
|
### pre.003 — WS standard logsSubscribe + hydration
|
||||||
|
|
||||||
Source logs, signal privé, intégration coordinator global et hydration observed.
|
Source logs, signal privé, réutilisation du coordinator d’hydration source-neutral et hydration observed ; activation simultanée toujours différée à pre.007.
|
||||||
|
|
||||||
### pre.004 — WS standard blockSubscribe
|
### pre.004 — WS standard blockSubscribe
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
<!-- file: docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md -->
|
<!-- file: docs/validation/030-V0_3_13_MULTI_SOURCE_LIVE_CONVERGENCE.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# Validation v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source
|
# Validation v0.3.13 — WS standard / Helius / HTTP live + convergence multi-source
|
||||||
|
|
||||||
## 1. Rôle
|
## 1. Rôle
|
||||||
|
|
||||||
Ce document enregistre les preuves réellement exécutées et les non-claims de `0.3.13`. `pre.001` ferme l'audit/planification ; `pre.002` matérialise la composition runtime multi-source et l'identité logique sans activer encore la supervision simultanée ni une nouvelle famille de source productive.
|
Ce document enregistre les preuves réellement exécutées et les non-claims de `0.3.13`. `pre.001` ferme l'audit/planification, `pre.002` matérialise la composition runtime multi-source et `pre.003` ajoute la source productive Standard Solana `logsSubscribe` + hydration HTTP observée, sans activer encore la supervision simultanée.
|
||||||
|
|
||||||
## 2. Archive stable contrôlée
|
## 2. Archive stable contrôlée
|
||||||
|
|
||||||
@@ -658,3 +658,156 @@ cargo tree --duplicates
|
|||||||
|
|
||||||
`pre.003` ne doit pas commencer si cette livraison révèle une anomalie de règles, API, version, archive ou dépendance non corrigée.
|
`pre.003` ne doit pas commencer si cette livraison révèle une anomalie de règles, API, version, archive ou dépendance non corrigée.
|
||||||
|
|
||||||
|
## 28. Gate opérateur pre.002-fix.001 reçu avant pre.003
|
||||||
|
|
||||||
|
Le correctif de format `0.3.13-pre.002-fix.001` a été validé par l’opérateur après `cargo clean` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all : PASS
|
||||||
|
cargo fmt --all -- --check : PASS, silencieux
|
||||||
|
General Rust rule audit : clean
|
||||||
|
Rust export completeness audit : 0 candidate
|
||||||
|
KSP workspace Rust rule audit : clean
|
||||||
|
Markdown table audit : clean (340 tables, 830 files)
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
clippy workspace/all-targets/all-features -D warnings : PASS
|
||||||
|
ksp-onchain-transport-lib unit tests : 389 pass
|
||||||
|
transport public_api : 52 pass
|
||||||
|
transport release_completeness : 44 pass
|
||||||
|
transport doc-tests : 4 pass
|
||||||
|
live transport smokes : ignored opt-in, comme prévu
|
||||||
|
ksp-worker-raw-transaction-ingest-lib unit tests : 70 pass
|
||||||
|
cross_layer_completeness : 4 pass
|
||||||
|
dependency_boundary : 9 pass
|
||||||
|
hardening : 16 pass
|
||||||
|
public_api : 10 pass
|
||||||
|
release_completeness : 4 pass
|
||||||
|
worker doc-tests : 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate `pre.002-fix.001 -> pre.003` est satisfait.
|
||||||
|
|
||||||
|
## 29. Implémentation pre.003 — Standard Logs + hydration
|
||||||
|
|
||||||
|
`pre.003` ajoute `RawTransactionIngestStandardLogsSource`, construite exclusivement à partir de surfaces Transport déjà publiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaLogsSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction reste sans I/O et valide avant spawn : endpoint/protocole, commitment, réseau, provenance sûre et existence d’une route `getTransaction` compatible sur le même cluster. Aucun endpoint URL, filtre brut, credential ou client inférieur n’est exposé par l’API Worker.
|
||||||
|
|
||||||
|
Au runtime :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SolanaStandardWsSession::connect
|
||||||
|
-> logs_subscribe
|
||||||
|
-> context.slot + signature seulement
|
||||||
|
-> coordinator d’hydration Worker source-neutral
|
||||||
|
-> get_transaction_observed
|
||||||
|
encoding = base64
|
||||||
|
maxSupportedTransactionVersion = 1
|
||||||
|
-> Common RAW
|
||||||
|
-> admission centrale existante
|
||||||
|
-> Store
|
||||||
|
```
|
||||||
|
|
||||||
|
`logs` et `err` ne sont jamais copiés dans le signal Worker. Le reconnect, le resubscribe et le backpressure de la subscription restent possédés par le même acteur WebSocket Transport. Retry/reroutage/rate-limit HTTP restent possédés par Transport.
|
||||||
|
|
||||||
|
La voie Yellowstone réutilise le même contrat d’hydration source-neutral et passe elle aussi à `maxSupportedTransactionVersion = 1`; aucune nouvelle queue, aucun second client HTTP et aucun acteur WebSocket Worker ne sont introduits.
|
||||||
|
|
||||||
|
## 30. Identité et bornes pre.003
|
||||||
|
|
||||||
|
L’identité logique Standard Logs est déterministe et opaque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
domain KSP live source key
|
||||||
|
family = standard_logs
|
||||||
|
network
|
||||||
|
safe provider identity
|
||||||
|
safe endpoint identity
|
||||||
|
commitment
|
||||||
|
private filter fingerprint
|
||||||
|
SHA-256 -> [u8; 32]
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rôle/pool HTTP d’hydration est volontairement exclu de l’identité live. Les trois variantes `All`, `AllWithVotes`, `Mentions(pubkey)` produisent des empreintes distinctes ; la valeur d’un pubkey `Mentions` n’apparaît pas dans `Debug`.
|
||||||
|
|
||||||
|
Les bornes du coordinator sont maintenant alignées sur les settings effectifs :
|
||||||
|
|
||||||
|
```text
|
||||||
|
in-flight hydration <= persistence_concurrency
|
||||||
|
pending source signals <= admission_queue_capacity
|
||||||
|
```
|
||||||
|
|
||||||
|
Une activation mono-source Yellowstone ou Standard Logs est autorisée. Une composition `2..32` reste validable mais son démarrage demeure `runtime_resources.multi_source_activation_pending` avant spawn jusqu’à `pre.007`.
|
||||||
|
|
||||||
|
## 31. Preuves déterministes ajoutées pre.003
|
||||||
|
|
||||||
|
Worker unit tests ajoutés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
validation protocole/commitment/route HTTP Standard Logs
|
||||||
|
source_key filter-sensitive et hydration-role-neutral
|
||||||
|
redaction Mentions/URL/source-key bytes
|
||||||
|
projection reference-only slot/signature
|
||||||
|
fixture logs -> une hydration HTTP -> exact matériau Common RAW
|
||||||
|
requête getTransaction = base64 + maxSupportedTransactionVersion 1
|
||||||
|
rejet duplicate Standard Logs identity
|
||||||
|
```
|
||||||
|
|
||||||
|
Canaris externes ajoutés/étendus :
|
||||||
|
|
||||||
|
```text
|
||||||
|
public_api : construction et composition Standard Logs depuis crate root
|
||||||
|
dependency_boundary : Transport facade only, sans Config/Job/backend/direct network crate
|
||||||
|
hardening : aucune copie logs/err/payload/url, Debug borné
|
||||||
|
release_completeness : nouvelle surface et nouveaux canaris obligatoires
|
||||||
|
```
|
||||||
|
|
||||||
|
Les preuves de reconnect/resubscribe/backpressure restent celles de `ksp-onchain-transport-lib`, car le Worker délègue ces mécanismes au `SolanaStandardWsSession` existant plutôt que de les réimplémenter.
|
||||||
|
|
||||||
|
## 32. Validations locales post-modification pre.003
|
||||||
|
|
||||||
|
Le toolchain Cargo/Rustfmt n’est pas disponible dans l’environnement d’assemblage. Les commandes Cargo post-`pre.003` sont donc `NON EXÉCUTÉ LOCAL` et devront être confirmées par l’opérateur.
|
||||||
|
|
||||||
|
Les audits statiques réellement exécutés sont consignés dans `deltas/0.3.13/pre.003.md`.
|
||||||
|
|
||||||
|
## 33. Non-claims pre.003
|
||||||
|
|
||||||
|
`pre.003` ne prétend pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
avoir ajouté blockSubscribe productif
|
||||||
|
avoir ajouté Helius transactionSubscribe productif
|
||||||
|
avoir ajouté HTTP live polling
|
||||||
|
avoir activé plusieurs sources simultanément
|
||||||
|
avoir implémenté le fanout d’observations Store multi-source
|
||||||
|
avoir rendu le coordinator partagé entre plusieurs tâches source simultanées
|
||||||
|
avoir ajouté du gap repair ou du backfill
|
||||||
|
avoir ajouté une dépendance réseau directe au Worker
|
||||||
|
avoir exécuté Cargo localement après modification
|
||||||
|
```
|
||||||
|
|
||||||
|
## 34. Gate opérateur requis avant pre.004
|
||||||
|
|
||||||
|
Après application du delta :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.004` ne doit pas commencer si une anomalie de format, règle, compilation, API, dépendance ou test apparaît.
|
||||||
|
|||||||
Reference in New Issue
Block a user