Compare commits
88 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9785cda39d | |||
| 0cea99cd5e | |||
| 71bc754feb | |||
| cabbfed730 | |||
| 8b6ef7778a | |||
| 06319655b0 | |||
| f636169783 | |||
| 0f8f38ff52 | |||
| 142998afb3 | |||
| 665d7d2bb6 | |||
| 1746d47412 | |||
| 9bb2f7bcae | |||
| 1f2720463b | |||
| 8bb530e9b4 | |||
| 72df965a9c | |||
| f688325170 | |||
| c711213dcc | |||
| 11ecdccd65 | |||
| 27ab84d198 | |||
| 28e4da3879 | |||
| 31c0a83e51 | |||
| 34fddc3bf5 | |||
| 13c3dac554 | |||
| f7ed053e8e | |||
| 2e8feb56dd | |||
| 793178b345 | |||
| ee0359efd5 | |||
| 1dc57a85c2 | |||
| e7e8960fec | |||
| 79e278064d | |||
| 474b894d1c | |||
| 0c76b74afe | |||
| b608ed2c50 | |||
| f9d4e6bc74 | |||
| 223f11b989 | |||
| 5095be0b0d | |||
| 2aa64cf0a4 | |||
| 90c266d68a | |||
| 8410b2f4ef | |||
| ba69a5ad20 | |||
| 07b9b20eb5 | |||
| 901a0515b7 | |||
| 50c01a19c9 | |||
| 3a8fa87acb | |||
| df1dbbb8fb | |||
| 531ca851c4 | |||
| b3fd74529d | |||
| 00b376d234 | |||
| fa720d0054 | |||
| 91438e8214 | |||
| 08e9a34183 | |||
| 85937b8cbb | |||
| 107986bf85 | |||
| a8cabbebd5 | |||
| 636a0c5f43 | |||
| bf46782db3 | |||
| 02919e959e | |||
| d231eba8be | |||
| 637382f364 | |||
| bf6ef5b426 | |||
| 822e8fcd86 | |||
| 63e1a866a3 | |||
| 22c88c449d | |||
| ed6c0ac10c | |||
| 355c5d1e9a | |||
| 61d8b1dee2 | |||
| 0db32a865c | |||
| a4061ee1e1 | |||
| ec64be340e | |||
| f1c383c392 | |||
| 984c327162 | |||
| e2689d1b04 | |||
| b5c84f3df8 | |||
| 36c683df9e | |||
| 510adcb49b | |||
| ab87fd23cd | |||
| 1a352aa8d0 | |||
| 919b4ed46d | |||
| f7c57f21c5 | |||
| d4f4237723 | |||
| ba574f3052 | |||
| 0683b70abe | |||
| eaf6fae94c | |||
| e19a202565 | |||
| 8309d829cc | |||
| aabe8dedb2 | |||
| d0da907c00 | |||
| 6b434750c5 |
54
CHANGELOG.md
54
CHANGELOG.md
@@ -1,8 +1,60 @@
|
|||||||
<!-- file: CHANGELOG.md -->
|
<!-- file: CHANGELOG.md -->
|
||||||
<!-- version: 29 -->
|
<!-- version: 33 -->
|
||||||
|
|
||||||
# Changelog KSP
|
# Changelog KSP
|
||||||
|
|
||||||
|
## 0.3.13 — convergence live multi-source du Worker RawTransaction — 2026-09-10
|
||||||
|
|
||||||
|
`0.3.13` généralise `ksp-worker-raw-transaction-ingest-lib` de la verticale Yellowstone de `0.3.12` vers une composition caller-owned de `1..32` sources logiques appartenant au même réseau. Cinq familles live convergent désormais vers le même pipeline Common RAW / admission / Store : Yellowstone, Standard Logs avec hydration `getTransaction observed`, Standard Block RAW-direct, Helius `transactionSubscribe` avec hydration et HTTP Block Polling run-local RAW-direct. Le Worker reste sans Config, Job Backfill, backend Store physique, client réseau direct ou SDK provider parallèle.
|
||||||
|
|
||||||
|
La convergence partage un registre global d'hydration entre Yellowstone, Standard Logs et Helius afin de coalescer `(network, signature, commitment)` avant fan-out HTTP. Après canonicalisation, les acquisitions de même `(network, signature)` sont sérialisées de manière bornée : un contenu identique conserve les observations/provenances distinctes, tandis qu'une divergence devient un `content_conflict` terminal explicite. Aucune majorité provider, préférence silencieuse ou stratégie `first-provider-wins` n'est introduite. Standard Block et HTTP Block Polling admettent directement les transactions Legacy/V0/V1 qualifiées avec `maxSupportedTransactionVersion = 1`.
|
||||||
|
|
||||||
|
Le supervisor multi-source possède toutes les tâches et applique des bornes source-neutral sur admission, pending signals, hydrations et persistence avec une fairness minimale entre sources reference-bearing. Les snapshots agrègent activity/health/backpressure/reconnect/replay/gap sans exposer le matériau provider. La politique reste conservative : une source configurée qui échoue est terminale faute d'équivalence de coverage prouvée ; le gap repair, les rôles degraded/failover et la preuve de couverture restent donc `0.3.14`. Le shutdown stop/fault/drain/abort+join est durci contre les races, leaders d'hydration abandonnés, Store lent et publications tardives.
|
||||||
|
|
||||||
|
Le gate déterministe final passe `cargo fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy workspace/all-targets/all-features avec `-D warnings`, `cargo test --workspace --all-targets --all-features` et les graphes Cargo : `1 850` tests passent, `0` échoue et `15` restent ignored opt-in/operator-only. Les smokes keyless Solana HTTP Devnet et WebSocket Devnet passent `1/1` chacun. Yellowstone provider-gated, Helius `transactionSubscribe` live, `blockSubscribe` provider dédié, HTTP polling Worker end-to-end et Worker multi-source vers Store restent explicitement non exécutés faute de gate provisionné ; aucune fixture ne les requalifie en PASS.
|
||||||
|
|
||||||
|
`prompts/033-V0_3_14_START_PROMPT.md` ouvre `0.3.14` exclusivement depuis le futur tag stable `v0.3.13`. Cette release doit fermer le gap repair/hardening multi-source limité au run actif — replay adressable, coverage redondante, scan HTTP borné, hydration de réparation, unresolved gaps et health policy — sans transformer le Worker en moteur historique ni créer de dépendance vers `ksp-job-backfill-lib`.
|
||||||
|
|
||||||
|
## 0.3.12 — Yellowstone + hydration HTTP + continuité de run du Worker RawTransaction — 2026-09-09
|
||||||
|
|
||||||
|
`0.3.12` ouvre la première source réseau productive de `ksp-worker-raw-transaction-ingest-lib` sans remettre en cause la fondation source-neutral de `0.3.11`. Le Worker dépend désormais de la façade `ksp-onchain-transport-lib` mais reste sans Config, Job Backfill, backend Store physique, `reqwest`, `tonic` ou proto provider direct. `RawTransactionIngestRuntimeResources` reçoit une source Yellowstone caller-composed et un pool HTTP cohérent ; `start` conserve la fondation sans source, tandis que `start_with_runtime_resources` lance la verticale live supervisée.
|
||||||
|
|
||||||
|
Les updates Yellowstone `Transaction`, `TransactionStatus` et les transactions incluses dans `Block` sont projetées vers des signaux source-neutral, coalescées de manière bornée par `(network, signature, commitment)`, hydratées via `getTransaction` observed puis converties par `ksp-raw-transaction-lib` avant admission/persistence Store. `BlockMeta` et `Slot` restent continuity-only. La provenance composite conserve les identités logiques sûres de la source et de la route d’hydration, sans URL, secret ni payload distant. Les gardes réseau/signature/slot/index/commitment empêchent les croisements de matériau avant admission.
|
||||||
|
|
||||||
|
La release matérialise une processing frontier strictement run-local : le Worker distingue travail pending/settled, n’avance jamais au travers d’un pending plus ancien et compacte ses états sans revendiquer une preuve de complétude durable. Le reconnect/replay reste Transport-owned : `from_slot` et `SubscribeReplayInfo` ne sont pas pilotés par le Worker. Le Worker projette un état source-neutral `Active/Reconnecting/Closing/Closed/Failed`, compte reconnect/replay-attempt/continuity-gap et fault avec `source_failed` lorsqu’une hausse du gap Transport prouve que la rétention ne permet plus la reprise demandée. Aucun Backfill ou repair automatique n’est déclenché.
|
||||||
|
|
||||||
|
Le hardening final ferme les duplicate storms, saturation/coalescence, Store lent/fautif, stop/fault pendant hydration/reconnect, overflow de compteurs, abandon des pending sans faux `settled`, abort/join des tâches privées et absence de retry HTTP possédé par le Worker. `pre.010` ajoute les canaris cross-layer Transport -> Worker -> Common RAW -> Store et verrouille les dependency firewalls, fixtures Legacy/V0, redaction et inventaires publics.
|
||||||
|
|
||||||
|
Le gate déterministe `pre.011` passe rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, `cargo test --workspace --all-targets --all-features`, les suites ciblées et les trois inspections Cargo demandées. Les smokes live keyless HTTP Devnet et WebSocket Devnet sont exécutés séparément et passent ; les smokes Yellowstone token-gated et le Worker end-to-end Yellowstone + HTTP + Store restent explicitement non exécutés faute de ressources fournies, sans faux PASS. `pre.012` réconcilie ensuite README/USAGE et les architectures concernées avec cette verticale réellement matérialisée.
|
||||||
|
|
||||||
|
La suite reste découpée : `0.3.13` ajoute WS standard, Helius `transactionSubscribe`, HTTP live polling et la convergence multi-source ; `0.3.14` ferme le gap repair/hardening multi-source. `prompts/032-V0_3_13_START_PROMPT.md` ouvre `0.3.13` uniquement depuis le tag stable `v0.3.12` et impose un `pre.001` d’audit/sizing avant toute extension productive.
|
||||||
|
|
||||||
|
## 0.3.11 — Fondation runtime source-neutral du Worker RawTransaction ingest — 2026-09-08
|
||||||
|
|
||||||
|
`0.3.11` introduit `ksp-worker-raw-transaction-ingest-lib` comme premier Worker concret KSP pour l’acquisition continue de `RawTransaction`, tout en fermant volontairement la release **sans source réseau productive**. La crate consomme `ksp-worker-api`, `ksp-raw-transaction-lib` et la seule façade `ksp-store-lib`; elle ne dépend ni de Config, ni d’un Job, ni d’un backend Store physique, ni encore de `ksp-onchain-transport-lib`. Son identité stable est `raw_transaction_ingest`, ses settings bornent la capacité d’admission, la concurrence de persistence et la deadline de drain, et son `start` utilise exclusivement le runtime Tokio actif fourni par le caller.
|
||||||
|
|
||||||
|
Le runtime possède un supervisor privé, des tâches source/persistence détenues, une admission centrale `mpsc` bornée avec backpressure, la canonicalisation/assembly Common RAW, une observation key déterministe et la persistence atomique en mode `Normal`. Les outcomes distinguent insert, idempotence et tombstone purgée ; un contenu divergent devient `worker_raw_transaction_ingest.content_conflict`, les erreurs Store restent classifiées sans texte distant, et aucun provider gagnant n’est choisi implicitement. Les snapshots concrets latest-value se projettent directement sur `WorkerSnapshotSource`, utilisent séquences/compteurs checked et conservent le terminal pour les lecteurs tardifs.
|
||||||
|
|
||||||
|
Le hardening final ferme les races stop/fault, les source failures, la saturation observable, le Store lent/fautif et le shutdown borné. À l’expiration de `shutdown_drain_timeout`, le Worker ferme les admissions, abort les tâches restantes, les rejoint toutes puis publie `drain_timeout`; aucune tâche privée ne survit au terminal. Les tests externes de hardening/release verrouillent les sept codes d’erreur publics, les huit modules de production, les vingt-quatre exports crate-root, la redaction, le dependency firewall et l’absence de surface historique/Backfill/Transport/Config/backend direct.
|
||||||
|
|
||||||
|
Le gate technique final `pre.011` passe rustfmt check, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 56 tests Worker, `cargo test --workspace --all-targets --all-features`, les arbres Cargo Worker normal/features et `cargo tree --duplicates`. La réconciliation `pre.012` ajoute ensuite les README/USAGE durables du Worker et réaligne l’architecture sur l’état matériel : la fondation `0.3.11` n’a aucun adapter HTTP/WS/Yellowstone productif, aucune API publique d’enqueue/source registration et aucun replay/gap-repair live.
|
||||||
|
|
||||||
|
La trajectoire live reste découpée : `0.3.12` ouvre Yellowstone transactions/blocks/status + hydration HTTP + continuité de run, `0.3.13` ajoute WS standard/Helius/HTTP live et la convergence multi-source, puis `0.3.14` ferme le gap repair/hardening multi-source. `prompts/031-V0_3_12_START_PROMPT.md` ouvre `0.3.12` uniquement depuis le tag stable `v0.3.11`; son `pre.001` doit réauditer les surfaces Yellowstone/Solana/provider actuelles, l’injection Transport, l’hydration, le frontier, `from_slot`/ReplayInfo, les smokes accessibles et le sizing avant toute implémentation lourde.
|
||||||
|
|
||||||
|
## 0.3.10 — Lower-layer RAW Transaction commune + qualification cross-source — 2026-09-08
|
||||||
|
|
||||||
|
`0.3.10` introduit `ksp-raw-transaction-lib` comme lower-layer source-neutral commune aux producteurs `RawTransaction`. La canonicalisation RAW v1 auparavant locale à `ksp-job-backfill-lib` est extraite sans modifier les golden bytes/hash existants ; le Backfill consomme désormais cette common crate sans créer de dépendance vers un Worker. La bibliothèque construit les modèles `RawTransaction`/`RawTransactionObservation` de `ksp-store-api`, conserve l'identité canonique `(network, signature)`, borne et redacted les entrées hostiles, et reste volontairement sans Transport, Config, runtime async, Job, Worker ou backend Store physique.
|
||||||
|
|
||||||
|
La common matérialise également un wire Solana source-neutral Legacy/V0/V1 avec sérialisation Base64 et extraction de signature depuis le matériau transactionnel complet. La représentation V1 couvre header, compiled instructions, configuration V1 et signatures terminales ; les gardes structurelles rejettent les états non supportés plutôt que d'inventer une compatibilité. `ksp-onchain-transport-lib` ajoute `get_block_observed`, qui conserve l'identité sûre de la route réellement gagnante tout en réutilisant la validation `getBlock` existante.
|
||||||
|
|
||||||
|
Les preuves cross-source ferment explicitement ce qui est qualifié et ce qui exige encore hydration. Un `blockSubscribe` standard `full/base64` peut produire le même RAW v1 qu'un `getBlock` HTTP sur le sous-ensemble déterministe prouvé. Helius `transactionSubscribe` reste un signal nécessitant hydration HTTP lorsque `blockTime`/version manquent. Yellowstone qualifie le wire Transaction V1 et la conservation du `block_time` dans Block, mais `Transaction` nécessite hydration pour le temps de bloc et le chemin Block complet reste conservateur tant que l'identité byte-for-byte entre meta protobuf et meta JSON HTTP n'est pas prouvée. Aucun RAW-direct Yellowstone complet ni feed EARLY n'est revendiqué sans preuve.
|
||||||
|
|
||||||
|
Le gate technique final `pre.007` passe les audits Rust/Markdown, `cargo check --workspace`, Clippy workspace/all-targets/all-features avec `-D warnings`, `cargo test --workspace --all-targets --all-features`, les suites ciblées common/Transport/Backfill et les graphes Cargo demandés ; les smokes live opt-in restent ignorés conformément à leur contrat. La réconciliation `pre.008` aligne ensuite la terminologie durable des couches Store sur `D1 RAW -> D2 STRUCTURAL -> D3 DECODED -> D4 DOMAIN`, réserve `N1–N4` aux niveaux architecturaux et complète `README.md`/`USAGE.md` de `ksp-raw-transaction-lib`. Son gate opérateur repasse format, audits, `cargo check --workspace` et Clippy strict sur `0.3.10-pre.8`.
|
||||||
|
|
||||||
|
Le runtime concret a été volontairement retiré de `0.3.10` après recalibrage du sizing. La trajectoire devient `0.3.11` fondation runtime/persistence déterministe, `0.3.12` Yellowstone + hydration/continuité, `0.3.13` WS standard/Helius/HTTP live + convergence multi-source, `0.3.14` gap repair/hardening/smokes, `0.3.15` Desk d'ingestion et `0.3.16` extension Backfill multi-source. Le Job historique paramétré et le Worker continu restent des producteurs indépendants du même Store.
|
||||||
|
|
||||||
|
`prompts/030-V0_3_11_START_PROMPT.md` ouvre `0.3.11` uniquement depuis le tag stable `v0.3.10`. Son `pre.001` doit auditer la base réelle, le graphe de dépendances, le runtime/supervision, les bornes de channels, la persistence/déduplication déterministe, les snapshots et le dimensionnement avant toute implémentation lourde ; aucune source live complexe n'est requise pour fermer cette release.
|
||||||
|
|
||||||
## 0.3.9 — Worker API générique + audit exhaustif d'acquisition RawTransaction — 2026-09-05
|
## 0.3.9 — Worker API générique + audit exhaustif d'acquisition RawTransaction — 2026-09-05
|
||||||
|
|
||||||
`0.3.9` introduit `ksp-worker-api` comme façade générique, courte et runtime-neutral pour services continus. La surface stable reste Core-only et expose `WorkerId`, `WorkerKindCode`, `WorkerState`, `WorkerHealth`, `WorkerActivity`, `WorkerLifecycle`, `WorkerStopToken`, `WorkerSnapshotSequence`, `WorkerSnapshot`, `WorkerSnapshotFuture` et `WorkerSnapshotSource`. Les identités sont bornées et redacted, le lifecycle possède des transitions explicites avec terminaux immuables, le stop token est partagé/idempotent, la séquence ne wrappe pas et le snapshot source latest-value reste object-safe, `Send + Sync` et implémentable depuis l'extérieur. L'API ne possède aucun runtime `start/stop` universel, aucune sémantique Job/checkpoint, aucun domaine Solana et aucune dépendance Transport/Store/Config/Tauri.
|
`0.3.9` introduit `ksp-worker-api` comme façade générique, courte et runtime-neutral pour services continus. La surface stable reste Core-only et expose `WorkerId`, `WorkerKindCode`, `WorkerState`, `WorkerHealth`, `WorkerActivity`, `WorkerLifecycle`, `WorkerStopToken`, `WorkerSnapshotSequence`, `WorkerSnapshot`, `WorkerSnapshotFuture` et `WorkerSnapshotSource`. Les identités sont bornées et redacted, le lifecycle possède des transitions explicites avec terminaux immuables, le stop token est partagé/idempotent, la séquence ne wrappe pas et le snapshot source latest-value reste object-safe, `Send + Sync` et implémentable depuis l'extérieur. L'API ne possède aucun runtime `start/stop` universel, aucune sémantique Job/checkpoint, aucun domaine Solana et aucune dépendance Transport/Store/Config/Tauri.
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# file: Cargo.toml
|
# file: Cargo.toml
|
||||||
# version: 489
|
# version: 565
|
||||||
|
|
||||||
[workspace]
|
[workspace]
|
||||||
resolver = "3"
|
resolver = "3"
|
||||||
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api"]
|
members = ["crates/ksp-app-backfill-desk", "crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-store-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-raw-transaction-lib", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib", "crates/ksp-worker-api", "crates/ksp-worker-raw-transaction-ingest-lib"]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
version = "0.3.9"
|
version = "0.3.13"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: README.md -->
|
<!-- file: README.md -->
|
||||||
<!-- version: 11 -->
|
<!-- version: 14 -->
|
||||||
|
|
||||||
# Khadhroony Solana Project
|
# Khadhroony Solana Project
|
||||||
|
|
||||||
@@ -57,7 +57,7 @@ La couche RAW dispose d'une façade Store backend-neutral et d'un premier job hi
|
|||||||
|
|
||||||
`ksp-worker-api` fournit désormais la fondation générique des services continus : identité Worker, lifecycle borné, health/activity, intention de stop coopératif et observation latest-value. Cette API reste Core-only, runtime-neutral et sans connaissance Solana, Transport, Store ou Job. Elle ne démarre ni n'arrête elle-même un runtime concret.
|
`ksp-worker-api` fournit désormais la fondation générique des services continus : identité Worker, lifecycle borné, health/activity, intention de stop coopératif et observation latest-value. Cette API reste Core-only, runtime-neutral et sans connaissance Solana, Transport, Store ou Job. Elle ne démarre ni n'arrête elle-même un runtime concret.
|
||||||
|
|
||||||
Le futur `ksp-worker-raw-transaction-ingest-lib` sera un consumer concret distinct du Job Backfill : il fonctionnera en continu entre Start et Stop, sans scope historique métier, afin d'acquérir et persister des `RawTransaction` selon les sources/capabilities configurées. `ksp-job-backfill-lib` conserve de son côté son rôle historique paramétré et borné. Les deux producteurs restent indépendants et convergent uniquement vers les mêmes contrats RAW/Store.
|
`ksp-worker-raw-transaction-ingest-lib` est le premier Worker concret distinct du Job Backfill. Il conserve sa fondation Start/Stop, admission bornée, Common RAW, Store backend-neutral, shutdown borné et snapshots latest-value. Ses sources productives couvrent maintenant Yellowstone + hydration HTTP `getTransaction`, Standard WS `logsSubscribe` + hydration, Standard WS `blockSubscribe` RAW-direct, Helius `transactionSubscribe` + hydration et Solana HTTP live block polling. Le polling HTTP démarre au `getSlot` observé du run, découvre uniquement les blocs live via `getBlocksWithLimit` puis matérialise les slots listés via `getBlock observed`; il n'est jamais utilisé comme Backfill implicite. Les signaux hydratés de même `(network, signature, commitment)` convergent vers le même coordinator, tandis que les voies block RAW-direct passent par la même admission Common RAW. Le reconnect/replay reste propriétaire de Transport, la frontier de traitement reste run-local et un gap de rétention prouvé provoque un fault au lieu de lancer une campagne historique. `ksp-job-backfill-lib` conserve son rôle historique paramétré et borné ; les deux producteurs restent indépendants et convergent uniquement vers les mêmes contrats RAW/Store.
|
||||||
|
|
||||||
## Points d'entrée
|
## Points d'entrée
|
||||||
|
|
||||||
|
|||||||
34
ROADMAP.md
34
ROADMAP.md
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: ROADMAP.md -->
|
<!-- file: ROADMAP.md -->
|
||||||
<!-- version: 104 -->
|
<!-- version: 111 -->
|
||||||
|
|
||||||
# Roadmap KSP
|
# Roadmap KSP
|
||||||
|
|
||||||
@@ -40,7 +40,7 @@ Le roadmap décrit les objectifs à atteindre et les grandes étapes prévues. U
|
|||||||
|
|
||||||
### Cadrage
|
### Cadrage
|
||||||
|
|
||||||
- [X] `0.2.0` — Audit bot3, ordre fonctionnel de `0.2.x`, architecture durable, discipline de sizing et pipeline RAW/CORE/DECODE/SPECIALIZED stabilisés.
|
- [X] `0.2.0` — Audit bot3, ordre fonctionnel de `0.2.x`, architecture durable, discipline de sizing et pipeline RAW/STRUCTURAL/DECODED/DOMAIN stabilisés.
|
||||||
- [X] `0.2.1` — HTTP foundation stable : matrice 52+14, runtime/routing/résilience/exécution HTTP, 4 canaris typés, `std.transport`, adapter Config -> Transport, canaries de clôture, smoke Devnet opt-in et documentation durable validés ; les 48 wrappers typés restants sont reportés à `0.2.2`–`0.2.4`.
|
- [X] `0.2.1` — HTTP foundation stable : matrice 52+14, runtime/routing/résilience/exécution HTTP, 4 canaris typés, `std.transport`, adapter Config -> Transport, canaries de clôture, smoke Devnet opt-in et documentation durable validés ; les 48 wrappers typés restants sont reportés à `0.2.2`–`0.2.4`.
|
||||||
|
|
||||||
### Releases fonctionnelles décidées/pressenties
|
### Releases fonctionnelles décidées/pressenties
|
||||||
@@ -87,13 +87,13 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
```
|
```
|
||||||
|
|
||||||
- **RAW** et **STRUCTURAL** ne nécessitent aucun décodage Program.
|
- **RAW** et **STRUCTURAL** ne nécessitent aucun décodage Program.
|
||||||
- La progression n'est pas une chaîne obligatoire pour chaque famille : une donnée event-only peut s'arrêter en N1, et un état de compte pourra aller directement vers un futur decoder si aucune décomposition STRUCTURAL utile n'existe.
|
- La progression n'est pas une chaîne obligatoire pour chaque famille : une donnée event-only peut s'arrêter en D1, et un état de compte pourra aller directement vers un futur decoder si aucune décomposition STRUCTURAL utile n'existe.
|
||||||
- À la fin des couches horizontales RAW/STRUCTURAL réellement persistées, ajouter les jobs/workers/apps nécessaires avant d'ouvrir la couche suivante.
|
- À la fin des couches horizontales RAW/STRUCTURAL réellement persistées, ajouter les jobs/workers/apps nécessaires avant d'ouvrir la couche suivante.
|
||||||
- À partir de **DECODED**, avancer verticalement groupe par groupe ; **DOMAIN** désigne les projections métier et la matérialisation est le processus qui les produit.
|
- À partir de **DECODED**, avancer verticalement groupe par groupe ; **DOMAIN** désigne les projections métier et la matérialisation est le processus qui les produit.
|
||||||
|
|
||||||
## 0.3.x — RAW / acquisition persistée
|
## 0.3.x — RAW / acquisition persistée
|
||||||
|
|
||||||
- [X] `0.3.1` — `ksp-store-api` stable : modèles N1 RAW backend-agnostic `RawTransaction` et `RawAccountState` avec observations, provenance, payload/hash/timestamps bornés, 10 capabilities object-safe, queries cursorisées sans plafond métier arbitraire, outcomes idempotence/conflit et lifecycle logique rétention/tombstone/force-rehydrate ; aucun backend physique, Config, runtime Store, notification dédiée ni surface STRUCTURAL/DECODED/DOMAIN.
|
- [X] `0.3.1` — `ksp-store-api` stable : modèles D1 RAW backend-agnostic `RawTransaction` et `RawAccountState` avec observations, provenance, payload/hash/timestamps bornés, 10 capabilities object-safe, queries cursorisées sans plafond métier arbitraire, outcomes idempotence/conflit et lifecycle logique rétention/tombstone/force-rehydrate ; aucun backend physique, Config, runtime Store, notification dédiée ni surface STRUCTURAL/DECODED/DOMAIN.
|
||||||
- [X] `0.3.2` — `ksp-store-lib` + `ksp-store-postgres-lib` stables comme fondation runtime/backend PostgreSQL : feature `postgres` par défaut, `Store` lié à un unique `RawNetworkId`, Config `std.store` avec targets/bases `devnet`/`mainnet`/`testnet`, pool Deadpool borné, `tokio-postgres`, TLS Rustls `Disabled`/`VerifyFull`, moteur de migrations privé `V000` + SHA-256/advisory lock, health/readiness portable et close borné. Gate complet + PostgreSQL réel major 17 verts ; aucune table/capability `RawTransaction`/`RawAccountState` métier n'est encore ajoutée.
|
- [X] `0.3.2` — `ksp-store-lib` + `ksp-store-postgres-lib` stables comme fondation runtime/backend PostgreSQL : feature `postgres` par défaut, `Store` lié à un unique `RawNetworkId`, Config `std.store` avec targets/bases `devnet`/`mainnet`/`testnet`, pool Deadpool borné, `tokio-postgres`, TLS Rustls `Disabled`/`VerifyFull`, moteur de migrations privé `V000` + SHA-256/advisory lock, health/readiness portable et close borné. Gate complet + PostgreSQL réel major 17 verts ; aucune table/capability `RawTransaction`/`RawAccountState` métier n'est encore ajoutée.
|
||||||
- [X] `0.3.3` — Vertical slice PostgreSQL `RawTransaction` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : six capabilities transaction/observation/rétention, V001 physique liée à un réseau, acquisition canonical+observation atomique, idempotence/conflit, get/list keyset cursorisé, archive/purge/tombstone/ForceRehydrate, hardening des erreurs et du schéma, concurrence et rollback validés sur PostgreSQL 17.
|
- [X] `0.3.3` — Vertical slice PostgreSQL `RawTransaction` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : six capabilities transaction/observation/rétention, V001 physique liée à un réseau, acquisition canonical+observation atomique, idempotence/conflit, get/list keyset cursorisé, archive/purge/tombstone/ForceRehydrate, hardening des erreurs et du schéma, concurrence et rollback validés sur PostgreSQL 17.
|
||||||
- [X] `0.3.4` — Vertical slice PostgreSQL `RawAccountState` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : quatre capabilities account ajoutées aux six transaction pour une conformance RAW 10/10, V002 additive de 32 ressources au-dessus de V000/V001 immuables, state+observation atomiques, idempotence/conflit exacts, metadata Yellowstone observation-only, get/list keyset `(slot,pubkey,state_hash)` avec cursor KSPA anti-replay, hardening cross-family et live validé sur PostgreSQL 17 sans rétention destructive account.
|
- [X] `0.3.4` — Vertical slice PostgreSQL `RawAccountState` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : quatre capabilities account ajoutées aux six transaction pour une conformance RAW 10/10, V002 additive de 32 ressources au-dessus de V000/V001 immuables, state+observation atomiques, idempotence/conflit exacts, metadata Yellowstone observation-only, get/list keyset `(slot,pubkey,state_hash)` avec cursor KSPA anti-replay, hardening cross-family et live validé sur PostgreSQL 17 sans rétention destructive account.
|
||||||
@@ -102,21 +102,25 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
- [X] `0.3.7` — `ksp-app-backfill-desk` livrée comme Desk Tauri KSP spécialisée : composition Config Mainnet/Devnet/Testnet, quatre scopes HTTP, validation/start single-run, monitoring latest-value `ksp-backfill-status`, Cancel ciblé/idempotent, Resume in-session par checkpoint Rust-only réémis pour un nouveau JobId, autocomplete libre depuis `ksp-core-lib`, hardening IPC/dependency boundaries et build Linux `.deb`/`.rpm`/`.AppImage`. Aucun SQL/backend/provider physique ni checkpoint durable n’est exposé.
|
- [X] `0.3.7` — `ksp-app-backfill-desk` livrée comme Desk Tauri KSP spécialisée : composition Config Mainnet/Devnet/Testnet, quatre scopes HTTP, validation/start single-run, monitoring latest-value `ksp-backfill-status`, Cancel ciblé/idempotent, Resume in-session par checkpoint Rust-only réémis pour un nouveau JobId, autocomplete libre depuis `ksp-core-lib`, hardening IPC/dependency boundaries et build Linux `.deb`/`.rpm`/`.AppImage`. Aucun SQL/backend/provider physique ni checkpoint durable n’est exposé.
|
||||||
- [X] `0.3.8` — `ksp-app-store-desk` V1 RAW livrée sur le gabarit KSP courant comme application Tauri read-only backend-neutral. La Desk compose Config + Logging + `ksp-store-lib`, expose health/runtime sûrs, DataTables `serverSide` pour `RawTransaction`, `RawAccountState` et leurs observations, détails bornés, provenance sûre et états de rétention/tombstone, sans SQL/backend physique/Transport dans l'application. La nouvelle inspection random-access `offset + limit + counts exacts` reste distincte de la pagination machine cursor/keyset conservée pour workers/backfills/replays. Le gate final et les builds Linux `.deb`/`.rpm` sont verts.
|
- [X] `0.3.8` — `ksp-app-store-desk` V1 RAW livrée sur le gabarit KSP courant comme application Tauri read-only backend-neutral. La Desk compose Config + Logging + `ksp-store-lib`, expose health/runtime sûrs, DataTables `serverSide` pour `RawTransaction`, `RawAccountState` et leurs observations, détails bornés, provenance sûre et états de rétention/tombstone, sans SQL/backend physique/Transport dans l'application. La nouvelle inspection random-access `offset + limit + counts exacts` reste distincte de la pagination machine cursor/keyset conservée pour workers/backfills/replays. Le gate final et les builds Linux `.deb`/`.rpm` sont verts.
|
||||||
- [X] `0.3.9` — `ksp-worker-api` stable comme API générique et volontairement courte pour services continus : identité/kind bornés, lifecycle explicite, health/activity sûrs, stop token partagé, séquence/snapshot latest-value et `WorkerSnapshotSource` object-safe, avec dépendance Core-only et sans runtime/start-stop universel, Solana, Transport, Store, Config ou Tauri. Après freeze Worker API, l'audit `RawTransaction` a été consolidé dans une synthèse Store-centrique exhaustive séparant possibilités/support/preuve et classant HTTP, WS standard/provider, Yellowstone gRPC, blocs/slots, discovery + hydration, replay de continuité, archives et sources EARLY. Le Job Backfill historique paramétré et le Worker Ingest live start/stop sont deux producteurs indépendants du même Store ; `mainnet` est désormais l'identité KSP canonique et `mainnet-beta` un alias legacy/externe. Le handoff retient `ksp-raw-transaction-lib` comme lower-layer commune de canonicalisation RAW v1 pour `0.3.10`, sans edge Job ↔ Worker.
|
- [X] `0.3.9` — `ksp-worker-api` stable comme API générique et volontairement courte pour services continus : identité/kind bornés, lifecycle explicite, health/activity sûrs, stop token partagé, séquence/snapshot latest-value et `WorkerSnapshotSource` object-safe, avec dépendance Core-only et sans runtime/start-stop universel, Solana, Transport, Store, Config ou Tauri. Après freeze Worker API, l'audit `RawTransaction` a été consolidé dans une synthèse Store-centrique exhaustive séparant possibilités/support/preuve et classant HTTP, WS standard/provider, Yellowstone gRPC, blocs/slots, discovery + hydration, replay de continuité, archives et sources EARLY. Le Job Backfill historique paramétré et le Worker Ingest live start/stop sont deux producteurs indépendants du même Store ; `mainnet` est désormais l'identité KSP canonique et `mainnet-beta` un alias legacy/externe. Le handoff retient `ksp-raw-transaction-lib` comme lower-layer commune de canonicalisation RAW v1 pour `0.3.10`, sans edge Job ↔ Worker.
|
||||||
- [ ] `0.3.10` — Introduire `ksp-raw-transaction-lib` puis `ksp-worker-raw-transaction-ingest-lib` **multi-source dès V1**. La lower-layer commune extrait de `ksp-job-backfill-lib` la canonicalisation RAW v1 sans changer ses golden bytes/hash ni ses campagnes historiques. Le worker continu est démarré/arrêté sans paramètres métier de campagne, ouvre les sources/rôles activés par la composition Config, acquiert depuis son démarrage, persiste `RawTransaction + RawTransactionObservation`, publie ses snapshots/notifications indépendamment de leurs lecteurs et répare uniquement ses propres gaps de continuité live. HTTP, WS, Yellowstone gRPC et extensions provider sont des capabilities orthogonales : HTTP peut servir au live/hydration et gRPC/WS ne deviennent jamais synonymes de Worker. La release porte seulement les adaptations Transport/Config/common réellement justifiées par l'audit (`get_block_observed`, projections full source-neutral, provenance sûre, replay `from_slot`, profils/capabilities nécessaires), avec support architectural conservé même lorsque certains smokes provider restent bloqués par tier. Aucun nouveau backfill multi-stratégie, decode Program, backend physique direct ni dépendance Worker ↔ Job.
|
- [X] `0.3.10` — Lower-layer RAW Transaction commune stabilisée : `ksp-raw-transaction-lib` possède la canonicalisation RAW v1 source-neutral partagée, le Backfill est migré sans changement des golden bytes/hash, Transport expose `get_block_observed`, le wire Solana Legacy/V0/V1 est matérialisé et les canaris HTTP/WS standard/Helius/Yellowstone qualifient précisément les voies directes ou nécessitant hydration. La release se ferme sans runtime Worker concret ; celui-ci commence en `0.3.11` selon le redécoupage `0.3.11`–`0.3.14`.
|
||||||
- [ ] `0.3.11` — Introduire `ksp-app-raw-transaction-ingest-desk`, Desk Tauri KSP spécialisée qui choisit et supervise **une ou plusieurs sources/méthodes** réellement admises par `0.3.10` : lifecycle, health, rates, backpressure, reconnect/recovery, gap state et compteurs sûrs. La Desk ne réimplémente ni discovery, hydration, déduplication, reprise ni persistence ; l'inspection détaillée des RAW persistés reste la responsabilité de `ksp-app-store-desk`.
|
- [X] `0.3.11` — Fondation runtime de `ksp-worker-raw-transaction-ingest-lib` livrée : crate/dependency firewall, settings source-neutral, handle/start-stop, lifecycle, snapshots, supervisor, admission bornée, canonicalisation Common RAW, persistence/déduplication Store déterministe, backpressure et shutdown/fault hardening. La release se ferme volontairement sans source réseau productive ; Yellowstone + hydration HTTP commencent en `0.3.12`.
|
||||||
- [ ] `0.3.12` — Étendre `ksp-job-backfill-lib` et `ksp-app-backfill-desk` au **backfill multi-source/multi-stratégie** à partir de la même matrice d'acquisition auditée en `0.3.9`. Conserver la stratégie actuelle `getSignaturesForAddress + getTransaction` comme première voie HTTP valide, puis ajouter seulement les voies historiques/catch-up/gap-repair réellement pertinentes et sûres (par exemple block/slot, replay provider lorsqu'il existe), sans supposer qu'une source live WS constitue un historique universel.
|
- [X] `0.3.12` — Première source live productive du Worker livrée : Yellowstone `Transaction`/`TransactionStatus`/`Block` vers signaux source-neutral, coalescence bornée puis hydration HTTP `getTransaction` observed avant Common RAW/Store ; `BlockMeta`/`Slot` restent continuity-only. Processing frontier run-local, reconnect/from_slot/ReplayInfo Transport-owned, compteurs source-neutral et fault sur gap de rétention prouvé sans Backfill automatique. Hardening duplicate/backpressure/stop/fault et canaris cross-layer fermés ; gate workspace complet vert, smokes live HTTP Devnet + WebSocket Devnet verts, Yellowstone token-gated/Worker end-to-end laissés explicitement non exécutés.
|
||||||
|
- [X] `0.3.13` — Convergence live multi-source du Worker RAW livrée : composition caller-owned `1..32` sources d’un même réseau, cinq familles Yellowstone / Standard Logs / Standard Block / Helius Transaction / HTTP Block Polling, hydration globale des trois voies reference-bearing, RAW-direct Legacy/V0/V1 pour les deux voies blocs, observations multiples, content conflict explicite, coalescence cross-source, fairness/backpressure/health source-neutral et shutdown hardening sans second actor Transport. Gate workspace complet vert (`1 850` PASS / `0` échec / `15` ignored) et smokes keyless HTTP + WebSocket Devnet verts ; les preuves provider/resource-gated non exécutées restent explicitement non revendiquées.
|
||||||
|
- [ ] `0.3.14` — Fermer le Worker par le **gap repair/hardening multi-source** : replay natif, source redondante, scan HTTP blocs, hydration de réparation, unresolved gaps visibles, politiques degraded/unhealthy/faulted, shutdown pendant repair, smokes provider accessibles et adapter EARLY seulement si réellement prouvé.
|
||||||
|
- [ ] `0.3.15` — Introduire `ksp-app-raw-transaction-ingest-desk`, Desk Tauri KSP spécialisée qui choisit et supervise une ou plusieurs sources/méthodes réellement admises par le Worker finalisé : lifecycle, health, rates, backpressure, reconnect/recovery, gap state et compteurs sûrs. La Desk ne réimplémente ni discovery, hydration, déduplication, reprise ni persistence.
|
||||||
|
- [ ] `0.3.16` — Étendre `ksp-job-backfill-lib` et `ksp-app-backfill-desk` au **backfill multi-source/multi-stratégie** à partir de la même matrice d'acquisition auditée en `0.3.9`. Conserver la stratégie actuelle `getSignaturesForAddress + getTransaction` comme première voie HTTP valide, puis ajouter seulement les voies historiques/catch-up/gap-repair réellement pertinentes et sûres, sans supposer qu'une source live WS constitue un historique universel.
|
||||||
|
|
||||||
### TODO/IDEAS — applications spécialisées et control plane
|
### TODO/IDEAS — applications spécialisées et control plane
|
||||||
|
|
||||||
- [ ] **TODO** — faire évoluer `ksp-app-store-desk` avec chaque nouvelle couche réellement persistée : RAW d'abord, puis STRUCTURAL, DECODED, journal de processing/materialization et projections DOMAIN selon les contrats effectivement disponibles. Les vues avancées restent backend-agnostiques et passent uniquement par `ksp-store-lib`.
|
- [ ] **TODO** — faire évoluer `ksp-app-store-desk` avec chaque nouvelle couche réellement persistée : RAW d'abord, puis STRUCTURAL, DECODED, journal de processing/materialization et projections DOMAIN selon les contrats effectivement disponibles. Les vues avancées restent backend-agnostiques et passent uniquement par `ksp-store-lib`.
|
||||||
- [ ] **TODO** — différer une future `ksp-app-control-desk` jusqu'à ce que KSP dispose au minimum d'un niveau N3/D3 de processing/materialization exploitable et de plusieurs decoders réels. Cette application sera une surface **end-user simplifiée** : démarrer/arrêter les flux autorisés, lancer une recherche courante, voir l'état global et les erreurs importantes. Elle ne recopiera pas le diagnostic détaillé, les réglages avancés, les tables complètes ni toutes les fonctions des `ksp-app-*-desk` spécialisées, qui resteront les outils d'administration, développement et investigation approfondie.
|
- [ ] **TODO** — différer une future `ksp-app-control-desk` jusqu'à ce que KSP dispose au minimum d'une couche D3 DECODED de processing/materialization exploitable et de plusieurs decoders réels. Cette application sera une surface **end-user simplifiée** : démarrer/arrêter les flux autorisés, lancer une recherche courante, voir l'état global et les erreurs importantes. Elle ne recopiera pas le diagnostic détaillé, les réglages avancés, les tables complètes ni toutes les fonctions des `ksp-app-*-desk` spécialisées, qui resteront les outils d'administration, développement et investigation approfondie.
|
||||||
|
|
||||||
### TODO/IDEAS — taxonomie N1, processing et rétention
|
### TODO/IDEAS — taxonomie D1, processing et rétention
|
||||||
|
|
||||||
- [ ] **TODO** — maintenir la matrice d’admission HTTP/WS/gRPC/provider lors de toute nouvelle famille N1 : plusieurs sources ne convergent vers un même struct que si elles satisfont la même sémantique sans perte.
|
- [ ] **TODO** — maintenir la matrice d’admission HTTP/WS/gRPC/provider lors de toute nouvelle famille D1 : plusieurs sources ne convergent vers un même struct que si elles satisfont la même sémantique sans perte.
|
||||||
- [X] **TODO `0.3.9`** — matrice RAW Transaction unique produite et consolidée dans `docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`, avec séparation possibilités/support/preuve, capabilities orthogonales aux producteurs, provenance/déduplication/continuité et handoffs indépendants `0.3.10` Worker live / `0.3.12` Backfill historique.
|
- [X] **TODO `0.3.9`** — matrice RAW Transaction unique produite et consolidée dans `docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`, avec séparation possibilités/support/preuve, capabilities orthogonales aux producteurs, provenance/déduplication/continuité et handoffs désormais scindés `0.3.11`–`0.3.14` Worker live / `0.3.16` Backfill historique.
|
||||||
- [ ] **TODO Helius `0.3.10`** — ajouter uniquement les profils/capabilities Helius réellement nécessaires au Worker live après revalidation des surfaces/tier courants ; réutiliser exclusivement `KSP_SECRET_HELIUS_API_KEY` via Config et la surface `transactionSubscribe` déjà Transport-owned, sans SDK provider ni second client parallèle.
|
- [X] **TODO Helius `0.3.13`** — la voie Worker Helius `transactionSubscribe` réutilise la surface Transport et les profils Config Helius déjà existants avec `KSP_SECRET_HELIUS_API_KEY`; aucun nouveau profil, secret, SDK provider, tier codé ou second client parallèle n’a été nécessaire. Le payload riche reste Transport-owned et la voie Worker hydrate via `getTransaction observed` tant que le RAW complet n’est pas prouvé directement.
|
||||||
- [X] **TODO réseau** — identité KSP canonique fixée à `mainnet` dans Config/Store/Transport/tests ; `mainnet-beta` reste uniquement un alias legacy/externe lorsque la frontière provider l'exige. Les données N1 RAW antérieures restent non autoritaires pendant cette phase et aucune compatibilité de base de test n'impose l'ancien libellé.
|
- [X] **TODO réseau** — identité KSP canonique fixée à `mainnet` dans Config/Store/Transport/tests ; `mainnet-beta` reste uniquement un alias legacy/externe lorsque la frontière provider l'exige. Les données D1 RAW antérieures restent non autoritaires pendant cette phase et aucune compatibilité de base de test n'impose l'ancien libellé.
|
||||||
- [X] `RawAccountState` + observation — contrat commun stabilisé en `0.3.1` avec bytes complets + slot, provenance séparée et enrichissements source-specific optionnels ; la persistence PostgreSQL physique est complétée en `0.3.4` avec les quatre capabilities account et la conformance RAW 10/10.
|
- [X] `RawAccountState` + observation — contrat commun stabilisé en `0.3.1` avec bytes complets + slot, provenance séparée et enrichissements source-specific optionnels ; la persistence PostgreSQL physique est complétée en `0.3.4` avec les quatre capabilities account et la conformance RAW 10/10.
|
||||||
- [ ] **TODO** — statut/commitment transactionnel restant : `0.3.5` couvre uniquement le fait passif d’exécution `slot + signature + outcome`; réauditer séparément `signatureSubscribe` et `getSignatureStatuses` lorsqu’un consumer de commitment/snapshot réel apparaît, sans fusionner snapshot, transition et execution update dans un modèle Option-soup.
|
- [ ] **TODO** — statut/commitment transactionnel restant : `0.3.5` couvre uniquement le fait passif d’exécution `slot + signature + outcome`; réauditer séparément `signatureSubscribe` et `getSignatureStatuses` lorsqu’un consumer de commitment/snapshot réel apparaît, sans fusionner snapshot, transition et execution update dans un modèle Option-soup.
|
||||||
- [ ] **IDEA** — logs realtime enrichis : `logsSubscribe` alimente déjà la projection minimale `TransactionExecutionEvent`, mais un éventuel `TransactionLogEvent` portant les lignes de log reste différé dans `docs/IDEAS.md` jusqu’à démonstration d’un consumer et de bornes explicites. `logMessages` reste dans `RawTransaction` jusqu’à STRUCTURAL ; le wake-up post-commit reste distinct et Store API-owned conformément à `KSP-NOTIFY-*`.
|
- [ ] **IDEA** — logs realtime enrichis : `logsSubscribe` alimente déjà la projection minimale `TransactionExecutionEvent`, mais un éventuel `TransactionLogEvent` portant les lignes de log reste différé dans `docs/IDEAS.md` jusqu’à démonstration d’un consumer et de bornes explicites. `logMessages` reste dans `RawTransaction` jusqu’à STRUCTURAL ; le wake-up post-commit reste distinct et Store API-owned conformément à `KSP-NOTIFY-*`.
|
||||||
@@ -129,13 +133,13 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
- [X] rétention physique `RawTransaction` PostgreSQL — `0.3.3` matérialise `Full -> Archived -> Purged`, tombstone et ForceRehydrate atomiques ; `Compacted` reste explicitement unsupported tant qu’aucune représentation compactée réelle n’existe.
|
- [X] rétention physique `RawTransaction` PostgreSQL — `0.3.3` matérialise `Full -> Archived -> Purged`, tombstone et ForceRehydrate atomiques ; `Compacted` reste explicitement unsupported tant qu’aucune représentation compactée réelle n’existe.
|
||||||
- [ ] **TODO** — policy de rétention/compaction : définir les critères d’éligibilité fondés sur les preuves de processing et la maintenance worker/job ; Store applique une transition demandée mais ne décide pas seul qu’un RAW peut être archivé/purgé, et la compaction physique ne sera ajoutée qu’avec un besoin réel.
|
- [ ] **TODO** — policy de rétention/compaction : définir les critères d’éligibilité fondés sur les preuves de processing et la maintenance worker/job ; Store applique une transition demandée mais ne décide pas seul qu’un RAW peut être archivé/purgé, et la compaction physique ne sera ajoutée qu’avec un besoin réel.
|
||||||
- [X] frontière `ksp-interface-lib` / `ksp-store-api` — ownership documenté et canaris de non-duplication stabilisés en `0.3.1`; les events passifs non persistés restent Interface, les modèles persistants/replayables restent Store API.
|
- [X] frontière `ksp-interface-lib` / `ksp-store-api` — ownership documenté et canaris de non-duplication stabilisés en `0.3.1`; les events passifs non persistés restent Interface, les modèles persistants/replayables restent Store API.
|
||||||
- [ ] **IDEA** — réauditer la structure de processing/decode/materialization historique kbot2/kbot3 lors de l’ouverture de N2/N3 ; conserver l’isolation instruction/CPI et les statuts terminal/versionnés, sans reprendre automatiquement le schéma SQL historique.
|
- [ ] **IDEA** — réauditer la structure de processing/decode/materialization historique kbot2/kbot3 lors de l’ouverture de D2/D3 ; conserver l’isolation instruction/CPI et les statuts terminal/versionnés, sans reprendre automatiquement le schéma SQL historique.
|
||||||
|
|
||||||
## Série STRUCTURAL suivante
|
## Série STRUCTURAL suivante
|
||||||
|
|
||||||
- [ ] Définir la persistence STRUCTURAL canonique Solana générique pour les familles réellement décomposables.
|
- [ ] Définir la persistence STRUCTURAL canonique Solana générique pour les familles réellement décomposables.
|
||||||
- [ ] Implémenter en priorité `RawTransaction -> STRUCTURAL` sans decoder Program : transaction/message, comptes/références, instructions top-level, CPI/inner instructions, logs/meta/balances/return data et relations structurelles.
|
- [ ] Implémenter en priorité `RawTransaction -> STRUCTURAL` sans decoder Program : transaction/message, comptes/références, instructions top-level, CPI/inner instructions, logs/meta/balances/return data et relations structurelles.
|
||||||
- [ ] Vérifier avant extension si d'autres familles N1 possèdent une vraie décomposition STRUCTURAL utile ; ne pas créer de niveau vide par convention.
|
- [ ] Vérifier avant extension si d'autres familles D1 possèdent une vraie décomposition STRUCTURAL utile ; ne pas créer de niveau vide par convention.
|
||||||
- [ ] Ajouter replay/backfill RAW -> STRUCTURAL avec processing versionné.
|
- [ ] Ajouter replay/backfill RAW -> STRUCTURAL avec processing versionné.
|
||||||
- [ ] Ajouter le worker/service STRUCTURAL utile sans le coupler à un worker RAW concret.
|
- [ ] Ajouter le worker/service STRUCTURAL utile sans le coupler à un worker RAW concret.
|
||||||
- [ ] Faire évoluer `ksp-app-store-desk` avec des vues/requêtes STRUCTURAL lorsque cette couche est réellement persistée ; ne pas créer une seconde application de browsing des mêmes données.
|
- [ ] Faire évoluer `ksp-app-store-desk` avec des vues/requêtes STRUCTURAL lorsque cette couche est réellement persistée ; ne pas créer une seconde application de browsing des mêmes données.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
# file: crates/ksp-job-backfill-lib/Cargo.toml
|
# file: crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
# version: 4
|
# version: 7
|
||||||
|
|
||||||
[package]
|
[package]
|
||||||
name = "ksp-job-backfill-lib"
|
name = "ksp-job-backfill-lib"
|
||||||
@@ -13,6 +13,7 @@ ksp-core-lib = { path = "../ksp-core-lib" }
|
|||||||
ksp-job-api = { path = "../ksp-job-api" }
|
ksp-job-api = { path = "../ksp-job-api" }
|
||||||
ksp-logging-lib = { path = "../ksp-logging-lib" }
|
ksp-logging-lib = { path = "../ksp-logging-lib" }
|
||||||
ksp-onchain-transport-lib = { path = "../ksp-onchain-transport-lib" }
|
ksp-onchain-transport-lib = { path = "../ksp-onchain-transport-lib" }
|
||||||
|
ksp-raw-transaction-lib = { path = "../ksp-raw-transaction-lib" }
|
||||||
ksp-store-lib = { path = "../ksp-store-lib", default-features = false }
|
ksp-store-lib = { path = "../ksp-store-lib", default-features = false }
|
||||||
serde_json.workspace = true
|
serde_json.workspace = true
|
||||||
sha2.workspace = true
|
sha2.workspace = true
|
||||||
@@ -20,6 +21,9 @@ tokio = { workspace = true, features = ["macros", "sync"] }
|
|||||||
|
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
|
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
|
||||||
|
tokio-tungstenite = { workspace = true, features = ["handshake"] }
|
||||||
|
tonic = { workspace = true, features = ["codegen", "server"] }
|
||||||
|
yellowstone-grpc-proto = { workspace = true, features = ["tonic"] }
|
||||||
|
|
||||||
[lints]
|
[lints]
|
||||||
workspace = true
|
workspace = true
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
<!-- file: crates/ksp-job-backfill-lib/README.md -->
|
<!-- file: crates/ksp-job-backfill-lib/README.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# ksp-job-backfill-lib
|
# ksp-job-backfill-lib
|
||||||
|
|
||||||
`ksp-job-backfill-lib` implémente le premier job historique concret de KSP : un backfill borné de transactions Solana vers la couche RAW durable.
|
`ksp-job-backfill-lib` implémente le premier job historique concret de KSP : un backfill borné de transactions Solana vers la couche RAW durable.
|
||||||
|
|
||||||
La crate compose les contrats Job, Transport observé et Store backend-neutral sans posséder leurs politiques internes. Elle couvre l'admission, la découverte historique, l'hydratation `getTransaction`, la conversion RAW v1, la persistance atomique, la concurrence bornée, la frontier contiguë, le checkpoint caller-owned, l'annulation coopérative et les snapshots latest-value.
|
La crate compose les contrats Job, Transport observé, common RAW et Store backend-neutral sans posséder leurs politiques internes. Elle couvre l'admission, la découverte historique, l'hydratation `getTransaction`, l'adaptation vers `ksp-raw-transaction-lib`, la persistance atomique, la concurrence bornée, la frontier contiguë, le checkpoint caller-owned, l'annulation coopérative et les snapshots latest-value.
|
||||||
|
|
||||||
## Identité
|
## Identité
|
||||||
|
|
||||||
@@ -94,6 +94,7 @@ ksp-core-lib
|
|||||||
ksp-job-api
|
ksp-job-api
|
||||||
ksp-logging-lib
|
ksp-logging-lib
|
||||||
ksp-onchain-transport-lib
|
ksp-onchain-transport-lib
|
||||||
|
ksp-raw-transaction-lib
|
||||||
ksp-store-lib (default-features = false)
|
ksp-store-lib (default-features = false)
|
||||||
futures-util
|
futures-util
|
||||||
serde_json
|
serde_json
|
||||||
|
|||||||
@@ -1,18 +1,13 @@
|
|||||||
// file: crates/ksp-job-backfill-lib/src/conversion.rs
|
// file: crates/ksp-job-backfill-lib/src/conversion.rs
|
||||||
// version: 4
|
// version: 5
|
||||||
|
|
||||||
use sha2::Digest; // rust-rules: trait-import
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
/// KSP-owned source-independent RAW transaction format identifier produced by this Backfill vertical.
|
|
||||||
pub const RAW_TRANSACTION_FORMAT_ID: &str = "ksp.solana.raw_transaction";
|
|
||||||
/// Initial KSP-owned RAW transaction format version produced by this Backfill vertical.
|
|
||||||
pub const RAW_TRANSACTION_FORMAT_VERSION: u32 = 1;
|
|
||||||
|
|
||||||
const RAW_TRANSACTION_METHOD_CODE: &str = "getTransaction";
|
const RAW_TRANSACTION_METHOD_CODE: &str = "getTransaction";
|
||||||
const RAW_TRANSACTION_OBSERVATION_CONTRACT_VERSION: u32 = 1;
|
const RAW_TRANSACTION_OBSERVATION_CONTRACT_VERSION: u32 = 1;
|
||||||
const RAW_TRANSACTION_PROTOCOL_CODE: &str = "solana.http.json_rpc";
|
const RAW_TRANSACTION_PROTOCOL_CODE: &str = "solana.http.json_rpc";
|
||||||
|
|
||||||
/// Complete in-memory RAW transaction acquisition ready for the later Store persistence tranche.
|
/// Complete in-memory RAW transaction acquisition ready for Store persistence.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub struct BackfillRawAcquisition {
|
pub struct BackfillRawAcquisition {
|
||||||
inner: Box<BackfillRawAcquisitionInner>,
|
inner: Box<BackfillRawAcquisitionInner>,
|
||||||
@@ -20,35 +15,34 @@ pub struct BackfillRawAcquisition {
|
|||||||
|
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
struct BackfillRawAcquisitionInner {
|
struct BackfillRawAcquisitionInner {
|
||||||
transaction: ksp_store_lib::RawTransaction,
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
observation: ksp_store_lib::RawTransactionObservation,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl crate::BackfillRawAcquisition {
|
impl crate::BackfillRawAcquisition {
|
||||||
/// Returns the canonical RAW transaction produced from the typed Transport response.
|
/// Returns the canonical RAW transaction produced through the source-neutral common layer.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub const fn transaction(&self) -> &ksp_store_lib::RawTransaction {
|
pub fn transaction(&self) -> &ksp_store_lib::RawTransaction {
|
||||||
return &self.inner.transaction;
|
return self.inner.acquisition.transaction();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Returns the acquisition observation whose provenance records the actual successful endpoint.
|
/// Returns the Backfill-owned acquisition observation linked by the common assembly layer.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub const fn observation(&self) -> &ksp_store_lib::RawTransactionObservation {
|
pub fn observation(&self) -> &ksp_store_lib::RawTransactionObservation {
|
||||||
return &self.inner.observation;
|
return self.inner.acquisition.observation();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Consumes the in-memory acquisition into the canonical transaction and its observation.
|
/// Consumes the in-memory acquisition into the canonical transaction and its Backfill-owned observation.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
pub fn into_parts(self) -> (ksp_store_lib::RawTransaction, ksp_store_lib::RawTransactionObservation) {
|
pub fn into_parts(self) -> (ksp_store_lib::RawTransaction, ksp_store_lib::RawTransactionObservation) {
|
||||||
let inner = *self.inner;
|
let inner = *self.inner;
|
||||||
return (inner.transaction, inner.observation);
|
return inner.acquisition.into_parts();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Result of hydrating one deterministic Backfill candidate through observed `getTransaction`.
|
/// Result of hydrating one deterministic Backfill candidate through observed `getTransaction`.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub enum BackfillHydrationOutcome {
|
pub enum BackfillHydrationOutcome {
|
||||||
/// The RPC returned one complete transaction and conversion produced canonical RAW plus provenance.
|
/// The RPC returned one complete transaction and common conversion produced canonical RAW plus Backfill provenance.
|
||||||
Available(crate::BackfillRawAcquisition),
|
Available(crate::BackfillRawAcquisition),
|
||||||
/// The RPC returned JSON `null`; only the canonical transaction identity exists and no provenance is fabricated.
|
/// The RPC returned JSON `null`; only the canonical transaction identity exists and no provenance is fabricated.
|
||||||
Missing(ksp_store_lib::RawTransactionReference),
|
Missing(ksp_store_lib::RawTransactionReference),
|
||||||
@@ -71,11 +65,12 @@ impl crate::BackfillHydrationOutcome {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Hydrates one candidate with the typed observed Transport path and converts it to canonical RAW v1.
|
/// Hydrates one candidate with the typed observed Transport path and converts it through the common RAW v1 layer.
|
||||||
///
|
///
|
||||||
/// The caller supplies the local receipt timestamp because wall-clock ownership remains outside this
|
/// The caller supplies the local receipt timestamp because wall-clock ownership remains outside this
|
||||||
/// pure conversion tranche. Transport retains endpoint selection and retry. This function never
|
/// pure conversion path. Transport retains endpoint selection and retry. Backfill retains campaign
|
||||||
/// persists to Store; persistence begins in `pre.007`.
|
/// provenance and observation-key ownership; the common crate owns signature parsing, RAW v1
|
||||||
|
/// canonicalization, content hashing and transaction/observation assembly.
|
||||||
pub async fn hydrate_backfill_candidate(
|
pub async fn hydrate_backfill_candidate(
|
||||||
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
request: &crate::BackfillRequest,
|
request: &crate::BackfillRequest,
|
||||||
@@ -105,12 +100,12 @@ pub async fn hydrate_backfill_candidate(
|
|||||||
std::option::Option::None => return std::result::Result::Ok(crate::BackfillHydrationOutcome::Missing(reference)),
|
std::option::Option::None => return std::result::Result::Ok(crate::BackfillHydrationOutcome::Missing(reference)),
|
||||||
};
|
};
|
||||||
let fields = CanonicalTransactionFields {
|
let fields = CanonicalTransactionFields {
|
||||||
slot: transaction.slot(),
|
|
||||||
block_time: transaction.block_time(),
|
block_time: transaction.block_time(),
|
||||||
transaction: transaction.transaction(),
|
|
||||||
meta: transaction.meta(),
|
meta: transaction.meta(),
|
||||||
version: transaction.version(),
|
slot: transaction.slot(),
|
||||||
|
transaction: transaction.transaction(),
|
||||||
transaction_index: transaction.transaction_index(),
|
transaction_index: transaction.transaction_index(),
|
||||||
|
version: transaction.version(),
|
||||||
};
|
};
|
||||||
let acquisition = convert_available_fields(request, reference, fields, provider.as_str(), endpoint.as_str(), received_at);
|
let acquisition = convert_available_fields(request, reference, fields, provider.as_str(), endpoint.as_str(), received_at);
|
||||||
return match acquisition {
|
return match acquisition {
|
||||||
@@ -119,49 +114,22 @@ pub async fn hydrate_backfill_candidate(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Decodes one validated Base58 signature to exactly 64 canonical bytes without a Solana SDK dependency.
|
/// Decodes one validated Base58 signature through the source-neutral common parser while preserving Backfill error semantics.
|
||||||
pub(crate) fn decode_backfill_signature(signature: &crate::BackfillSignature) -> ksp_core_lib::Result<ksp_store_lib::RawTransactionSignature> {
|
pub(crate) fn decode_backfill_signature(signature: &crate::BackfillSignature) -> ksp_core_lib::Result<ksp_store_lib::RawTransactionSignature> {
|
||||||
let text = signature.as_str().as_bytes();
|
let parsed = ksp_raw_transaction_lib::parse_raw_transaction_signature(signature.as_str());
|
||||||
let mut decoded = [0_u8; 64];
|
return match parsed {
|
||||||
let mut leading_zeroes = 0_usize;
|
std::result::Result::Ok(signature) => std::result::Result::Ok(signature),
|
||||||
for byte in text {
|
std::result::Result::Err(_) => std::result::Result::Err(conversion_error("signature")),
|
||||||
if *byte != b'1' {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
leading_zeroes += 1;
|
|
||||||
}
|
|
||||||
for byte in text {
|
|
||||||
let digit = match base58_digit(*byte) {
|
|
||||||
std::option::Option::Some(digit) => digit,
|
|
||||||
std::option::Option::None => return std::result::Result::Err(conversion_error("signature")),
|
|
||||||
};
|
|
||||||
let mut carry = u32::from(digit);
|
|
||||||
for output in decoded.iter_mut().rev() {
|
|
||||||
let value = (u32::from(*output) * 58) + carry;
|
|
||||||
*output = (value & 0xff) as u8;
|
|
||||||
carry = value >> 8;
|
|
||||||
}
|
|
||||||
if carry != 0 {
|
|
||||||
return std::result::Result::Err(conversion_error("signature"));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let significant_len = match decoded.iter().position(|byte| return *byte != 0) {
|
|
||||||
std::option::Option::Some(index) => decoded.len() - index,
|
|
||||||
std::option::Option::None => 0,
|
|
||||||
};
|
};
|
||||||
if leading_zeroes + significant_len != decoded.len() {
|
|
||||||
return std::result::Result::Err(conversion_error("signature"));
|
|
||||||
}
|
|
||||||
return std::result::Result::Ok(ksp_store_lib::RawTransactionSignature::new(decoded));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
struct CanonicalTransactionFields<'a> {
|
struct CanonicalTransactionFields<'a> {
|
||||||
slot: u64,
|
|
||||||
block_time: std::option::Option<i64>,
|
block_time: std::option::Option<i64>,
|
||||||
transaction: &'a ksp_onchain_transport_lib::SolanaEncodedTransaction,
|
|
||||||
meta: &'a ksp_onchain_transport_lib::SolanaWireField<serde_json::Value>,
|
meta: &'a ksp_onchain_transport_lib::SolanaWireField<serde_json::Value>,
|
||||||
version: &'a ksp_onchain_transport_lib::SolanaWireField<ksp_onchain_transport_lib::SolanaTransactionVersion>,
|
slot: u64,
|
||||||
|
transaction: &'a ksp_onchain_transport_lib::SolanaEncodedTransaction,
|
||||||
transaction_index: &'a ksp_onchain_transport_lib::SolanaWireField<u32>,
|
transaction_index: &'a ksp_onchain_transport_lib::SolanaWireField<u32>,
|
||||||
|
version: &'a ksp_onchain_transport_lib::SolanaWireField<ksp_onchain_transport_lib::SolanaTransactionVersion>,
|
||||||
}
|
}
|
||||||
|
|
||||||
fn canonical_reference(request: &crate::BackfillRequest, candidate: &crate::BackfillCandidate) -> ksp_core_lib::Result<ksp_store_lib::RawTransactionReference> {
|
fn canonical_reference(request: &crate::BackfillRequest, candidate: &crate::BackfillCandidate) -> ksp_core_lib::Result<ksp_store_lib::RawTransactionReference> {
|
||||||
@@ -184,30 +152,9 @@ fn convert_available_fields(
|
|||||||
endpoint: &str,
|
endpoint: &str,
|
||||||
received_at: ksp_store_lib::RawTimestamp,
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
) -> ksp_core_lib::Result<crate::BackfillRawAcquisition> {
|
) -> ksp_core_lib::Result<crate::BackfillRawAcquisition> {
|
||||||
let block_time = convert_block_time(fields.block_time);
|
let transaction = canonical_transaction(&reference, fields);
|
||||||
let block_time = match block_time {
|
let transaction = match transaction {
|
||||||
std::result::Result::Ok(block_time) => block_time,
|
std::result::Result::Ok(transaction) => transaction,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
let bytes = canonical_payload_bytes(&fields);
|
|
||||||
let bytes = match bytes {
|
|
||||||
std::result::Result::Ok(bytes) => bytes,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
let hash: [u8; 32] = sha2::Sha256::digest(bytes.as_slice()).into();
|
|
||||||
let format_id = ksp_store_lib::RawFormatId::new(crate::RAW_TRANSACTION_FORMAT_ID);
|
|
||||||
let format_id = match format_id {
|
|
||||||
std::result::Result::Ok(format_id) => format_id,
|
|
||||||
std::result::Result::Err(_) => return std::result::Result::Err(conversion_error("payload.format_id")),
|
|
||||||
};
|
|
||||||
let payload = ksp_store_lib::RawPayload::try_new(
|
|
||||||
format_id,
|
|
||||||
crate::RAW_TRANSACTION_FORMAT_VERSION,
|
|
||||||
bytes.into_boxed_slice(),
|
|
||||||
ksp_store_lib::RawContentHash::new(hash),
|
|
||||||
);
|
|
||||||
let payload = match payload {
|
|
||||||
std::result::Result::Ok(payload) => payload,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
let provenance = build_provenance(request, provider, endpoint, received_at);
|
let provenance = build_provenance(request, provider, endpoint, received_at);
|
||||||
@@ -215,182 +162,60 @@ fn convert_available_fields(
|
|||||||
std::result::Result::Ok(provenance) => provenance,
|
std::result::Result::Ok(provenance) => provenance,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
let observation_key = observation_key(request, &reference, provider, endpoint);
|
let observation_key = observation_key(request, transaction.reference(), provider, endpoint);
|
||||||
let transaction = ksp_store_lib::RawTransaction::new(reference.clone(), fields.slot, block_time, payload);
|
let acquisition = ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(transaction, observation_key, provenance);
|
||||||
let observation = ksp_store_lib::RawTransactionObservation::new(observation_key, reference, provenance);
|
return std::result::Result::Ok(crate::BackfillRawAcquisition { inner: Box::new(BackfillRawAcquisitionInner { acquisition }) });
|
||||||
return std::result::Result::Ok(crate::BackfillRawAcquisition { inner: Box::new(BackfillRawAcquisitionInner { transaction, observation }) });
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn convert_block_time(value: std::option::Option<i64>) -> ksp_core_lib::Result<std::option::Option<ksp_store_lib::RawTimestamp>> {
|
fn canonical_transaction(
|
||||||
let seconds = match value {
|
reference: &ksp_store_lib::RawTransactionReference,
|
||||||
std::option::Option::Some(seconds) => seconds,
|
fields: CanonicalTransactionFields<'_>,
|
||||||
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
) -> ksp_core_lib::Result<ksp_store_lib::RawTransaction> {
|
||||||
};
|
let transaction_data = match fields.transaction {
|
||||||
let seconds = match u64::try_from(seconds) {
|
ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary { data, encoding }
|
||||||
std::result::Result::Ok(seconds) => seconds,
|
if *encoding == ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64 =>
|
||||||
std::result::Result::Err(_) => return std::result::Result::Err(conversion_error("block_time")),
|
{
|
||||||
};
|
data.clone()
|
||||||
let millis = match seconds.checked_mul(1_000) {
|
|
||||||
std::option::Option::Some(millis) => millis,
|
|
||||||
std::option::Option::None => return std::result::Result::Err(conversion_error("block_time")),
|
|
||||||
};
|
|
||||||
let timestamp = ksp_store_lib::RawTimestamp::from_unix_millis(millis);
|
|
||||||
return match timestamp {
|
|
||||||
std::result::Result::Ok(timestamp) => std::result::Result::Ok(std::option::Option::Some(timestamp)),
|
|
||||||
std::result::Result::Err(_) => std::result::Result::Err(conversion_error("block_time")),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
fn canonical_payload_bytes(fields: &CanonicalTransactionFields<'_>) -> ksp_core_lib::Result<std::vec::Vec<u8>> {
|
|
||||||
let (transaction_data, transaction_encoding) = match fields.transaction {
|
|
||||||
ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary { data, encoding } => {
|
|
||||||
if *encoding != ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64 {
|
|
||||||
return std::result::Result::Err(conversion_error("transaction.encoding"));
|
|
||||||
}
|
|
||||||
(data.as_str(), "base64")
|
|
||||||
},
|
},
|
||||||
ksp_onchain_transport_lib::SolanaEncodedTransaction::LegacyBinary(_) | ksp_onchain_transport_lib::SolanaEncodedTransaction::Json(_) => {
|
ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary { .. }
|
||||||
|
| ksp_onchain_transport_lib::SolanaEncodedTransaction::LegacyBinary(_)
|
||||||
|
| ksp_onchain_transport_lib::SolanaEncodedTransaction::Json(_) => {
|
||||||
return std::result::Result::Err(conversion_error("transaction.encoding"));
|
return std::result::Result::Err(conversion_error("transaction.encoding"));
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
let mut bytes = std::vec::Vec::new();
|
let meta = map_wire_field(fields.meta, |value| return value.clone());
|
||||||
bytes.extend_from_slice(b"{\"transaction\":[");
|
let version = map_wire_field(fields.version, |value| {
|
||||||
if let std::result::Result::Err(error) = append_json_string(&mut bytes, transaction_data) {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
bytes.push(b',');
|
|
||||||
if let std::result::Result::Err(error) = append_json_string(&mut bytes, transaction_encoding) {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
bytes.push(b']');
|
|
||||||
if let std::result::Result::Err(error) = append_wire_value(&mut bytes, "meta", fields.meta, |output, value| return append_canonical_json(output, value)) {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
let version_result = append_wire_value(&mut bytes, "version", fields.version, |output, value| {
|
|
||||||
return match value {
|
return match value {
|
||||||
ksp_onchain_transport_lib::SolanaTransactionVersion::Legacy => append_json_string(output, "legacy"),
|
ksp_onchain_transport_lib::SolanaTransactionVersion::Legacy => ksp_raw_transaction_lib::RawTransactionVersion::Legacy,
|
||||||
ksp_onchain_transport_lib::SolanaTransactionVersion::Number(number) => {
|
ksp_onchain_transport_lib::SolanaTransactionVersion::Number(number) => ksp_raw_transaction_lib::RawTransactionVersion::Number(*number),
|
||||||
output.extend_from_slice(number.to_string().as_bytes());
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
};
|
};
|
||||||
});
|
});
|
||||||
if let std::result::Result::Err(error) = version_result {
|
let transaction_index = map_wire_field(fields.transaction_index, |value| return *value);
|
||||||
return std::result::Result::Err(error);
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
}
|
reference.network().clone(),
|
||||||
let transaction_index_result = append_wire_value(&mut bytes, "transactionIndex", fields.transaction_index, |output, value| {
|
reference.signature(),
|
||||||
output.extend_from_slice(value.to_string().as_bytes());
|
fields.slot,
|
||||||
return std::result::Result::Ok(());
|
fields.block_time,
|
||||||
});
|
transaction_data,
|
||||||
if let std::result::Result::Err(error) = transaction_index_result {
|
meta,
|
||||||
return std::result::Result::Err(error);
|
version,
|
||||||
}
|
transaction_index,
|
||||||
bytes.push(b'}');
|
);
|
||||||
return std::result::Result::Ok(bytes);
|
let canonical = ksp_raw_transaction_lib::canonicalize_raw_transaction(material);
|
||||||
|
return match canonical {
|
||||||
|
std::result::Result::Ok(transaction) => std::result::Result::Ok(transaction),
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(conversion_error("common.canonicalization")),
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
fn append_wire_value<T, F>(
|
fn map_wire_field<T, U, F>(field: &ksp_onchain_transport_lib::SolanaWireField<T>, mut map_value: F) -> ksp_raw_transaction_lib::RawTransactionWireField<U>
|
||||||
output: &mut std::vec::Vec<u8>,
|
|
||||||
key: &str,
|
|
||||||
field: &ksp_onchain_transport_lib::SolanaWireField<T>,
|
|
||||||
mut append_value: F,
|
|
||||||
) -> ksp_core_lib::Result<()>
|
|
||||||
where
|
where
|
||||||
F: FnMut(&mut std::vec::Vec<u8>, &T) -> ksp_core_lib::Result<()>,
|
F: FnMut(&T) -> U,
|
||||||
{
|
{
|
||||||
return match field {
|
return match field {
|
||||||
ksp_onchain_transport_lib::SolanaWireField::Omitted => std::result::Result::Ok(()),
|
ksp_onchain_transport_lib::SolanaWireField::Omitted => ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
ksp_onchain_transport_lib::SolanaWireField::Null => {
|
ksp_onchain_transport_lib::SolanaWireField::Null => ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
output.push(b',');
|
ksp_onchain_transport_lib::SolanaWireField::Value(value) => ksp_raw_transaction_lib::RawTransactionWireField::Value(map_value(value)),
|
||||||
let key_result = append_json_string(output, key);
|
|
||||||
if let std::result::Result::Err(error) = key_result {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
output.extend_from_slice(b":null");
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
ksp_onchain_transport_lib::SolanaWireField::Value(value) => {
|
|
||||||
output.push(b',');
|
|
||||||
let key_result = append_json_string(output, key);
|
|
||||||
if let std::result::Result::Err(error) = key_result {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
output.push(b':');
|
|
||||||
append_value(output, value)
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
fn append_canonical_json(output: &mut std::vec::Vec<u8>, value: &serde_json::Value) -> ksp_core_lib::Result<()> {
|
|
||||||
return match value {
|
|
||||||
serde_json::Value::Null => {
|
|
||||||
output.extend_from_slice(b"null");
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
serde_json::Value::Bool(value) => {
|
|
||||||
if *value {
|
|
||||||
output.extend_from_slice(b"true");
|
|
||||||
} else {
|
|
||||||
output.extend_from_slice(b"false");
|
|
||||||
}
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
serde_json::Value::Number(value) => {
|
|
||||||
output.extend_from_slice(value.to_string().as_bytes());
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
serde_json::Value::String(value) => append_json_string(output, value.as_str()),
|
|
||||||
serde_json::Value::Array(values) => {
|
|
||||||
output.push(b'[');
|
|
||||||
for (index, item) in values.iter().enumerate() {
|
|
||||||
if index != 0 {
|
|
||||||
output.push(b',');
|
|
||||||
}
|
|
||||||
let item_result = append_canonical_json(output, item);
|
|
||||||
if let std::result::Result::Err(error) = item_result {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
output.push(b']');
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
serde_json::Value::Object(values) => {
|
|
||||||
output.push(b'{');
|
|
||||||
let mut keys = values.keys().collect::<std::vec::Vec<_>>();
|
|
||||||
keys.sort_unstable();
|
|
||||||
for (index, key) in keys.iter().enumerate() {
|
|
||||||
if index != 0 {
|
|
||||||
output.push(b',');
|
|
||||||
}
|
|
||||||
let key_result = append_json_string(output, key.as_str());
|
|
||||||
if let std::result::Result::Err(error) = key_result {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
output.push(b':');
|
|
||||||
let item = values.get(key.as_str());
|
|
||||||
let item = match item {
|
|
||||||
std::option::Option::Some(item) => item,
|
|
||||||
std::option::Option::None => return std::result::Result::Err(conversion_error("payload.meta")),
|
|
||||||
};
|
|
||||||
let item_result = append_canonical_json(output, item);
|
|
||||||
if let std::result::Result::Err(error) = item_result {
|
|
||||||
return std::result::Result::Err(error);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
output.push(b'}');
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
fn append_json_string(output: &mut std::vec::Vec<u8>, value: &str) -> ksp_core_lib::Result<()> {
|
|
||||||
let encoded = serde_json::to_vec(value);
|
|
||||||
return match encoded {
|
|
||||||
std::result::Result::Ok(encoded) => {
|
|
||||||
output.extend_from_slice(encoded.as_slice());
|
|
||||||
std::result::Result::Ok(())
|
|
||||||
},
|
|
||||||
std::result::Result::Err(_) => std::result::Result::Err(conversion_error("payload.json")),
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -463,18 +288,6 @@ fn hash_bytes(hasher: &mut sha2::Sha256, value: &[u8]) {
|
|||||||
hasher.update(value);
|
hasher.update(value);
|
||||||
}
|
}
|
||||||
|
|
||||||
fn base58_digit(byte: u8) -> std::option::Option<u8> {
|
|
||||||
return match byte {
|
|
||||||
b'1'..=b'9' => std::option::Option::Some(byte - b'1'),
|
|
||||||
b'A'..=b'H' => std::option::Option::Some((byte - b'A') + 9),
|
|
||||||
b'J'..=b'N' => std::option::Option::Some((byte - b'J') + 17),
|
|
||||||
b'P'..=b'Z' => std::option::Option::Some((byte - b'P') + 22),
|
|
||||||
b'a'..=b'k' => std::option::Option::Some((byte - b'a') + 33),
|
|
||||||
b'm'..=b'z' => std::option::Option::Some((byte - b'm') + 44),
|
|
||||||
_ => std::option::Option::None,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
fn conversion_error(field: &'static str) -> ksp_core_lib::Error {
|
fn conversion_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID, "invalid deterministic Backfill RAW conversion")
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID, "invalid deterministic Backfill RAW conversion")
|
||||||
.with_context("field", field);
|
.with_context("field", field);
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-job-backfill-lib/src/lib.rs
|
// file: crates/ksp-job-backfill-lib/src/lib.rs
|
||||||
// version: 6
|
// version: 7
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
#![deny(unreachable_pub)]
|
#![deny(unreachable_pub)]
|
||||||
@@ -30,10 +30,6 @@ pub use self::checkpoint::BackfillCheckpoint;
|
|||||||
pub use self::conversion::BackfillHydrationOutcome;
|
pub use self::conversion::BackfillHydrationOutcome;
|
||||||
/// Complete in-memory RAW transaction acquisition ready for later Store persistence.
|
/// Complete in-memory RAW transaction acquisition ready for later Store persistence.
|
||||||
pub use self::conversion::BackfillRawAcquisition;
|
pub use self::conversion::BackfillRawAcquisition;
|
||||||
/// KSP-owned source-independent RAW transaction format identifier produced by this Backfill vertical.
|
|
||||||
pub use self::conversion::RAW_TRANSACTION_FORMAT_ID;
|
|
||||||
/// Initial KSP-owned RAW transaction format version produced by this Backfill vertical.
|
|
||||||
pub use self::conversion::RAW_TRANSACTION_FORMAT_VERSION;
|
|
||||||
/// Hydrates one candidate through observed Transport and converts a non-null response to canonical RAW v1.
|
/// Hydrates one candidate through observed Transport and converts a non-null response to canonical RAW v1.
|
||||||
pub use self::conversion::hydrate_backfill_candidate;
|
pub use self::conversion::hydrate_backfill_candidate;
|
||||||
/// One deterministic transaction candidate produced by bounded discovery.
|
/// One deterministic transaction candidate produced by bounded discovery.
|
||||||
@@ -112,6 +108,10 @@ pub use self::runtime::BackfillJobRuntime;
|
|||||||
pub use self::runtime::BackfillJobSnapshot;
|
pub use self::runtime::BackfillJobSnapshot;
|
||||||
/// Cloneable runtime-neutral-facing latest-value source for concrete Backfill snapshots.
|
/// Cloneable runtime-neutral-facing latest-value source for concrete Backfill snapshots.
|
||||||
pub use self::runtime::BackfillSnapshotSource;
|
pub use self::runtime::BackfillSnapshotSource;
|
||||||
|
/// KSP-owned source-independent RAW transaction format identifier re-exported from the common RAW layer.
|
||||||
|
pub use ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_ID;
|
||||||
|
/// Frozen RAW transaction format version re-exported from the common RAW layer.
|
||||||
|
pub use ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_VERSION;
|
||||||
|
|
||||||
/// Internal contiguous completion frontier used by bounded execution.
|
/// Internal contiguous completion frontier used by bounded execution.
|
||||||
pub(crate) use self::checkpoint::CompletionFrontier;
|
pub(crate) use self::checkpoint::CompletionFrontier;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-job-backfill-lib/src/request.rs
|
// file: crates/ksp-job-backfill-lib/src/request.rs
|
||||||
// version: 6
|
// version: 7
|
||||||
|
|
||||||
use sha2::Digest; // rust-rules: trait-import
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
@@ -12,9 +12,9 @@ pub const MAX_BACKFILL_PAGES: usize = 10_000;
|
|||||||
/// Maximum page size admitted for one `getSignaturesForAddress` request.
|
/// Maximum page size admitted for one `getSignaturesForAddress` request.
|
||||||
pub const MAX_BACKFILL_PAGE_SIZE: usize = 1_000;
|
pub const MAX_BACKFILL_PAGE_SIZE: usize = 1_000;
|
||||||
/// Maximum Base58 text length possible for one canonical 64-byte Solana signature.
|
/// Maximum Base58 text length possible for one canonical 64-byte Solana signature.
|
||||||
pub const MAX_BACKFILL_SIGNATURE_TEXT_BYTES: usize = 88;
|
pub const MAX_BACKFILL_SIGNATURE_TEXT_BYTES: usize = ksp_raw_transaction_lib::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES;
|
||||||
/// Minimum Base58 text length possible for one canonical 64-byte Solana signature.
|
/// Minimum Base58 text length possible for one canonical 64-byte Solana signature.
|
||||||
pub const MIN_BACKFILL_SIGNATURE_TEXT_BYTES: usize = 64;
|
pub const MIN_BACKFILL_SIGNATURE_TEXT_BYTES: usize = ksp_raw_transaction_lib::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES;
|
||||||
|
|
||||||
/// Commitment levels intentionally admitted by the historical Backfill vertical.
|
/// Commitment levels intentionally admitted by the historical Backfill vertical.
|
||||||
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
|||||||
@@ -1,23 +1,28 @@
|
|||||||
// file: crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
// file: crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
// version: 6
|
// version: 9
|
||||||
|
|
||||||
//! Dependency firewall canaries through the concrete cancellation and latest-value runtime tranche.
|
//! Dependency firewall canaries through the concrete cancellation and latest-value runtime tranche.
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pre_009_manifest_uses_only_planned_ksp_edges_and_private_tokio_runtime() {
|
fn pre_003_manifest_adds_common_raw_edge_without_crossing_store_or_runtime_boundaries() {
|
||||||
let manifest = include_str!("../Cargo.toml");
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let production_manifest = match manifest.split_once("[dev-dependencies]") {
|
||||||
|
std::option::Option::Some((production, _)) => production,
|
||||||
|
std::option::Option::None => manifest,
|
||||||
|
};
|
||||||
for required in [
|
for required in [
|
||||||
"futures-util = { workspace = true, features = [\"std\"] }",
|
"futures-util = { workspace = true, features = [\"std\"] }",
|
||||||
"ksp-core-lib = { path = \"../ksp-core-lib\" }",
|
"ksp-core-lib = { path = \"../ksp-core-lib\" }",
|
||||||
"ksp-job-api = { path = \"../ksp-job-api\" }",
|
"ksp-job-api = { path = \"../ksp-job-api\" }",
|
||||||
"ksp-logging-lib = { path = \"../ksp-logging-lib\" }",
|
"ksp-logging-lib = { path = \"../ksp-logging-lib\" }",
|
||||||
"ksp-onchain-transport-lib = { path = \"../ksp-onchain-transport-lib\" }",
|
"ksp-onchain-transport-lib = { path = \"../ksp-onchain-transport-lib\" }",
|
||||||
|
"ksp-raw-transaction-lib = { path = \"../ksp-raw-transaction-lib\" }",
|
||||||
"ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }",
|
"ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }",
|
||||||
"serde_json.workspace = true",
|
"serde_json.workspace = true",
|
||||||
"sha2.workspace = true",
|
"sha2.workspace = true",
|
||||||
"tokio = { workspace = true, features = [\"macros\", \"sync\"] }",
|
"tokio = { workspace = true, features = [\"macros\", \"sync\"] }",
|
||||||
] {
|
] {
|
||||||
assert!(manifest.contains(required), "required Backfill dependency missing: {required}");
|
assert!(production_manifest.contains(required), "required Backfill dependency missing: {required}");
|
||||||
}
|
}
|
||||||
for forbidden in [
|
for forbidden in [
|
||||||
"ksp-config-lib",
|
"ksp-config-lib",
|
||||||
@@ -32,13 +37,56 @@ fn pre_009_manifest_uses_only_planned_ksp_edges_and_private_tokio_runtime() {
|
|||||||
"serde = {",
|
"serde = {",
|
||||||
"tonic",
|
"tonic",
|
||||||
] {
|
] {
|
||||||
assert!(!manifest.contains(forbidden), "forbidden Backfill dependency present: {forbidden}");
|
assert!(!production_manifest.contains(forbidden), "forbidden Backfill dependency present: {forbidden}");
|
||||||
}
|
}
|
||||||
let root = include_str!("../src/lib.rs");
|
let root = include_str!("../src/lib.rs");
|
||||||
assert!(!root.contains("tokio::"), "Tokio implementation types must not leak through the public crate root");
|
assert!(!root.contains("tokio::"), "Tokio implementation types must not leak through the public crate root");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_websocket_parity_dependency_stays_test_only() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let parts = manifest.splitn(2, "[dev-dependencies]").collect::<std::vec::Vec<_>>();
|
||||||
|
let (dependencies, dev_dependencies) = match parts.as_slice() {
|
||||||
|
[dependencies, dev_dependencies] => (*dependencies, *dev_dependencies),
|
||||||
|
_ => {
|
||||||
|
assert_eq!(parts.len(), 2, "Backfill manifest must retain exactly one dev-dependencies section");
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
assert!(!dependencies.contains("tokio-tungstenite"), "WebSocket parity client must not become a production Backfill dependency");
|
||||||
|
assert!(
|
||||||
|
dev_dependencies.contains("tokio-tungstenite = { workspace = true, features = [\"handshake\"] }"),
|
||||||
|
"WebSocket parity client must remain an explicit test-only dependency",
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_yellowstone_parity_dependencies_stay_test_only() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let parts = manifest.splitn(2, "[dev-dependencies]").collect::<std::vec::Vec<_>>();
|
||||||
|
let (dependencies, dev_dependencies) = match parts.as_slice() {
|
||||||
|
[dependencies, dev_dependencies] => (*dependencies, *dev_dependencies),
|
||||||
|
_ => {
|
||||||
|
assert_eq!(parts.len(), 2, "Backfill manifest must retain exactly one dev-dependencies section");
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
assert!(!dependencies.contains("tonic"), "Yellowstone parity server must not become a production Backfill dependency");
|
||||||
|
assert!(!dependencies.contains("yellowstone-grpc-proto"), "Yellowstone protobuf must not become a production Backfill dependency");
|
||||||
|
assert!(
|
||||||
|
dev_dependencies.contains("tonic = { workspace = true, features = [\"codegen\", \"server\"] }"),
|
||||||
|
"Yellowstone parity server must remain an explicit test-only dependency",
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
dev_dependencies.contains("yellowstone-grpc-proto = { workspace = true, features = [\"tonic\"] }"),
|
||||||
|
"Yellowstone protobuf must remain an explicit test-only dependency",
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pre_009_production_sources_keep_transport_store_and_scheduler_ownership_separate() {
|
fn pre_009_production_sources_keep_transport_store_and_scheduler_ownership_separate() {
|
||||||
let neutral_sources = [
|
let neutral_sources = [
|
||||||
@@ -56,7 +104,12 @@ fn pre_009_production_sources_keep_transport_store_and_scheduler_ownership_separ
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
let conversion = include_str!("../src/conversion.rs");
|
let conversion = include_str!("../src/conversion.rs");
|
||||||
assert!(conversion.contains("serde_json::"));
|
assert!(conversion.contains("ksp_raw_transaction_lib::canonicalize_raw_transaction"));
|
||||||
|
assert!(conversion.contains("ksp_raw_transaction_lib::assemble_raw_transaction_acquisition"));
|
||||||
|
assert!(conversion.contains("ksp_raw_transaction_lib::parse_raw_transaction_signature"));
|
||||||
|
assert!(!conversion.contains("fn append_canonical_json"));
|
||||||
|
assert!(!conversion.contains("fn base58_digit"));
|
||||||
|
assert!(!conversion.contains("RawPayload::try_new"));
|
||||||
assert!(conversion.contains("get_transaction_observed"));
|
assert!(conversion.contains("get_transaction_observed"));
|
||||||
assert!(conversion.contains("SolanaTransactionEncoding::Base64"));
|
assert!(conversion.contains("SolanaTransactionEncoding::Base64"));
|
||||||
assert!(conversion.contains("std::option::Option::Some(0)"));
|
assert!(conversion.contains("std::option::Option::Some(0)"));
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-job-backfill-lib/tests/hardening.rs
|
// file: crates/ksp-job-backfill-lib/tests/hardening.rs
|
||||||
// version: 4
|
// version: 7
|
||||||
|
|
||||||
//! Adversarial, security, visibility and external-boundary hardening canaries for `pre.010`.
|
//! Adversarial, security, visibility and external-boundary hardening canaries for `pre.010`.
|
||||||
|
|
||||||
@@ -268,13 +268,14 @@ fn pre_010_manifest_dependency_surface_is_exact_and_backend_neutral() {
|
|||||||
"ksp-job-api",
|
"ksp-job-api",
|
||||||
"ksp-logging-lib",
|
"ksp-logging-lib",
|
||||||
"ksp-onchain-transport-lib",
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
"ksp-store-lib",
|
"ksp-store-lib",
|
||||||
"serde_json.workspace",
|
"serde_json.workspace",
|
||||||
"sha2.workspace",
|
"sha2.workspace",
|
||||||
"tokio",
|
"tokio",
|
||||||
])
|
])
|
||||||
);
|
);
|
||||||
assert_eq!(dev, std::collections::BTreeSet::from(["tokio"]));
|
assert_eq!(dev, std::collections::BTreeSet::from(["tokio", "tokio-tungstenite", "tonic", "yellowstone-grpc-proto"]));
|
||||||
assert!(manifest.contains("ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }"));
|
assert!(manifest.contains("ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }"));
|
||||||
assert!(manifest.contains("tokio = { workspace = true, features = [\"macros\", \"sync\"] }"));
|
assert!(manifest.contains("tokio = { workspace = true, features = [\"macros\", \"sync\"] }"));
|
||||||
assert!(!manifest.contains("ksp-store-postgres-lib"));
|
assert!(!manifest.contains("ksp-store-postgres-lib"));
|
||||||
@@ -328,6 +329,7 @@ fn pre_010_lower_layers_have_no_dependency_return_to_job() {
|
|||||||
include_str!("../../ksp-core-lib/Cargo.toml"),
|
include_str!("../../ksp-core-lib/Cargo.toml"),
|
||||||
include_str!("../../ksp-logging-lib/Cargo.toml"),
|
include_str!("../../ksp-logging-lib/Cargo.toml"),
|
||||||
include_str!("../../ksp-onchain-transport-lib/Cargo.toml"),
|
include_str!("../../ksp-onchain-transport-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-raw-transaction-lib/Cargo.toml"),
|
||||||
include_str!("../../ksp-store-api/Cargo.toml"),
|
include_str!("../../ksp-store-api/Cargo.toml"),
|
||||||
include_str!("../../ksp-store-lib/Cargo.toml"),
|
include_str!("../../ksp-store-lib/Cargo.toml"),
|
||||||
include_str!("../../ksp-store-postgres-lib/Cargo.toml"),
|
include_str!("../../ksp-store-postgres-lib/Cargo.toml"),
|
||||||
|
|||||||
158
crates/ksp-job-backfill-lib/tests/http_block_material.rs
Normal file
158
crates/ksp-job-backfill-lib/tests/http_block_material.rs
Normal file
@@ -0,0 +1,158 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/tests/http_block_material.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! Cross-layer canary for the future Worker-owned HTTP block adapter without changing production ownership.
|
||||||
|
|
||||||
|
fn pool_for_url(url: &str) -> ksp_core_lib::Result<ksp_onchain_transport_lib::HttpTransportPool> {
|
||||||
|
let endpoint_url = match ksp_onchain_transport_lib::HttpEndpointUrl::parse(url) {
|
||||||
|
Ok(endpoint_url) => endpoint_url,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let role = ksp_onchain_transport_lib::HttpEndpointRoleSettings::new(
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("default"),
|
||||||
|
true,
|
||||||
|
std::vec![ksp_onchain_transport_lib::HttpRequestKind::wildcard()],
|
||||||
|
10,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleLimits::new(
|
||||||
|
std::option::Option::None,
|
||||||
|
std::option::Option::None,
|
||||||
|
std::option::Option::None,
|
||||||
|
std::option::Option::None,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
let endpoint = ksp_onchain_transport_lib::HttpEndpointSettings::new(
|
||||||
|
"http-block-fixture",
|
||||||
|
true,
|
||||||
|
ksp_onchain_transport_lib::HttpProviderName::new("fixture-provider"),
|
||||||
|
ksp_onchain_transport_lib::HttpClusterName::new("devnet"),
|
||||||
|
endpoint_url,
|
||||||
|
std::time::Duration::from_secs(1),
|
||||||
|
std::time::Duration::from_secs(1),
|
||||||
|
std::option::Option::Some(1),
|
||||||
|
std::vec![role],
|
||||||
|
);
|
||||||
|
return ksp_onchain_transport_lib::HttpTransportPool::new(ksp_onchain_transport_lib::HttpTransportSettings::new(
|
||||||
|
std::vec![endpoint],
|
||||||
|
ksp_onchain_transport_lib::HttpRetrySettings::new(0, std::time::Duration::from_millis(1), std::time::Duration::from_millis(1)),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn serve_once(body: &'static str) -> std::io::Result<(std::string::String, std::thread::JoinHandle<std::io::Result<()>>)> {
|
||||||
|
let listener = match std::net::TcpListener::bind("127.0.0.1:0") {
|
||||||
|
Ok(listener) => listener,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let address = match listener.local_addr() {
|
||||||
|
Ok(address) => address,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let handle = std::thread::spawn(move || {
|
||||||
|
let (mut stream, _) = match listener.accept() {
|
||||||
|
Ok(accepted) => accepted,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let mut bytes = std::vec::Vec::new();
|
||||||
|
let mut buffer = [0_u8; 1024];
|
||||||
|
loop {
|
||||||
|
let count = match std::io::Read::read(&mut stream, &mut buffer) {
|
||||||
|
Ok(count) => count,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
if count == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
bytes.extend_from_slice(&buffer[..count]);
|
||||||
|
if bytes.windows(4).any(|window| return window == b"\r\n\r\n") {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let response = format!("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", body.len(), body);
|
||||||
|
if let Err(error) = std::io::Write::write_all(&mut stream, response.as_bytes()) {
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
return Ok(());
|
||||||
|
});
|
||||||
|
return Ok((format!("http://{address}"), handle));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_meta(field: &ksp_onchain_transport_lib::SolanaWireField<serde_json::Value>) -> ksp_raw_transaction_lib::RawTransactionWireField<serde_json::Value> {
|
||||||
|
return match field {
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Omitted => ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Null => ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(value) => ksp_raw_transaction_lib::RawTransactionWireField::Value(value.clone()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_version(
|
||||||
|
field: &ksp_onchain_transport_lib::SolanaWireField<ksp_onchain_transport_lib::SolanaTransactionVersion>,
|
||||||
|
) -> ksp_raw_transaction_lib::RawTransactionWireField<ksp_raw_transaction_lib::RawTransactionVersion> {
|
||||||
|
return match field {
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Omitted => ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Null => ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(ksp_onchain_transport_lib::SolanaTransactionVersion::Legacy) => {
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(ksp_raw_transaction_lib::RawTransactionVersion::Legacy)
|
||||||
|
},
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(ksp_onchain_transport_lib::SolanaTransactionVersion::Number(number)) => {
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(ksp_raw_transaction_lib::RawTransactionVersion::Number(*number))
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_http_observed_block_projects_each_base64_transaction_to_common_material_without_new_production_edge() {
|
||||||
|
const BODY: &str = concat!(
|
||||||
|
"{\"jsonrpc\":\"2.0\",\"result\":{\"previousBlockhash\":\"previous\",\"blockhash\":\"block\",\"parentSlot\":430000122,",
|
||||||
|
"\"rewards\":[],\"numRewardPartitions\":0,\"blockTime\":1787072400,\"blockHeight\":410000000,",
|
||||||
|
"\"transactions\":[{\"transaction\":[\"AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
|
||||||
|
"AAAAAA\",\"base64\"],\"meta\":{\"err\":null,\"fee\":5000},\"version\":\"legacy\"},",
|
||||||
|
"{\"transaction\":[\"AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEA\",",
|
||||||
|
"\"base64\"],\"meta\":{\"err\":null,\"fee\":6000},\"version\":0}]},\"id\":1}",
|
||||||
|
);
|
||||||
|
let (url, handle) = serve_once(BODY).expect("fixture server must start");
|
||||||
|
let pool = pool_for_url(url.as_str()).expect("fixture pool must build");
|
||||||
|
let config = ksp_onchain_transport_lib::SolanaGetBlockConfig::new(
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
std::option::Option::Some(false),
|
||||||
|
);
|
||||||
|
let observed = pool
|
||||||
|
.get_block_observed(&ksp_onchain_transport_lib::HttpRoleName::new("default"), 430_000_123, std::option::Option::Some(&config))
|
||||||
|
.await
|
||||||
|
.expect("observed block fixture must succeed");
|
||||||
|
assert_eq!(observed.endpoint_name(), "http-block-fixture");
|
||||||
|
assert_eq!(observed.provider().as_str(), "fixture-provider");
|
||||||
|
let block = observed.value().as_ref().expect("fixture block must be present");
|
||||||
|
let transactions = block.transactions().value().expect("fixture transactions must be present");
|
||||||
|
assert_eq!(transactions.len(), 2);
|
||||||
|
let network = ksp_store_lib::RawNetworkId::new("devnet").expect("fixture network must be valid");
|
||||||
|
let expected_signatures = [[0_u8; 64], [1_u8; 64]];
|
||||||
|
for (index, transaction) in transactions.iter().enumerate() {
|
||||||
|
let (data, meta, version) = match transaction.transaction() {
|
||||||
|
ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary {
|
||||||
|
data,
|
||||||
|
encoding: ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64,
|
||||||
|
} => (data.as_str(), map_meta(transaction.meta()), map_version(transaction.version())),
|
||||||
|
_ => panic!("fixture must remain full Base64 transaction material"),
|
||||||
|
};
|
||||||
|
let transaction_index = u32::try_from(index).expect("fixture transaction index must fit u32");
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64_with_embedded_signature(
|
||||||
|
network.clone(),
|
||||||
|
430_000_123,
|
||||||
|
block.block_time(),
|
||||||
|
data,
|
||||||
|
meta,
|
||||||
|
version,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(transaction_index),
|
||||||
|
)
|
||||||
|
.expect("HTTP block material must map to common RAW material");
|
||||||
|
let raw = ksp_raw_transaction_lib::canonicalize_raw_transaction(material).expect("mapped HTTP block material must canonicalize");
|
||||||
|
assert_eq!(raw.reference().signature().as_bytes(), &expected_signatures[index]);
|
||||||
|
assert_eq!(raw.slot(), 430_000_123);
|
||||||
|
assert_eq!(raw.block_time().map(|value| return value.unix_millis()), std::option::Option::Some(1_787_072_400_000));
|
||||||
|
assert!(raw.payload().bytes().ends_with(format!("\"transactionIndex\":{transaction_index}}}").as_bytes()));
|
||||||
|
}
|
||||||
|
handle.join().expect("fixture server must join").expect("fixture server must complete successfully");
|
||||||
|
return;
|
||||||
|
}
|
||||||
503
crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs
Normal file
503
crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs
Normal file
@@ -0,0 +1,503 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
//! Cross-layer parity canaries qualifying standard `blockSubscribe` and Helius `transactionSubscribe` for RAW v1 ingestion.
|
||||||
|
|
||||||
|
use futures_util::SinkExt; // rust-rules: trait-import
|
||||||
|
use futures_util::StreamExt; // rust-rules: trait-import
|
||||||
|
|
||||||
|
const FIXTURE_BLOCK_TIME_MILLIS: u64 = 1_787_072_400_000;
|
||||||
|
const FIXTURE_SLOT: u64 = 430_000_123;
|
||||||
|
const ZERO_SIGNATURE_TEXT: &str = "1111111111111111111111111111111111111111111111111111111111111111";
|
||||||
|
const ZERO_TRANSACTION_BASE64: &str = "AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
|
||||||
|
|
||||||
|
type FixtureWebSocket = tokio_tungstenite::WebSocketStream<tokio::net::TcpStream>;
|
||||||
|
|
||||||
|
fn http_pool_for_url(url: &str) -> ksp_core_lib::Result<ksp_onchain_transport_lib::HttpTransportPool> {
|
||||||
|
let endpoint_url = match ksp_onchain_transport_lib::HttpEndpointUrl::parse(url) {
|
||||||
|
Ok(endpoint_url) => endpoint_url,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let role = ksp_onchain_transport_lib::HttpEndpointRoleSettings::new(
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("default"),
|
||||||
|
true,
|
||||||
|
std::vec![ksp_onchain_transport_lib::HttpRequestKind::wildcard()],
|
||||||
|
10,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleLimits::new(
|
||||||
|
std::option::Option::None,
|
||||||
|
std::option::Option::None,
|
||||||
|
std::option::Option::None,
|
||||||
|
std::option::Option::None,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
let endpoint = ksp_onchain_transport_lib::HttpEndpointSettings::new(
|
||||||
|
"ws-parity-http-fixture",
|
||||||
|
true,
|
||||||
|
ksp_onchain_transport_lib::HttpProviderName::new("fixture-http-provider"),
|
||||||
|
ksp_onchain_transport_lib::HttpClusterName::new("devnet"),
|
||||||
|
endpoint_url,
|
||||||
|
std::time::Duration::from_secs(1),
|
||||||
|
std::time::Duration::from_secs(1),
|
||||||
|
std::option::Option::Some(1),
|
||||||
|
std::vec![role],
|
||||||
|
);
|
||||||
|
return ksp_onchain_transport_lib::HttpTransportPool::new(ksp_onchain_transport_lib::HttpTransportSettings::new(
|
||||||
|
std::vec![endpoint],
|
||||||
|
ksp_onchain_transport_lib::HttpRetrySettings::new(0, std::time::Duration::from_millis(1), std::time::Duration::from_millis(1)),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn serve_http_once(body: &'static str) -> std::io::Result<(std::string::String, std::thread::JoinHandle<std::io::Result<()>>)> {
|
||||||
|
let listener = match std::net::TcpListener::bind("127.0.0.1:0") {
|
||||||
|
Ok(listener) => listener,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let address = match listener.local_addr() {
|
||||||
|
Ok(address) => address,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let handle = std::thread::spawn(move || {
|
||||||
|
let (mut stream, _) = match listener.accept() {
|
||||||
|
Ok(accepted) => accepted,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let mut bytes = std::vec::Vec::new();
|
||||||
|
let mut buffer = [0_u8; 1024];
|
||||||
|
loop {
|
||||||
|
let count = match std::io::Read::read(&mut stream, &mut buffer) {
|
||||||
|
Ok(count) => count,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
if count == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
bytes.extend_from_slice(&buffer[..count]);
|
||||||
|
if bytes.windows(4).any(|window| return window == b"\r\n\r\n") {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let response = format!("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", body.len(), body);
|
||||||
|
if let Err(error) = std::io::Write::write_all(&mut stream, response.as_bytes()) {
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
return Ok(());
|
||||||
|
});
|
||||||
|
return Ok((format!("http://{address}"), handle));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ws_endpoint(url: &str, protocol: ksp_onchain_transport_lib::WsProtocolKind) -> ksp_core_lib::Result<ksp_onchain_transport_lib::WsEndpointSettings> {
|
||||||
|
let parsed = match ksp_onchain_transport_lib::WsEndpointUrl::parse(url) {
|
||||||
|
Ok(parsed) => parsed,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
return Ok(ksp_onchain_transport_lib::WsEndpointSettings::new(
|
||||||
|
"ws-raw-parity-fixture",
|
||||||
|
true,
|
||||||
|
ksp_onchain_transport_lib::WsProviderName::new("fixture-ws-provider"),
|
||||||
|
ksp_onchain_transport_lib::WsClusterName::new("devnet"),
|
||||||
|
protocol,
|
||||||
|
parsed,
|
||||||
|
ksp_onchain_transport_lib::WsSessionSettings::default(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn read_ws_request(websocket: &mut FixtureWebSocket) -> std::result::Result<serde_json::Value, std::string::String> {
|
||||||
|
let message = match websocket.next().await {
|
||||||
|
std::option::Option::Some(std::result::Result::Ok(message)) => message,
|
||||||
|
std::option::Option::Some(std::result::Result::Err(_)) => return Err("fixture WebSocket request decode failed".to_owned()),
|
||||||
|
std::option::Option::None => return Err("fixture WebSocket request was absent".to_owned()),
|
||||||
|
};
|
||||||
|
let text = match message.to_text() {
|
||||||
|
Ok(text) => text,
|
||||||
|
Err(_) => return Err("fixture WebSocket request was not text".to_owned()),
|
||||||
|
};
|
||||||
|
return serde_json::from_str(text).map_err(|_| return "fixture WebSocket request JSON was invalid".to_owned());
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn send_ws_json(websocket: &mut FixtureWebSocket, value: serde_json::Value) -> std::result::Result<(), std::string::String> {
|
||||||
|
return websocket
|
||||||
|
.send(tokio_tungstenite::tungstenite::Message::Text(value.to_string().into()))
|
||||||
|
.await
|
||||||
|
.map_err(|_| return "fixture WebSocket response send failed".to_owned());
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn send_ws_result(
|
||||||
|
websocket: &mut FixtureWebSocket,
|
||||||
|
request: &serde_json::Value,
|
||||||
|
result: serde_json::Value,
|
||||||
|
) -> std::result::Result<(), std::string::String> {
|
||||||
|
let id = match request.get("id").and_then(serde_json::Value::as_u64) {
|
||||||
|
std::option::Option::Some(id) => id,
|
||||||
|
std::option::Option::None => return Err("fixture request id was absent".to_owned()),
|
||||||
|
};
|
||||||
|
return send_ws_json(websocket, serde_json::json!({"jsonrpc":"2.0","id":id,"result":result})).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_ws_close(websocket: &mut FixtureWebSocket) {
|
||||||
|
loop {
|
||||||
|
match websocket.next().await {
|
||||||
|
std::option::Option::Some(std::result::Result::Ok(tokio_tungstenite::tungstenite::Message::Close(_))) => return,
|
||||||
|
std::option::Option::Some(std::result::Result::Ok(_)) => {},
|
||||||
|
std::option::Option::Some(std::result::Result::Err(_)) | std::option::Option::None => return,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_meta(field: &ksp_onchain_transport_lib::SolanaWireField<serde_json::Value>) -> ksp_raw_transaction_lib::RawTransactionWireField<serde_json::Value> {
|
||||||
|
return match field {
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Omitted => ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Null => ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(value) => ksp_raw_transaction_lib::RawTransactionWireField::Value(value.clone()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_version(
|
||||||
|
field: &ksp_onchain_transport_lib::SolanaWireField<ksp_onchain_transport_lib::SolanaTransactionVersion>,
|
||||||
|
) -> ksp_raw_transaction_lib::RawTransactionWireField<ksp_raw_transaction_lib::RawTransactionVersion> {
|
||||||
|
return match field {
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Omitted => ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Null => ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(ksp_onchain_transport_lib::SolanaTransactionVersion::Legacy) => {
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(ksp_raw_transaction_lib::RawTransactionVersion::Legacy)
|
||||||
|
},
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(ksp_onchain_transport_lib::SolanaTransactionVersion::Number(number)) => {
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(ksp_raw_transaction_lib::RawTransactionVersion::Number(*number))
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn raw_from_block_transaction(
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
slot: u64,
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
transaction: &ksp_onchain_transport_lib::SolanaBlockTransaction,
|
||||||
|
transaction_index: u32,
|
||||||
|
) -> std::option::Option<ksp_store_lib::RawTransaction> {
|
||||||
|
let data = match transaction.transaction() {
|
||||||
|
ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary { data, encoding: ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64 } => {
|
||||||
|
data.as_str()
|
||||||
|
},
|
||||||
|
_ => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let material = match ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64_with_embedded_signature(
|
||||||
|
network,
|
||||||
|
slot,
|
||||||
|
block_time,
|
||||||
|
data,
|
||||||
|
map_meta(transaction.meta()),
|
||||||
|
map_version(transaction.version()),
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(transaction_index),
|
||||||
|
) {
|
||||||
|
Ok(material) => material,
|
||||||
|
Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return ksp_raw_transaction_lib::canonicalize_raw_transaction(material).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn raw_from_confirmed_transaction(
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
transaction: &ksp_onchain_transport_lib::SolanaConfirmedTransaction,
|
||||||
|
) -> std::option::Option<ksp_store_lib::RawTransaction> {
|
||||||
|
let data = match transaction.transaction() {
|
||||||
|
ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary { data, encoding: ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64 } => {
|
||||||
|
data.as_str()
|
||||||
|
},
|
||||||
|
_ => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let material = match ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64_with_embedded_signature(
|
||||||
|
network,
|
||||||
|
transaction.slot(),
|
||||||
|
transaction.block_time(),
|
||||||
|
data,
|
||||||
|
map_meta(transaction.meta()),
|
||||||
|
map_version(transaction.version()),
|
||||||
|
match transaction.transaction_index() {
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Omitted => ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Null => ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(value) => ksp_raw_transaction_lib::RawTransactionWireField::Value(*value),
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
Ok(material) => material,
|
||||||
|
Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return ksp_raw_transaction_lib::canonicalize_raw_transaction(material).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn raw_from_helius_full(
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
notification: &ksp_onchain_transport_lib::HeliusFullTransactionNotification,
|
||||||
|
) -> std::option::Option<ksp_store_lib::RawTransaction> {
|
||||||
|
let object = match notification.transaction().as_object() {
|
||||||
|
std::option::Option::Some(object) => object,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let encoded = match object.get("transaction") {
|
||||||
|
std::option::Option::Some(value) => match value.as_array() {
|
||||||
|
std::option::Option::Some(encoded) => encoded,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
},
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
if encoded.len() != 2 {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
let encoding = match encoded.get(1) {
|
||||||
|
std::option::Option::Some(value) => match value.as_str() {
|
||||||
|
std::option::Option::Some(encoding) => encoding,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
},
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
if encoding != "base64" {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
let data = match encoded.first() {
|
||||||
|
std::option::Option::Some(value) => match value.as_str() {
|
||||||
|
std::option::Option::Some(data) => data,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
},
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let meta = match object.get("meta") {
|
||||||
|
std::option::Option::None => ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
std::option::Option::Some(serde_json::Value::Null) => ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
|
std::option::Option::Some(value) => ksp_raw_transaction_lib::RawTransactionWireField::Value(value.clone()),
|
||||||
|
};
|
||||||
|
let signature = match ksp_raw_transaction_lib::parse_raw_transaction_signature(notification.signature()) {
|
||||||
|
Ok(signature) => signature,
|
||||||
|
Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let transaction_index = match u32::try_from(notification.transaction_index()) {
|
||||||
|
Ok(transaction_index) => transaction_index,
|
||||||
|
Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
notification.slot(),
|
||||||
|
std::option::Option::None,
|
||||||
|
data,
|
||||||
|
meta,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(transaction_index),
|
||||||
|
);
|
||||||
|
return ksp_raw_transaction_lib::canonicalize_raw_transaction(material).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_standard_block_subscribe_full_base64_matches_http_get_block_raw_v1_for_legacy_transaction() {
|
||||||
|
const HTTP_BODY: &str = concat!(
|
||||||
|
"{\"jsonrpc\":\"2.0\",\"result\":{\"previousBlockhash\":\"previous\",\"blockhash\":\"block\",\"parentSlot\":430000122,",
|
||||||
|
"\"rewards\":[],\"numRewardPartitions\":0,\"blockTime\":1787072400,\"blockHeight\":410000000,",
|
||||||
|
"\"transactions\":[{\"transaction\":[\"AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
|
||||||
|
"AAAAAA\",\"base64\"],\"meta\":{\"err\":null,\"fee\":5000},\"version\":\"legacy\"}]},\"id\":1}",
|
||||||
|
);
|
||||||
|
let (http_url, http_server) = serve_http_once(HTTP_BODY).expect("HTTP parity fixture must start");
|
||||||
|
let http_pool = http_pool_for_url(http_url.as_str()).expect("HTTP parity pool must build");
|
||||||
|
let http_config = ksp_onchain_transport_lib::SolanaGetBlockConfig::new(
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
std::option::Option::Some(false),
|
||||||
|
);
|
||||||
|
let http = http_pool
|
||||||
|
.get_block_observed(&ksp_onchain_transport_lib::HttpRoleName::new("default"), FIXTURE_SLOT, std::option::Option::Some(&http_config))
|
||||||
|
.await
|
||||||
|
.expect("HTTP block parity fixture must decode");
|
||||||
|
let http_block = http.value().as_ref().expect("HTTP parity block must be present").clone();
|
||||||
|
http_server.join().expect("HTTP parity fixture must join").expect("HTTP parity fixture must complete");
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("WS parity listener must bind");
|
||||||
|
let address = listener.local_addr().expect("WS parity listener address must resolve");
|
||||||
|
let ws_block = serde_json::from_str::<serde_json::Value>(HTTP_BODY).expect("HTTP parity JSON must parse")["result"].clone();
|
||||||
|
let server = tokio::spawn(async move {
|
||||||
|
let (stream, _) = match listener.accept().await {
|
||||||
|
Ok(accepted) => accepted,
|
||||||
|
Err(_) => return Err("WS parity server accept failed".to_owned()),
|
||||||
|
};
|
||||||
|
let mut websocket = match tokio_tungstenite::accept_async(stream).await {
|
||||||
|
Ok(websocket) => websocket,
|
||||||
|
Err(_) => return Err("WS parity handshake failed".to_owned()),
|
||||||
|
};
|
||||||
|
let subscribe = match read_ws_request(&mut websocket).await {
|
||||||
|
Ok(subscribe) => subscribe,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
if subscribe.get("method") != std::option::Option::Some(&serde_json::json!("blockSubscribe")) {
|
||||||
|
return Err("WS parity server received unexpected subscribe method".to_owned());
|
||||||
|
}
|
||||||
|
if let Err(error) = send_ws_result(&mut websocket, &subscribe, serde_json::json!(501)).await {
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
if let Err(error) = send_ws_json(
|
||||||
|
&mut websocket,
|
||||||
|
serde_json::json!({
|
||||||
|
"jsonrpc":"2.0",
|
||||||
|
"method":"blockNotification",
|
||||||
|
"params":{"subscription":501,"result":{"context":{"slot":FIXTURE_SLOT},"value":{"slot":FIXTURE_SLOT,"block":ws_block,"err":null}}}
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
let unsubscribe = match read_ws_request(&mut websocket).await {
|
||||||
|
Ok(unsubscribe) => unsubscribe,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
if unsubscribe.get("method") != std::option::Option::Some(&serde_json::json!("blockUnsubscribe")) {
|
||||||
|
return Err("WS parity server received unexpected unsubscribe method".to_owned());
|
||||||
|
}
|
||||||
|
if let Err(error) = send_ws_result(&mut websocket, &unsubscribe, serde_json::json!(true)).await {
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
wait_for_ws_close(&mut websocket).await;
|
||||||
|
return Ok::<(), std::string::String>(());
|
||||||
|
});
|
||||||
|
let endpoint = ws_endpoint(format!("ws://{address}").as_str(), ksp_onchain_transport_lib::WsProtocolKind::SolanaStandard)
|
||||||
|
.expect("standard WS parity endpoint must build");
|
||||||
|
let session = ksp_onchain_transport_lib::SolanaStandardWsSession::connect(endpoint).await.expect("standard WS parity session must connect");
|
||||||
|
let ws_config = ksp_onchain_transport_lib::SolanaBlockSubscribeConfig::new(
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
std::option::Option::Some(false),
|
||||||
|
);
|
||||||
|
let mut subscription = session
|
||||||
|
.block_subscribe(&ksp_onchain_transport_lib::SolanaBlockSubscribeFilter::All, std::option::Option::Some(&ws_config))
|
||||||
|
.await
|
||||||
|
.expect("standard blockSubscribe parity subscription must register");
|
||||||
|
let notification = subscription
|
||||||
|
.recv()
|
||||||
|
.await
|
||||||
|
.expect("standard blockSubscribe parity notification must arrive")
|
||||||
|
.expect("standard blockSubscribe parity notification must decode");
|
||||||
|
let ws_block = notification.value().block().expect("standard blockSubscribe parity block must be present");
|
||||||
|
assert_eq!(ws_block, &http_block);
|
||||||
|
let network = ksp_store_lib::RawNetworkId::new("devnet").expect("parity fixture network must be valid");
|
||||||
|
let http_transaction =
|
||||||
|
http_block.transactions().value().expect("HTTP parity transactions must be present").first().expect("HTTP parity transaction must exist");
|
||||||
|
let ws_transaction = ws_block.transactions().value().expect("WS parity transactions must be present").first().expect("WS parity transaction must exist");
|
||||||
|
let http_raw = raw_from_block_transaction(network.clone(), FIXTURE_SLOT, http_block.block_time(), http_transaction, 0)
|
||||||
|
.expect("HTTP block transaction must canonicalize");
|
||||||
|
let ws_raw = raw_from_block_transaction(network, notification.value().slot(), ws_block.block_time(), ws_transaction, 0)
|
||||||
|
.expect("WS block transaction must canonicalize");
|
||||||
|
assert_eq!(ws_raw.reference(), http_raw.reference());
|
||||||
|
assert_eq!(ws_raw.slot(), http_raw.slot());
|
||||||
|
assert_eq!(ws_raw.block_time(), http_raw.block_time());
|
||||||
|
assert_eq!(ws_raw.payload().bytes(), http_raw.payload().bytes());
|
||||||
|
assert_eq!(ws_raw.payload().content_hash(), http_raw.payload().content_hash());
|
||||||
|
assert!(subscription.unsubscribe().await.expect("standard blockSubscribe parity unsubscribe must complete"));
|
||||||
|
session.close().await.expect("standard WS parity session must close");
|
||||||
|
server.await.expect("standard WS parity server task must join").expect("standard WS parity server must complete");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_helius_full_base64_without_block_time_and_version_requires_http_hydration_for_raw_v1() {
|
||||||
|
const HTTP_BODY: &str = concat!(
|
||||||
|
"{\"jsonrpc\":\"2.0\",\"result\":{\"slot\":430000123,\"blockTime\":1787072400,",
|
||||||
|
"\"transaction\":[\"AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\",\"base64\"],",
|
||||||
|
"\"meta\":{\"err\":null,\"fee\":5000},\"version\":\"legacy\",\"transactionIndex\":0},\"id\":1}",
|
||||||
|
);
|
||||||
|
let (http_url, http_server) = serve_http_once(HTTP_BODY).expect("Helius hydration HTTP fixture must start");
|
||||||
|
let http_pool = http_pool_for_url(http_url.as_str()).expect("Helius hydration HTTP pool must build");
|
||||||
|
let http_config = ksp_onchain_transport_lib::SolanaGetTransactionConfig::new(
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
);
|
||||||
|
let http = http_pool
|
||||||
|
.get_transaction_observed(&ksp_onchain_transport_lib::HttpRoleName::new("default"), ZERO_SIGNATURE_TEXT, std::option::Option::Some(&http_config))
|
||||||
|
.await
|
||||||
|
.expect("Helius hydration HTTP fixture must decode");
|
||||||
|
let http_transaction = http.value().as_ref().expect("Helius hydration HTTP transaction must be present");
|
||||||
|
let network = ksp_store_lib::RawNetworkId::new("devnet").expect("Helius hydration fixture network must be valid");
|
||||||
|
let http_raw = raw_from_confirmed_transaction(network.clone(), http_transaction).expect("hydrated HTTP transaction must canonicalize");
|
||||||
|
http_server.join().expect("Helius hydration HTTP fixture must join").expect("Helius hydration HTTP fixture must complete");
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("Helius parity listener must bind");
|
||||||
|
let address = listener.local_addr().expect("Helius parity listener address must resolve");
|
||||||
|
let server = tokio::spawn(async move {
|
||||||
|
let (stream, _) = match listener.accept().await {
|
||||||
|
Ok(accepted) => accepted,
|
||||||
|
Err(_) => return Err("Helius parity server accept failed".to_owned()),
|
||||||
|
};
|
||||||
|
let mut websocket = match tokio_tungstenite::accept_async(stream).await {
|
||||||
|
Ok(websocket) => websocket,
|
||||||
|
Err(_) => return Err("Helius parity handshake failed".to_owned()),
|
||||||
|
};
|
||||||
|
let subscribe = match read_ws_request(&mut websocket).await {
|
||||||
|
Ok(subscribe) => subscribe,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
if subscribe.get("method") != std::option::Option::Some(&serde_json::json!("transactionSubscribe")) {
|
||||||
|
return Err("Helius parity server received unexpected subscribe method".to_owned());
|
||||||
|
}
|
||||||
|
if let Err(error) = send_ws_result(&mut websocket, &subscribe, serde_json::json!(601)).await {
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
if let Err(error) = send_ws_json(
|
||||||
|
&mut websocket,
|
||||||
|
serde_json::json!({
|
||||||
|
"jsonrpc":"2.0",
|
||||||
|
"method":"transactionNotification",
|
||||||
|
"params":{
|
||||||
|
"subscription":601,
|
||||||
|
"result":{
|
||||||
|
"transaction":{"transaction":[ZERO_TRANSACTION_BASE64,"base64"],"meta":{"err":null,"fee":5000}},
|
||||||
|
"signature":ZERO_SIGNATURE_TEXT,
|
||||||
|
"slot":FIXTURE_SLOT,
|
||||||
|
"transactionIndex":0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
let unsubscribe = match read_ws_request(&mut websocket).await {
|
||||||
|
Ok(unsubscribe) => unsubscribe,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
if unsubscribe.get("method") != std::option::Option::Some(&serde_json::json!("transactionUnsubscribe")) {
|
||||||
|
return Err("Helius parity server received unexpected unsubscribe method".to_owned());
|
||||||
|
}
|
||||||
|
if let Err(error) = send_ws_result(&mut websocket, &unsubscribe, serde_json::json!(true)).await {
|
||||||
|
return Err(error);
|
||||||
|
}
|
||||||
|
wait_for_ws_close(&mut websocket).await;
|
||||||
|
return Ok::<(), std::string::String>(());
|
||||||
|
});
|
||||||
|
let endpoint = ws_endpoint(format!("ws://{address}").as_str(), ksp_onchain_transport_lib::WsProtocolKind::HeliusLaserStream)
|
||||||
|
.expect("Helius parity endpoint must build");
|
||||||
|
let session = ksp_onchain_transport_lib::HeliusLaserStreamWsSession::connect(endpoint).await.expect("Helius parity session must connect");
|
||||||
|
let request = ksp_onchain_transport_lib::HeliusTransactionSubscribeRequest::new(
|
||||||
|
ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter::default(),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::HeliusTransactionSubscribeOptions::new(
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::HeliusTransactionSubscribeEncoding::Base64),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionDetails::Full),
|
||||||
|
std::option::Option::Some(false),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
let mut subscription = session.transaction_subscribe(&request).await.expect("Helius parity transaction subscription must register");
|
||||||
|
let notification = subscription.recv().await.expect("Helius parity notification must arrive").expect("Helius parity notification must decode");
|
||||||
|
assert!(matches!(¬ification, ksp_onchain_transport_lib::HeliusTransactionNotification::Full(_)));
|
||||||
|
let full = match notification {
|
||||||
|
ksp_onchain_transport_lib::HeliusTransactionNotification::Full(full) => full,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
let helius_raw = raw_from_helius_full(network, &full).expect("best-effort Helius full projection must canonicalize for negative parity proof");
|
||||||
|
assert_eq!(helius_raw.reference(), http_raw.reference());
|
||||||
|
assert_eq!(helius_raw.slot(), http_raw.slot());
|
||||||
|
assert!(helius_raw.block_time().is_none());
|
||||||
|
assert_eq!(http_raw.block_time().map(|value| return value.unix_millis()), std::option::Option::Some(FIXTURE_BLOCK_TIME_MILLIS),);
|
||||||
|
assert_ne!(helius_raw.block_time(), http_raw.block_time());
|
||||||
|
assert_ne!(helius_raw.payload().bytes(), http_raw.payload().bytes());
|
||||||
|
assert_ne!(helius_raw.payload().content_hash(), http_raw.payload().content_hash());
|
||||||
|
assert!(subscription.unsubscribe().await.expect("Helius parity transaction unsubscribe must complete"));
|
||||||
|
session.close().await.expect("Helius parity session must close");
|
||||||
|
server.await.expect("Helius parity server task must join").expect("Helius parity server must complete");
|
||||||
|
}
|
||||||
471
crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs
Normal file
471
crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs
Normal file
@@ -0,0 +1,471 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
//! Deterministic local Yellowstone gRPC parity canary for source-neutral RAW transaction v1 projection.
|
||||||
|
|
||||||
|
struct FixtureStream<T> {
|
||||||
|
receiver: tokio::sync::mpsc::Receiver<T>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<T> FixtureStream<T> {
|
||||||
|
fn new(receiver: tokio::sync::mpsc::Receiver<T>) -> Self {
|
||||||
|
return Self { receiver };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<T> futures_util::Stream for FixtureStream<T> {
|
||||||
|
type Item = T;
|
||||||
|
|
||||||
|
fn poll_next(self: std::pin::Pin<&mut Self>, context: &mut std::task::Context<'_>) -> std::task::Poll<std::option::Option<Self::Item>> {
|
||||||
|
return self.get_mut().receiver.poll_recv(context);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone)]
|
||||||
|
struct FixtureGeyser;
|
||||||
|
|
||||||
|
#[allow(clippy::implicit_return)] // tonic::async_trait generates async wrapper tails outside the authored fixture bodies.
|
||||||
|
#[tonic::async_trait]
|
||||||
|
impl yellowstone_grpc_proto::geyser::geyser_server::Geyser for FixtureGeyser {
|
||||||
|
type SubscribeStream = FixtureStream<std::result::Result<yellowstone_grpc_proto::geyser::SubscribeUpdate, tonic::Status>>;
|
||||||
|
|
||||||
|
async fn subscribe(
|
||||||
|
&self,
|
||||||
|
request: tonic::Request<tonic::Streaming<yellowstone_grpc_proto::geyser::SubscribeRequest>>,
|
||||||
|
) -> std::result::Result<tonic::Response<Self::SubscribeStream>, tonic::Status> {
|
||||||
|
let mut inbound = request.into_inner();
|
||||||
|
let (sender, receiver) = tokio::sync::mpsc::channel(4);
|
||||||
|
tokio::spawn(async move {
|
||||||
|
let initial = inbound.message().await;
|
||||||
|
if !matches!(initial, std::result::Result::Ok(std::option::Option::Some(_))) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if sender.send(std::result::Result::Ok(transaction_update())).await.is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let _ = sender.send(std::result::Result::Ok(block_update())).await;
|
||||||
|
});
|
||||||
|
return std::result::Result::Ok(tonic::Response::new(FixtureStream::new(receiver)));
|
||||||
|
}
|
||||||
|
|
||||||
|
type SubscribeDeshredStream = futures_util::stream::Empty<std::result::Result<yellowstone_grpc_proto::geyser::SubscribeUpdateDeshred, tonic::Status>>;
|
||||||
|
|
||||||
|
async fn subscribe_deshred(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<tonic::Streaming<yellowstone_grpc_proto::geyser::SubscribeDeshredRequest>>,
|
||||||
|
) -> std::result::Result<tonic::Response<Self::SubscribeDeshredStream>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("deshred is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
type SubscribeGossipStream = futures_util::stream::Empty<std::result::Result<yellowstone_grpc_proto::geyser::SubscribeUpdateGossip, tonic::Status>>;
|
||||||
|
|
||||||
|
async fn subscribe_gossip(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::SubscribeGossipRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<Self::SubscribeGossipStream>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("gossip is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn subscribe_replay_info(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::SubscribeReplayInfoRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<yellowstone_grpc_proto::geyser::SubscribeReplayInfoResponse>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("replay info is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn ping(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::PingRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<yellowstone_grpc_proto::geyser::PongResponse>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("unary is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn get_latest_blockhash(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::GetLatestBlockhashRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<yellowstone_grpc_proto::geyser::GetLatestBlockhashResponse>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("unary is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn get_block_height(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::GetBlockHeightRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<yellowstone_grpc_proto::geyser::GetBlockHeightResponse>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("unary is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn get_slot(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::GetSlotRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<yellowstone_grpc_proto::geyser::GetSlotResponse>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("unary is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn is_blockhash_valid(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::IsBlockhashValidRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<yellowstone_grpc_proto::geyser::IsBlockhashValidResponse>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("unary is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn get_version(
|
||||||
|
&self,
|
||||||
|
_request: tonic::Request<yellowstone_grpc_proto::geyser::GetVersionRequest>,
|
||||||
|
) -> std::result::Result<tonic::Response<yellowstone_grpc_proto::geyser::GetVersionResponse>, tonic::Status> {
|
||||||
|
return std::result::Result::Err(tonic::Status::unimplemented("unary is outside the pre.006 fixture"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FixtureServer {
|
||||||
|
endpoint_url: std::string::String,
|
||||||
|
shutdown: std::option::Option<tokio::sync::oneshot::Sender<()>>,
|
||||||
|
task: tokio::task::JoinHandle<()>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FixtureServer {
|
||||||
|
async fn start() -> std::result::Result<Self, &'static str> {
|
||||||
|
let bind_address = match "127.0.0.1:0".parse::<std::net::SocketAddr>() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err("fixture bind address must parse"),
|
||||||
|
};
|
||||||
|
let incoming = match tonic::transport::server::TcpIncoming::bind(bind_address) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err("fixture gRPC listener must bind"),
|
||||||
|
};
|
||||||
|
let local_address = match incoming.local_addr() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err("fixture gRPC listener must expose local address"),
|
||||||
|
};
|
||||||
|
let (shutdown, shutdown_receiver) = tokio::sync::oneshot::channel();
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
let service = yellowstone_grpc_proto::geyser::geyser_server::GeyserServer::new(FixtureGeyser);
|
||||||
|
let result = tonic::transport::Server::builder()
|
||||||
|
.serve_with_incoming_shutdown(service, incoming, async move {
|
||||||
|
let _ = shutdown_receiver.await;
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
});
|
||||||
|
return std::result::Result::Ok(Self { endpoint_url: format!("http://{local_address}"), shutdown: std::option::Option::Some(shutdown), task });
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn stop(mut self) {
|
||||||
|
if let std::option::Option::Some(shutdown) = self.shutdown.take() {
|
||||||
|
let _ = shutdown.send(());
|
||||||
|
}
|
||||||
|
let result = self.task.await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn transaction_info() -> yellowstone_grpc_proto::geyser::SubscribeUpdateTransactionInfo {
|
||||||
|
return yellowstone_grpc_proto::geyser::SubscribeUpdateTransactionInfo {
|
||||||
|
signature: vec![9_u8; 64],
|
||||||
|
is_vote: false,
|
||||||
|
transaction: std::option::Option::Some(yellowstone_grpc_proto::solana::storage::confirmed_block::Transaction {
|
||||||
|
signatures: vec![vec![9_u8; 64]],
|
||||||
|
message: std::option::Option::Some(yellowstone_grpc_proto::solana::storage::confirmed_block::Message {
|
||||||
|
header: std::option::Option::Some(yellowstone_grpc_proto::solana::storage::confirmed_block::MessageHeader {
|
||||||
|
num_required_signatures: 1,
|
||||||
|
num_readonly_signed_accounts: 0,
|
||||||
|
num_readonly_unsigned_accounts: 1,
|
||||||
|
}),
|
||||||
|
account_keys: vec![vec![1_u8; 32], vec![2_u8; 32]],
|
||||||
|
recent_blockhash: vec![3_u8; 32],
|
||||||
|
instructions: vec![yellowstone_grpc_proto::solana::storage::confirmed_block::CompiledInstruction {
|
||||||
|
program_id_index: 1,
|
||||||
|
accounts: vec![0_u8],
|
||||||
|
data: vec![4_u8, 5, 6],
|
||||||
|
}],
|
||||||
|
versioned: true,
|
||||||
|
address_table_lookups: vec![],
|
||||||
|
config: std::option::Option::Some(yellowstone_grpc_proto::solana::storage::confirmed_block::TransactionConfig {
|
||||||
|
priority_fee: std::option::Option::Some(7),
|
||||||
|
compute_unit_limit: std::option::Option::Some(8),
|
||||||
|
loaded_accounts_data_size_limit: std::option::Option::Some(9),
|
||||||
|
heap_size: std::option::Option::Some(32_768),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
meta: std::option::Option::Some(yellowstone_grpc_proto::solana::storage::confirmed_block::TransactionStatusMeta {
|
||||||
|
fee: 5_000,
|
||||||
|
pre_balances: vec![100, 200],
|
||||||
|
post_balances: vec![90, 210],
|
||||||
|
compute_units_consumed: std::option::Option::Some(123),
|
||||||
|
cost_units: std::option::Option::Some(456),
|
||||||
|
..std::default::Default::default()
|
||||||
|
}),
|
||||||
|
index: 3,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn transaction_update() -> yellowstone_grpc_proto::geyser::SubscribeUpdate {
|
||||||
|
return yellowstone_grpc_proto::geyser::SubscribeUpdate {
|
||||||
|
filters: vec!["transactions-pre-006".to_owned()],
|
||||||
|
update_oneof: std::option::Option::Some(yellowstone_grpc_proto::geyser::subscribe_update::UpdateOneof::Transaction(
|
||||||
|
yellowstone_grpc_proto::geyser::SubscribeUpdateTransaction { transaction: std::option::Option::Some(transaction_info()), slot: 42 },
|
||||||
|
)),
|
||||||
|
created_at: std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn block_update() -> yellowstone_grpc_proto::geyser::SubscribeUpdate {
|
||||||
|
return yellowstone_grpc_proto::geyser::SubscribeUpdate {
|
||||||
|
filters: vec!["blocks-pre-006".to_owned()],
|
||||||
|
update_oneof: std::option::Option::Some(yellowstone_grpc_proto::geyser::subscribe_update::UpdateOneof::Block(
|
||||||
|
yellowstone_grpc_proto::geyser::SubscribeUpdateBlock {
|
||||||
|
slot: 42,
|
||||||
|
blockhash: ksp_core_lib::Pubkey::new_from_array([21_u8; 32]).to_string(),
|
||||||
|
rewards: std::option::Option::None,
|
||||||
|
block_time: std::option::Option::Some(yellowstone_grpc_proto::solana::storage::confirmed_block::UnixTimestamp { timestamp: 1_787_104_000 }),
|
||||||
|
block_height: std::option::Option::Some(yellowstone_grpc_proto::solana::storage::confirmed_block::BlockHeight { block_height: 41 }),
|
||||||
|
transactions: vec![transaction_info()],
|
||||||
|
parent_slot: 41,
|
||||||
|
parent_blockhash: ksp_core_lib::Pubkey::new_from_array([22_u8; 32]).to_string(),
|
||||||
|
executed_transaction_count: 1,
|
||||||
|
updated_account_count: 0,
|
||||||
|
accounts: vec![],
|
||||||
|
entries_count: 0,
|
||||||
|
entries: vec![],
|
||||||
|
},
|
||||||
|
)),
|
||||||
|
created_at: std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fixture_settings(url: &str) -> ksp_core_lib::Result<ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings> {
|
||||||
|
let endpoint_url = match ksp_onchain_transport_lib::YellowstoneGrpcEndpointUrl::parse(url) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_onchain_transport_lib::YellowstoneGrpcEndpointSettings::new(
|
||||||
|
"pre-006-fixture",
|
||||||
|
true,
|
||||||
|
ksp_onchain_transport_lib::YellowstoneGrpcProviderName::new("fixture-provider"),
|
||||||
|
ksp_onchain_transport_lib::YellowstoneGrpcClusterName::new("devnet"),
|
||||||
|
endpoint_url,
|
||||||
|
ksp_onchain_transport_lib::YellowstoneGrpcSessionSettings::default(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn project_wire(
|
||||||
|
transaction: &ksp_onchain_transport_lib::YellowstoneStoredTransaction,
|
||||||
|
) -> ksp_core_lib::Result<ksp_raw_transaction_lib::RawSolanaTransactionWire> {
|
||||||
|
let message = transaction.message();
|
||||||
|
let header = message.header();
|
||||||
|
let required = match u8::try_from(header.num_required_signatures()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(projection_error("num_required_signatures")),
|
||||||
|
};
|
||||||
|
let readonly_signed = match u8::try_from(header.num_readonly_signed_accounts()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(projection_error("num_readonly_signed_accounts")),
|
||||||
|
};
|
||||||
|
let readonly_unsigned = match u8::try_from(header.num_readonly_unsigned_accounts()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(projection_error("num_readonly_unsigned_accounts")),
|
||||||
|
};
|
||||||
|
let version = if message.config().is_some() {
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageVersion::V1
|
||||||
|
} else if message.versioned() {
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageVersion::V0
|
||||||
|
} else {
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageVersion::Legacy
|
||||||
|
};
|
||||||
|
let signatures = transaction.signatures().iter().map(|value| return *value.as_bytes()).collect();
|
||||||
|
let account_keys = message.account_keys().iter().map(solana_pubkey_bytes).collect();
|
||||||
|
let mut instructions = std::vec::Vec::with_capacity(message.instructions().len());
|
||||||
|
for instruction in message.instructions() {
|
||||||
|
let program_id_index = match u8::try_from(instruction.program_id_index()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(projection_error("program_id_index")),
|
||||||
|
};
|
||||||
|
instructions.push(ksp_raw_transaction_lib::RawSolanaCompiledInstruction::new(
|
||||||
|
program_id_index,
|
||||||
|
instruction.accounts().to_vec(),
|
||||||
|
instruction.data().to_vec(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let address_table_lookups = message
|
||||||
|
.address_table_lookups()
|
||||||
|
.iter()
|
||||||
|
.map(|lookup| {
|
||||||
|
return ksp_raw_transaction_lib::RawSolanaAddressTableLookup::new(
|
||||||
|
lookup.account_key().to_bytes(),
|
||||||
|
lookup.writable_indexes().to_vec(),
|
||||||
|
lookup.readonly_indexes().to_vec(),
|
||||||
|
);
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
let config = message.config().map(|value| {
|
||||||
|
return ksp_raw_transaction_lib::RawSolanaTransactionConfig::new(
|
||||||
|
value.priority_fee(),
|
||||||
|
value.compute_unit_limit(),
|
||||||
|
value.loaded_accounts_data_size_limit(),
|
||||||
|
value.heap_size(),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
let raw_message = ksp_raw_transaction_lib::RawSolanaTransactionMessage::new(
|
||||||
|
version,
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageHeader::new(required, readonly_signed, readonly_unsigned),
|
||||||
|
account_keys,
|
||||||
|
*message.recent_blockhash().as_bytes(),
|
||||||
|
instructions,
|
||||||
|
address_table_lookups,
|
||||||
|
config,
|
||||||
|
);
|
||||||
|
return std::result::Result::Ok(ksp_raw_transaction_lib::RawSolanaTransactionWire::new(signatures, raw_message));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn solana_pubkey_bytes(value: &ksp_core_lib::Pubkey) -> [u8; 32] {
|
||||||
|
return value.to_bytes();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn projection_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(ksp_core_lib::ErrorCode::new("test", "yellowstone_raw_projection"), "Yellowstone fixture field does not fit common wire")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn error_code_text(error: &ksp_core_lib::Error) -> std::string::String {
|
||||||
|
return format!("{}.{}", error.code().domain(), error.code().code());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn expected_v1_wire() -> ksp_raw_transaction_lib::RawSolanaTransactionWire {
|
||||||
|
let message = ksp_raw_transaction_lib::RawSolanaTransactionMessage::new(
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageVersion::V1,
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageHeader::new(1, 0, 1),
|
||||||
|
vec![[1_u8; 32], [2_u8; 32]],
|
||||||
|
[3_u8; 32],
|
||||||
|
vec![ksp_raw_transaction_lib::RawSolanaCompiledInstruction::new(1, vec![0_u8], vec![4_u8, 5, 6])],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::Some(ksp_raw_transaction_lib::RawSolanaTransactionConfig::new(
|
||||||
|
std::option::Option::Some(7),
|
||||||
|
std::option::Option::Some(8),
|
||||||
|
std::option::Option::Some(9),
|
||||||
|
std::option::Option::Some(32_768),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
return ksp_raw_transaction_lib::RawSolanaTransactionWire::new(vec![[9_u8; 64]], message);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fail_test(message: std::string::String) {
|
||||||
|
assert!(message.is_empty(), "{message}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_006_yellowstone_transaction_and_block_preserve_exact_v1_wire_and_block_time() {
|
||||||
|
let server = match FixtureServer::start().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert!(error.is_empty(), "fixture server start failed: {error}");
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let settings = match fixture_settings(server.endpoint_url.as_str()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("fixture settings failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let channel = match ksp_onchain_transport_lib::YellowstoneGrpcChannel::connect(&settings).await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("fixture channel connect failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let request = ksp_onchain_transport_lib::YellowstoneSubscribeRequest::new();
|
||||||
|
let mut session = match channel.open_standard_subscribe(request).await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("fixture subscribe failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let expected = match ksp_raw_transaction_lib::serialize_solana_transaction_wire(&expected_v1_wire()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("expected V1 wire serialization failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let transaction = match session.next_update().await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => {
|
||||||
|
fail_test("transaction update must be present".to_owned());
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("transaction update decode failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let transaction = match transaction {
|
||||||
|
ksp_onchain_transport_lib::YellowstoneSubscribeUpdate::Transaction(value) => value,
|
||||||
|
_ => {
|
||||||
|
fail_test("first Yellowstone fixture update must be a transaction".to_owned());
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
assert_eq!(transaction.slot(), 42);
|
||||||
|
assert_eq!(transaction.transaction().index(), 3);
|
||||||
|
assert_eq!(transaction.transaction().meta().fee(), 5_000);
|
||||||
|
let transaction_wire = match project_wire(transaction.transaction().transaction()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("transaction projection failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let transaction_bytes = match ksp_raw_transaction_lib::serialize_solana_transaction_wire(&transaction_wire) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("transaction V1 wire serialization failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
assert_eq!(transaction_bytes, expected);
|
||||||
|
let block = match session.next_update().await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => {
|
||||||
|
fail_test("block update must be present".to_owned());
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("block update decode failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let block = match block {
|
||||||
|
ksp_onchain_transport_lib::YellowstoneSubscribeUpdate::Block(value) => value,
|
||||||
|
_ => {
|
||||||
|
fail_test("second Yellowstone fixture update must be a block".to_owned());
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
assert_eq!(block.slot(), 42);
|
||||||
|
assert_eq!(block.block_time(), std::option::Option::Some(1_787_104_000));
|
||||||
|
assert_eq!(block.transactions().len(), 1);
|
||||||
|
assert_eq!(block.transactions()[0].index(), 3);
|
||||||
|
let block_wire = match project_wire(block.transactions()[0].transaction()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("block transaction projection failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let block_bytes = match ksp_raw_transaction_lib::serialize_solana_transaction_wire(&block_wire) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
fail_test(format!("block V1 wire serialization failed: {}", error_code_text(&error)));
|
||||||
|
return;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
assert_eq!(block_bytes, expected);
|
||||||
|
assert_eq!(block_bytes, transaction_bytes);
|
||||||
|
assert!(transaction.transaction().meta().error().is_none());
|
||||||
|
assert_eq!(block.transactions()[0].meta().fee(), transaction.transaction().meta().fee());
|
||||||
|
assert!(session.close().await.is_ok());
|
||||||
|
server.stop().await;
|
||||||
|
}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-job-backfill-lib/unit_tests/conversion.rs
|
// file: crates/ksp-job-backfill-lib/unit_tests/conversion.rs
|
||||||
// version: 2
|
// version: 3
|
||||||
|
|
||||||
fn signature_text() -> std::option::Option<crate::BackfillSignature> {
|
fn signature_text() -> std::option::Option<crate::BackfillSignature> {
|
||||||
return match crate::BackfillSignature::new("1".repeat(64)) {
|
return match crate::BackfillSignature::new("1".repeat(64)) {
|
||||||
@@ -46,6 +46,14 @@ fn candidate(network: &str, signature: crate::BackfillSignature) -> std::option:
|
|||||||
return std::option::Option::Some(crate::BackfillCandidate::new(crate::BackfillCandidateIdentity::new(network, signature), std::option::Option::Some(42)));
|
return std::option::Option::Some(crate::BackfillCandidate::new(crate::BackfillCandidateIdentity::new(network, signature), std::option::Option::Some(42)));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn raw_reference(network: &str) -> std::option::Option<ksp_store_lib::RawTransactionReference> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_store_lib::RawTransactionReference::new(network, ksp_store_lib::RawTransactionSignature::new([0_u8; 64])));
|
||||||
|
}
|
||||||
|
|
||||||
fn received_at() -> std::option::Option<ksp_store_lib::RawTimestamp> {
|
fn received_at() -> std::option::Option<ksp_store_lib::RawTimestamp> {
|
||||||
return match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_001_000) {
|
return match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_001_000) {
|
||||||
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
@@ -169,21 +177,26 @@ fn pre_006_wire_omission_and_null_produce_distinct_canonical_bytes() {
|
|||||||
let version_null = ksp_onchain_transport_lib::SolanaWireField::Null;
|
let version_null = ksp_onchain_transport_lib::SolanaWireField::Null;
|
||||||
let transaction_index_omitted = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
let transaction_index_omitted = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
let transaction_index_null = ksp_onchain_transport_lib::SolanaWireField::Null;
|
let transaction_index_null = ksp_onchain_transport_lib::SolanaWireField::Null;
|
||||||
let omitted = super::canonical_payload_bytes(&fields(&transaction, &meta_omitted, &version_omitted, &transaction_index_omitted, std::option::Option::None));
|
let reference = match raw_reference("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let omitted =
|
||||||
|
super::canonical_transaction(&reference, fields(&transaction, &meta_omitted, &version_omitted, &transaction_index_omitted, std::option::Option::None));
|
||||||
assert!(omitted.is_ok());
|
assert!(omitted.is_ok());
|
||||||
let omitted = match omitted {
|
let omitted = match omitted {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => return,
|
std::result::Result::Err(_) => return,
|
||||||
};
|
};
|
||||||
assert_eq!(omitted, b"{\"transaction\":[\"AQID\",\"base64\"]}");
|
assert_eq!(omitted.payload().bytes(), b"{\"transaction\":[\"AQID\",\"base64\"]}");
|
||||||
let nulls = super::canonical_payload_bytes(&fields(&transaction, &meta_null, &version_null, &transaction_index_null, std::option::Option::None));
|
let nulls = super::canonical_transaction(&reference, fields(&transaction, &meta_null, &version_null, &transaction_index_null, std::option::Option::None));
|
||||||
assert!(nulls.is_ok());
|
assert!(nulls.is_ok());
|
||||||
let nulls = match nulls {
|
let nulls = match nulls {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(_) => return,
|
std::result::Result::Err(_) => return,
|
||||||
};
|
};
|
||||||
assert_eq!(nulls, b"{\"transaction\":[\"AQID\",\"base64\"],\"meta\":null,\"version\":null,\"transactionIndex\":null}");
|
assert_eq!(nulls.payload().bytes(), b"{\"transaction\":[\"AQID\",\"base64\"],\"meta\":null,\"version\":null,\"transactionIndex\":null}");
|
||||||
assert_ne!(omitted, nulls);
|
assert_ne!(omitted.payload().bytes(), nulls.payload().bytes());
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -198,8 +211,12 @@ fn pre_006_non_base64_transaction_shapes_are_rejected() {
|
|||||||
let meta = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
let meta = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
let version = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
let version = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
let transaction_index = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
let transaction_index = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let reference = match raw_reference("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
for transaction in [&base58, &legacy, &json] {
|
for transaction in [&base58, &legacy, &json] {
|
||||||
let result = super::canonical_payload_bytes(&fields(transaction, &meta, &version, &transaction_index, std::option::Option::None));
|
let result = super::canonical_transaction(&reference, fields(transaction, &meta, &version, &transaction_index, std::option::Option::None));
|
||||||
assert!(result.is_err());
|
assert!(result.is_err());
|
||||||
if let std::result::Result::Err(error) = result {
|
if let std::result::Result::Err(error) = result {
|
||||||
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID);
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID);
|
||||||
@@ -210,12 +227,26 @@ fn pre_006_non_base64_transaction_shapes_are_rejected() {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pre_006_negative_and_unrepresentable_block_times_are_terminal_conversion_errors() {
|
fn pre_006_negative_and_unrepresentable_block_times_are_terminal_conversion_errors() {
|
||||||
let negative = super::convert_block_time(std::option::Option::Some(-1));
|
let reference = match raw_reference("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let transaction = ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary {
|
||||||
|
data: "AQID".to_owned(),
|
||||||
|
encoding: ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64,
|
||||||
|
};
|
||||||
|
let meta = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let version = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let transaction_index = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let negative = super::canonical_transaction(&reference, fields(&transaction, &meta, &version, &transaction_index, std::option::Option::Some(-1)));
|
||||||
assert!(negative.is_err());
|
assert!(negative.is_err());
|
||||||
let oversized = super::convert_block_time(std::option::Option::Some(i64::MAX));
|
let oversized = super::canonical_transaction(&reference, fields(&transaction, &meta, &version, &transaction_index, std::option::Option::Some(i64::MAX)));
|
||||||
assert!(oversized.is_err());
|
assert!(oversized.is_err());
|
||||||
let absent = super::convert_block_time(std::option::Option::None);
|
let absent = super::canonical_transaction(&reference, fields(&transaction, &meta, &version, &transaction_index, std::option::Option::None));
|
||||||
assert!(matches!(absent, std::result::Result::Ok(std::option::Option::None)));
|
assert!(absent.is_ok());
|
||||||
|
if let std::result::Result::Ok(absent) = absent {
|
||||||
|
assert!(absent.block_time().is_none());
|
||||||
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -242,6 +273,12 @@ fn pre_006_observation_key_is_deterministic_and_endpoint_specific() {
|
|||||||
let other_endpoint = super::observation_key(&request, &reference, "provider", "endpoint-b");
|
let other_endpoint = super::observation_key(&request, &reference, "provider", "endpoint-b");
|
||||||
let other_provider = super::observation_key(&request, &reference, "provider-2", "endpoint-a");
|
let other_provider = super::observation_key(&request, &reference, "provider-2", "endpoint-a");
|
||||||
assert_eq!(first, same);
|
assert_eq!(first, same);
|
||||||
|
assert_eq!(
|
||||||
|
first.as_bytes(),
|
||||||
|
&[
|
||||||
|
184, 85, 15, 15, 33, 101, 243, 112, 222, 145, 139, 212, 251, 14, 199, 130, 57, 26, 253, 221, 184, 140, 87, 240, 16, 116, 62, 0, 11, 19, 103, 2
|
||||||
|
]
|
||||||
|
);
|
||||||
assert_ne!(first, other_endpoint);
|
assert_ne!(first, other_endpoint);
|
||||||
assert_ne!(first, other_provider);
|
assert_ne!(first, other_provider);
|
||||||
return;
|
return;
|
||||||
@@ -294,6 +331,15 @@ fn pre_006_missing_outcome_contains_only_network_scoped_reference() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_backfill_uses_common_raw_v1_contract_without_changing_frozen_identity() {
|
||||||
|
assert_eq!(crate::RAW_TRANSACTION_FORMAT_ID, ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
assert_eq!(crate::RAW_TRANSACTION_FORMAT_VERSION, ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_VERSION);
|
||||||
|
assert_eq!(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES, ksp_raw_transaction_lib::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES);
|
||||||
|
assert_eq!(crate::MAX_BACKFILL_SIGNATURE_TEXT_BYTES, ksp_raw_transaction_lib::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pre_006_fix_001_raw_acquisition_uses_one_private_indirection() {
|
fn pre_006_fix_001_raw_acquisition_uses_one_private_indirection() {
|
||||||
assert_eq!(std::mem::size_of::<crate::BackfillRawAcquisition>(), std::mem::size_of::<usize>(),);
|
assert_eq!(std::mem::size_of::<crate::BackfillRawAcquisition>(), std::mem::size_of::<usize>(),);
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: crates/ksp-onchain-transport-lib/USAGE.md -->
|
<!-- file: crates/ksp-onchain-transport-lib/USAGE.md -->
|
||||||
<!-- version: 24 -->
|
<!-- version: 25 -->
|
||||||
|
|
||||||
# Utilisation de `ksp-onchain-transport-lib`
|
# Utilisation de `ksp-onchain-transport-lib`
|
||||||
|
|
||||||
@@ -449,7 +449,9 @@ let inflation_rate = pool.get_inflation_rate(&role).await;
|
|||||||
let stake_minimum = pool.get_stake_minimum_delegation(&role, Some(&context)).await;
|
let stake_minimum = pool.get_stake_minimum_delegation(&role, Some(&context)).await;
|
||||||
```
|
```
|
||||||
|
|
||||||
`getBlock` possède également une forme bare-encoding legacy séparée et deprecated. Les valeurs Economics restent celles du runtime : le consumer ne doit pas supposer localement un taux d'inflation ou un minimum de délégation constant.
|
`getBlock` possède également une forme bare-encoding legacy séparée et deprecated. Pour les consumers qui doivent conserver la provenance exacte d’un pool multi-endpoint, `get_block_observed(...)` exécute la même forme moderne et retourne la valeur typée avec le nom sûr de l’endpoint et le provider réellement gagnants après sélection/retry/reroute. Cette projection n’expose ni URL, ni headers, ni body HTTP brut. Une réponse RPC `null` reste `None` dans la valeur observée.
|
||||||
|
|
||||||
|
Lorsque `transactionDetails = full` et `encoding = base64`, chaque `SolanaBlockTransaction` conserve le transaction wire Base64, `meta` et `version`. Le consumer peut alors projeter ces DTOs vers sa couche métier sans faire dépendre Transport d’une canonicalisation RAW particulière. Les valeurs Economics restent celles du runtime : le consumer ne doit pas supposer localement un taux d'inflation ou un minimum de délégation constant.
|
||||||
|
|
||||||
## 6. Exécution JSON-RPC standard générique
|
## 6. Exécution JSON-RPC standard générique
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
{"jsonrpc":"2.0","result":{"previousBlockhash":"previous-blockhash-fixture","blockhash":"blockhash-fixture","parentSlot":430000122,"rewards":[],"numRewardPartitions":0,"blockTime":1787072400,"blockHeight":410000000,"transactions":[{"transaction":["AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA","base64"],"meta":{"err":null,"fee":5000},"version":"legacy"}]},"id":1}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/grpc_stream.rs
|
// file: crates/ksp-onchain-transport-lib/src/grpc_stream.rs
|
||||||
// version: 3
|
// version: 4
|
||||||
|
|
||||||
use tonic_prost::prost::Message; // rust-rules: trait-import
|
use tonic_prost::prost::Message; // rust-rules: trait-import
|
||||||
|
|
||||||
@@ -39,6 +39,43 @@ pub struct YellowstoneGrpcSubscribeSnapshot {
|
|||||||
terminal_error_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
terminal_error_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Cloneable latest-value observer for one standard Yellowstone Subscribe session snapshot.
|
||||||
|
///
|
||||||
|
/// The observer exposes only the already-safe transport snapshot and keeps the internal Tokio watch channel private. Cloning the observer does not duplicate
|
||||||
|
/// the gRPC stream, request state or reconnect actor.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
receiver: tokio::sync::watch::Receiver<crate::YellowstoneGrpcSubscribeSnapshot>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
fn new(receiver: tokio::sync::watch::Receiver<crate::YellowstoneGrpcSubscribeSnapshot>) -> Self {
|
||||||
|
return Self { receiver };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the current safe reconnect/replay snapshot without waiting for another actor transition.
|
||||||
|
#[must_use]
|
||||||
|
pub fn current(&self) -> crate::YellowstoneGrpcSubscribeSnapshot {
|
||||||
|
return *self.receiver.borrow();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits for one newer safe reconnect/replay snapshot.
|
||||||
|
///
|
||||||
|
/// `None` means the owning Subscribe actor dropped the latest-value publisher and no further snapshot can arrive.
|
||||||
|
pub async fn wait_for_change(&mut self) -> std::option::Option<crate::YellowstoneGrpcSubscribeSnapshot> {
|
||||||
|
return match self.receiver.changed().await {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(*self.receiver.borrow_and_update()),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_struct("YellowstoneGrpcSubscribeSnapshotSource").field("current", &self.current()).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl YellowstoneGrpcSubscribeSnapshot {
|
impl YellowstoneGrpcSubscribeSnapshot {
|
||||||
const fn new(initial_from_slot: std::option::Option<u64>) -> Self {
|
const fn new(initial_from_slot: std::option::Option<u64>) -> Self {
|
||||||
return Self {
|
return Self {
|
||||||
@@ -154,6 +191,12 @@ impl SolanaYellowstoneGrpcSubscribeSession {
|
|||||||
return *self.snapshot_rx.borrow();
|
return *self.snapshot_rx.borrow();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Returns one cloneable latest-value observer for reconnect/replay snapshot transitions.
|
||||||
|
#[must_use]
|
||||||
|
pub fn snapshot_source(&self) -> crate::YellowstoneGrpcSubscribeSnapshotSource {
|
||||||
|
return crate::YellowstoneGrpcSubscribeSnapshotSource::new(self.snapshot_rx.clone());
|
||||||
|
}
|
||||||
|
|
||||||
/// Queues one complete standard Yellowstone request mutation without waiting for network dispatch.
|
/// Queues one complete standard Yellowstone request mutation without waiting for network dispatch.
|
||||||
///
|
///
|
||||||
/// The mutation is rejected synchronously when local validation fails, the encoded request exceeds the configured outbound bound, the bounded request
|
/// The mutation is rejected synchronously when local validation fails, the encoded request exceeds the configured outbound bound, the bounded request
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/grpc_subscribe.rs
|
// file: crates/ksp-onchain-transport-lib/src/grpc_subscribe.rs
|
||||||
// version: 8
|
// version: 9
|
||||||
|
|
||||||
const MAX_GRPC_BLOCKHASH_TEXT_LENGTH_BYTES: usize = 128;
|
const MAX_GRPC_BLOCKHASH_TEXT_LENGTH_BYTES: usize = 128;
|
||||||
const MAX_GRPC_BLOCK_VECTOR_COUNT: usize = 65_536;
|
const MAX_GRPC_BLOCK_VECTOR_COUNT: usize = 65_536;
|
||||||
@@ -2529,13 +2529,34 @@ impl YellowstoneSubscribeEntryFilter {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Opaque deterministic identity material for one complete Yellowstone Subscribe request.
|
||||||
|
///
|
||||||
|
/// The encoded bytes remain private. `Hash` feeds those canonical bytes to a caller-provided hasher while `Debug` exposes only their length.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct YellowstoneSubscribeRequestIdentity {
|
||||||
|
bytes: std::vec::Vec<u8>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::hash::Hash for crate::YellowstoneSubscribeRequestIdentity {
|
||||||
|
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
|
||||||
|
state.write(&(self.bytes.len() as u64).to_be_bytes());
|
||||||
|
state.write(self.bytes.as_slice());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::YellowstoneSubscribeRequestIdentity {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_struct("YellowstoneSubscribeRequestIdentity").field("byte_len", &self.bytes.len()).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Provider-neutral standard Yellowstone subscribe request owned by KSP.
|
/// Provider-neutral standard Yellowstone subscribe request owned by KSP.
|
||||||
///
|
///
|
||||||
/// The seven upstream maps are represented independently and retain named empty entries. An entirely empty map is the logical KSP representation of no active
|
/// The seven upstream maps are represented independently and retain named empty entries. An entirely empty map is the logical KSP representation of no active
|
||||||
/// filter in that family; protobuf map encoding does not distinguish an omitted map from an empty map. Filter-group names are globally unique across all seven
|
/// filter in that family; protobuf map encoding does not distinguish an omitted map from an empty map. Filter-group names are globally unique across all seven
|
||||||
/// maps so the names echoed by `SubscribeUpdate.filters` remain unambiguous. `Debug` exposes only counts and common scalar options, never filter names or
|
/// maps so the names echoed by `SubscribeUpdate.filters` remain unambiguous. `Debug` exposes only counts and common scalar options, never filter names or
|
||||||
/// future
|
/// future filter payloads.
|
||||||
/// filter payloads.
|
|
||||||
#[derive(Clone, Eq, PartialEq)]
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
pub struct YellowstoneSubscribeRequest {
|
pub struct YellowstoneSubscribeRequest {
|
||||||
accounts: std::collections::BTreeMap<crate::YellowstoneSubscribeFilterName, crate::YellowstoneSubscribeAccountFilter>,
|
accounts: std::collections::BTreeMap<crate::YellowstoneSubscribeFilterName, crate::YellowstoneSubscribeAccountFilter>,
|
||||||
@@ -2551,7 +2572,7 @@ pub struct YellowstoneSubscribeRequest {
|
|||||||
from_slot: std::option::Option<u64>,
|
from_slot: std::option::Option<u64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl YellowstoneSubscribeRequest {
|
impl crate::YellowstoneSubscribeRequest {
|
||||||
/// Creates an empty subscribe request. Empty requests are valid because later bidi lifecycle code uses request mutations to clear filters or carry ping
|
/// Creates an empty subscribe request. Empty requests are valid because later bidi lifecycle code uses request mutations to clear filters or carry ping
|
||||||
/// state.
|
/// state.
|
||||||
#[must_use]
|
#[must_use]
|
||||||
@@ -2762,6 +2783,40 @@ impl YellowstoneSubscribeRequest {
|
|||||||
return self.from_slot;
|
return self.from_slot;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Builds one opaque deterministic identity for the complete logical Subscribe request.
|
||||||
|
///
|
||||||
|
/// The identity preserves filter-family separation, globally sorted filter names, exact filter wire payloads and common request options. Its Debug surface
|
||||||
|
/// exposes only the encoded byte length. Callers may hash the opaque value but cannot recover the underlying identity bytes through this API.
|
||||||
|
pub fn identity(&self) -> ksp_core_lib::Result<crate::YellowstoneSubscribeRequestIdentity> {
|
||||||
|
let validation = self.validate();
|
||||||
|
if let std::result::Result::Err(error) = validation {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let mut bytes = std::vec::Vec::new();
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"accounts", &self.accounts, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"blocks", &self.blocks, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"blocks_meta", &self.blocks_meta, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"entry", &self.entry, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"slots", &self.slots, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"transactions", &self.transactions, |filter| return filter.to_wire());
|
||||||
|
append_subscribe_identity_map(&mut bytes, b"transactions_status", &self.transactions_status, |filter| return filter.to_wire());
|
||||||
|
let mut common = match self.to_wire() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
common.accounts.clear();
|
||||||
|
common.blocks.clear();
|
||||||
|
common.blocks_meta.clear();
|
||||||
|
common.entry.clear();
|
||||||
|
common.slots.clear();
|
||||||
|
common.transactions.clear();
|
||||||
|
common.transactions_status.clear();
|
||||||
|
let common = yellowstone_grpc_proto::prost::Message::encode_to_vec(&common);
|
||||||
|
append_subscribe_identity_component(&mut bytes, b"common");
|
||||||
|
append_subscribe_identity_component(&mut bytes, common.as_slice());
|
||||||
|
return std::result::Result::Ok(crate::YellowstoneSubscribeRequestIdentity { bytes });
|
||||||
|
}
|
||||||
|
|
||||||
/// Validates all deterministic common subscribe-request bounds before any network I/O.
|
/// Validates all deterministic common subscribe-request bounds before any network I/O.
|
||||||
pub fn validate(&self) -> ksp_core_lib::Result<()> {
|
pub fn validate(&self) -> ksp_core_lib::Result<()> {
|
||||||
if self.total_filter_count() > MAX_GRPC_SUBSCRIBE_FILTER_GROUP_COUNT {
|
if self.total_filter_count() > MAX_GRPC_SUBSCRIBE_FILTER_GROUP_COUNT {
|
||||||
@@ -3710,13 +3765,38 @@ fn commitment_to_wire(commitment: std::option::Option<crate::SolanaCommitment>)
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::default::Default for YellowstoneSubscribeRequest {
|
fn append_subscribe_identity_component(output: &mut std::vec::Vec<u8>, value: &[u8]) {
|
||||||
|
output.extend_from_slice(&(value.len() as u64).to_be_bytes());
|
||||||
|
output.extend_from_slice(value);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_subscribe_identity_map<V, W, F>(
|
||||||
|
output: &mut std::vec::Vec<u8>,
|
||||||
|
family: &[u8],
|
||||||
|
values: &std::collections::BTreeMap<crate::YellowstoneSubscribeFilterName, V>,
|
||||||
|
mut to_wire: F,
|
||||||
|
) where
|
||||||
|
W: yellowstone_grpc_proto::prost::Message,
|
||||||
|
F: FnMut(&V) -> W,
|
||||||
|
{
|
||||||
|
append_subscribe_identity_component(output, family);
|
||||||
|
output.extend_from_slice(&(values.len() as u64).to_be_bytes());
|
||||||
|
for (name, filter) in values {
|
||||||
|
append_subscribe_identity_component(output, name.as_str().as_bytes());
|
||||||
|
let wire = yellowstone_grpc_proto::prost::Message::encode_to_vec(&to_wire(filter));
|
||||||
|
append_subscribe_identity_component(output, wire.as_slice());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::default::Default for crate::YellowstoneSubscribeRequest {
|
||||||
fn default() -> Self {
|
fn default() -> Self {
|
||||||
return Self::new();
|
return Self::new();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::fmt::Debug for YellowstoneSubscribeRequest {
|
impl std::fmt::Debug for crate::YellowstoneSubscribeRequest {
|
||||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
return formatter
|
return formatter
|
||||||
.debug_struct("YellowstoneSubscribeRequest")
|
.debug_struct("YellowstoneSubscribeRequest")
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/lib.rs
|
// file: crates/ksp-onchain-transport-lib/src/lib.rs
|
||||||
// version: 45
|
// version: 47
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
#![deny(unreachable_pub)]
|
#![deny(unreachable_pub)]
|
||||||
@@ -143,6 +143,8 @@ pub use self::grpc_settings::YellowstoneGrpcTransportSettings;
|
|||||||
pub use self::grpc_stream::SolanaYellowstoneGrpcSubscribeSession;
|
pub use self::grpc_stream::SolanaYellowstoneGrpcSubscribeSession;
|
||||||
/// Safe Yellowstone reconnect/replay continuity snapshot.
|
/// Safe Yellowstone reconnect/replay continuity snapshot.
|
||||||
pub use self::grpc_stream::YellowstoneGrpcSubscribeSnapshot;
|
pub use self::grpc_stream::YellowstoneGrpcSubscribeSnapshot;
|
||||||
|
/// Cloneable latest-value observer for one standard Yellowstone Subscribe session snapshot.
|
||||||
|
pub use self::grpc_stream::YellowstoneGrpcSubscribeSnapshotSource;
|
||||||
/// Safe Yellowstone bidirectional Subscribe lifecycle state.
|
/// Safe Yellowstone bidirectional Subscribe lifecycle state.
|
||||||
pub use self::grpc_stream::YellowstoneGrpcSubscribeState;
|
pub use self::grpc_stream::YellowstoneGrpcSubscribeState;
|
||||||
/// One validated standard Yellowstone account predicate.
|
/// One validated standard Yellowstone account predicate.
|
||||||
@@ -213,6 +215,8 @@ pub use self::grpc_subscribe::YellowstoneSubscribePingUpdate;
|
|||||||
pub use self::grpc_subscribe::YellowstoneSubscribePongUpdate;
|
pub use self::grpc_subscribe::YellowstoneSubscribePongUpdate;
|
||||||
/// Provider-neutral standard Yellowstone Subscribe request.
|
/// Provider-neutral standard Yellowstone Subscribe request.
|
||||||
pub use self::grpc_subscribe::YellowstoneSubscribeRequest;
|
pub use self::grpc_subscribe::YellowstoneSubscribeRequest;
|
||||||
|
/// Opaque deterministic identity for one complete Yellowstone Subscribe request.
|
||||||
|
pub use self::grpc_subscribe::YellowstoneSubscribeRequestIdentity;
|
||||||
/// Complete slot-family filter group for standard Yellowstone Subscribe.
|
/// Complete slot-family filter group for standard Yellowstone Subscribe.
|
||||||
pub use self::grpc_subscribe::YellowstoneSubscribeSlotFilter;
|
pub use self::grpc_subscribe::YellowstoneSubscribeSlotFilter;
|
||||||
/// Complete transaction-family filter shared by transactions and transaction-status maps.
|
/// Complete transaction-family filter shared by transactions and transaction-status maps.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/rpc_blocks.rs
|
// file: crates/ksp-onchain-transport-lib/src/rpc_blocks.rs
|
||||||
// version: 7
|
// version: 8
|
||||||
|
|
||||||
const MAX_GET_BLOCKS_RANGE: u64 = 500_000;
|
const MAX_GET_BLOCKS_RANGE: u64 = 500_000;
|
||||||
const MAX_GET_RECENT_PERFORMANCE_SAMPLES: u64 = 720;
|
const MAX_GET_RECENT_PERFORMANCE_SAMPLES: u64 = 720;
|
||||||
@@ -591,25 +591,42 @@ impl crate::HttpTransportPool {
|
|||||||
slot: u64,
|
slot: u64,
|
||||||
config: std::option::Option<&crate::SolanaGetBlockConfig>,
|
config: std::option::Option<&crate::SolanaGetBlockConfig>,
|
||||||
) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedBlock>> {
|
) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedBlock>> {
|
||||||
if let std::option::Option::Some(config) = config
|
let params = get_block_params(slot, config);
|
||||||
&& config.commitment() == std::option::Option::Some(crate::SolanaCommitment::Processed)
|
let params = match params {
|
||||||
{
|
std::result::Result::Ok(params) => params,
|
||||||
return std::result::Result::Err(
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
ksp_core_lib::Error::new(
|
};
|
||||||
crate::ERROR_CODE_INVALID_RPC_PARAMETERS,
|
|
||||||
"getBlock commitment must be confirmed or finalized when explicitly provided",
|
|
||||||
)
|
|
||||||
.with_context("rpc_method", "getBlock")
|
|
||||||
.with_context("commitment", "processed"),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
let mut params = std::vec![serde_json::json!(slot)];
|
|
||||||
if let std::option::Option::Some(config) = config {
|
|
||||||
params.push((*config).to_json_value());
|
|
||||||
}
|
|
||||||
return self.execute_get_block(role, params).await;
|
return self.execute_get_block(role, params).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Executes the current object-form `getBlock` request and reports the safe identity of the endpoint that produced the successful response.
|
||||||
|
///
|
||||||
|
/// Routing, admission, timeout and retry behavior are identical to [`Self::get_block`]. The returned observation never contains an endpoint URL,
|
||||||
|
/// HTTP headers or a raw HTTP body.
|
||||||
|
pub async fn get_block_observed(
|
||||||
|
&self,
|
||||||
|
role: &crate::HttpRoleName,
|
||||||
|
slot: u64,
|
||||||
|
config: std::option::Option<&crate::SolanaGetBlockConfig>,
|
||||||
|
) -> ksp_core_lib::Result<crate::HttpObservedValue<std::option::Option<crate::SolanaConfirmedBlock>>> {
|
||||||
|
let params = get_block_params(slot, config);
|
||||||
|
let params = match params {
|
||||||
|
std::result::Result::Ok(params) => params,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let observed = self.execute_blocks_rpc_observed("getBlock", role, params).await;
|
||||||
|
let observed = match observed {
|
||||||
|
std::result::Result::Ok(observed) => observed,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let (value, endpoint_name, provider) = observed.into_parts();
|
||||||
|
let block = decode_get_block(value);
|
||||||
|
return match block {
|
||||||
|
std::result::Result::Ok(block) => std::result::Result::Ok(crate::HttpObservedValue::new(block, endpoint_name, provider)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/// Executes the deprecated bare-encoding `getBlock` request form retained by Solana RPC for backwards compatibility.
|
/// Executes the deprecated bare-encoding `getBlock` request form retained by Solana RPC for backwards compatibility.
|
||||||
#[deprecated(note = "use HttpTransportPool::get_block with SolanaGetBlockConfig; the bare encoding request form is deprecated")]
|
#[deprecated(note = "use HttpTransportPool::get_block with SolanaGetBlockConfig; the bare encoding request form is deprecated")]
|
||||||
pub async fn get_block_legacy(
|
pub async fn get_block_legacy(
|
||||||
@@ -635,16 +652,8 @@ impl crate::HttpTransportPool {
|
|||||||
params: std::vec::Vec<serde_json::Value>,
|
params: std::vec::Vec<serde_json::Value>,
|
||||||
) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedBlock>> {
|
) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedBlock>> {
|
||||||
let value = self.execute_blocks_rpc("getBlock", role, params).await;
|
let value = self.execute_blocks_rpc("getBlock", role, params).await;
|
||||||
let value = match value {
|
return match value {
|
||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => decode_get_block(value),
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
if value.is_null() {
|
|
||||||
return std::result::Result::Ok(std::option::Option::None);
|
|
||||||
}
|
|
||||||
let block = crate::SolanaConfirmedBlock::decode_wire("getBlock", value);
|
|
||||||
return match block {
|
|
||||||
std::result::Result::Ok(block) => std::result::Result::Ok(std::option::Option::Some(block)),
|
|
||||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -829,6 +838,48 @@ impl crate::HttpTransportPool {
|
|||||||
};
|
};
|
||||||
return self.execute_standard_rpc(role, method, params).await;
|
return self.execute_standard_rpc(role, method, params).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn execute_blocks_rpc_observed(
|
||||||
|
&self,
|
||||||
|
method_name: &'static str,
|
||||||
|
role: &crate::HttpRoleName,
|
||||||
|
params: std::vec::Vec<serde_json::Value>,
|
||||||
|
) -> ksp_core_lib::Result<crate::HttpObservedValue<serde_json::Value>> {
|
||||||
|
let method = blocks_descriptor(method_name);
|
||||||
|
let method = match method {
|
||||||
|
std::result::Result::Ok(method) => method,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return self.execute_standard_rpc_observed(role, method, params).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn decode_get_block(value: serde_json::Value) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedBlock>> {
|
||||||
|
if value.is_null() {
|
||||||
|
return std::result::Result::Ok(std::option::Option::None);
|
||||||
|
}
|
||||||
|
let block = crate::SolanaConfirmedBlock::decode_wire("getBlock", value);
|
||||||
|
return match block {
|
||||||
|
std::result::Result::Ok(block) => std::result::Result::Ok(std::option::Option::Some(block)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_block_params(slot: u64, config: std::option::Option<&crate::SolanaGetBlockConfig>) -> ksp_core_lib::Result<std::vec::Vec<serde_json::Value>> {
|
||||||
|
if let std::option::Option::Some(config) = config
|
||||||
|
&& config.commitment() == std::option::Option::Some(crate::SolanaCommitment::Processed)
|
||||||
|
{
|
||||||
|
return std::result::Result::Err(
|
||||||
|
ksp_core_lib::Error::new(crate::ERROR_CODE_INVALID_RPC_PARAMETERS, "getBlock commitment must be confirmed or finalized when explicitly provided")
|
||||||
|
.with_context("rpc_method", "getBlock")
|
||||||
|
.with_context("commitment", "processed"),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let mut params = std::vec![serde_json::json!(slot)];
|
||||||
|
if let std::option::Option::Some(config) = config {
|
||||||
|
params.push((*config).to_json_value());
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(params);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(serde::Deserialize)]
|
#[derive(serde::Deserialize)]
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/tests/public_api.rs
|
// file: crates/ksp-onchain-transport-lib/tests/public_api.rs
|
||||||
// version: 49
|
// version: 52
|
||||||
|
|
||||||
//! Integration tests for the public `ksp-onchain-transport-lib` consumer contract.
|
//! Integration tests for the public `ksp-onchain-transport-lib` consumer contract.
|
||||||
|
|
||||||
@@ -1013,6 +1013,10 @@ fn public_v0_2_9_pre_010_yellowstone_reconnect_snapshot_is_available_from_crate_
|
|||||||
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
||||||
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
||||||
let _session_snapshot = ksp_onchain_transport_lib::SolanaYellowstoneGrpcSubscribeSession::snapshot;
|
let _session_snapshot = ksp_onchain_transport_lib::SolanaYellowstoneGrpcSubscribeSession::snapshot;
|
||||||
|
let _snapshot_source = ksp_onchain_transport_lib::SolanaYellowstoneGrpcSubscribeSession::snapshot_source;
|
||||||
|
let _source_type = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshotSource>();
|
||||||
|
let _source_current = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshotSource::current;
|
||||||
|
let _source_wait = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshotSource::wait_for_change;
|
||||||
let _reconnect_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::reconnect_count;
|
let _reconnect_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::reconnect_count;
|
||||||
let _gap_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::continuity_gap_count;
|
let _gap_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::continuity_gap_count;
|
||||||
let _duplicate_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::duplicate_update_count;
|
let _duplicate_count = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::duplicate_update_count;
|
||||||
@@ -1021,3 +1025,22 @@ fn public_v0_2_9_pre_010_yellowstone_reconnect_snapshot_is_available_from_crate_
|
|||||||
let _observed = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::last_observed_slot;
|
let _observed = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::last_observed_slot;
|
||||||
let _terminal = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::terminal_error_code;
|
let _terminal = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot::terminal_error_code;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_10_pre_004_observed_get_block_surface_is_available_from_crate_root() {
|
||||||
|
let method = ksp_onchain_transport_lib::HttpTransportPool::get_block_observed;
|
||||||
|
let _ = method;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_13_pre_002_yellowstone_subscribe_identity_is_opaque_and_available_from_crate_root() {
|
||||||
|
let request = ksp_onchain_transport_lib::YellowstoneSubscribeRequest::new();
|
||||||
|
let identity = request.identity().expect("empty validated request identity must build");
|
||||||
|
let _identity_type = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneSubscribeRequestIdentity>();
|
||||||
|
let debug = std::format!("{identity:?}");
|
||||||
|
assert!(debug.contains("YellowstoneSubscribeRequestIdentity"));
|
||||||
|
assert!(debug.contains("byte_len"));
|
||||||
|
assert!(!debug.contains("bytes:"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/tests/release_completeness.rs
|
// file: crates/ksp-onchain-transport-lib/tests/release_completeness.rs
|
||||||
// version: 42
|
// version: 44
|
||||||
|
|
||||||
//! Release-level completeness canaries for staged HTTP and WebSocket Transport coverage.
|
//! Release-level completeness canaries for staged HTTP and WebSocket Transport coverage.
|
||||||
|
|
||||||
@@ -1307,6 +1307,9 @@ fn release_v0_2_9_pre_010_adds_bounded_reconnect_replay_and_conservative_continu
|
|||||||
for required in [
|
for required in [
|
||||||
"YellowstoneGrpcSubscribeState::Reconnecting",
|
"YellowstoneGrpcSubscribeState::Reconnecting",
|
||||||
"YellowstoneGrpcSubscribeSnapshot",
|
"YellowstoneGrpcSubscribeSnapshot",
|
||||||
|
"YellowstoneGrpcSubscribeSnapshotSource",
|
||||||
|
"snapshot_source",
|
||||||
|
"wait_for_change",
|
||||||
"reconnect_subscribe_stream",
|
"reconnect_subscribe_stream",
|
||||||
"replay_first_available",
|
"replay_first_available",
|
||||||
"SubscribeReplayInfo",
|
"SubscribeReplayInfo",
|
||||||
@@ -1336,3 +1339,18 @@ fn release_v0_2_9_pre_010_adds_bounded_reconnect_replay_and_conservative_continu
|
|||||||
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
let _snapshot = std::any::type_name::<ksp_onchain_transport_lib::YellowstoneGrpcSubscribeSnapshot>();
|
||||||
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
let _state = ksp_onchain_transport_lib::YellowstoneGrpcSubscribeState::Reconnecting;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn release_v0_3_13_pre_002_yellowstone_subscribe_identity_remains_opaque_and_dependency_neutral() {
|
||||||
|
let source = include_str!("../src/grpc_subscribe.rs");
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let identity_start = source.find("pub struct YellowstoneSubscribeRequestIdentity").expect("identity struct must remain present");
|
||||||
|
let request_start = source.find("/// Provider-neutral standard Yellowstone subscribe request").expect("request contract marker must remain present");
|
||||||
|
let identity_surface = &source[identity_start..request_start];
|
||||||
|
assert!(source.contains("pub fn identity(&self)"));
|
||||||
|
assert!(root.contains("pub use self::grpc_subscribe::YellowstoneSubscribeRequestIdentity;"));
|
||||||
|
assert!(!identity_surface.contains("pub fn bytes("));
|
||||||
|
assert!(!identity_surface.contains("pub fn as_bytes("));
|
||||||
|
assert!(!root.contains("pub use yellowstone_grpc_proto"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_stream.rs
|
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_stream.rs
|
||||||
// version: 4
|
// version: 5
|
||||||
|
|
||||||
#[derive(Clone, Copy)]
|
#[derive(Clone, Copy)]
|
||||||
enum FixtureMode {
|
enum FixtureMode {
|
||||||
@@ -159,6 +159,9 @@ impl yellowstone_grpc_proto::geyser::geyser_server::Geyser for FixtureGeyser {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
if matches!(mode, FixtureMode::ReconnectReplay) && subscribe_call == 2 {
|
||||||
|
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
|
||||||
|
}
|
||||||
return std::result::Result::Ok(tonic::Response::new(super::MpscStream::new(outbound_rx)));
|
return std::result::Result::Ok(tonic::Response::new(super::MpscStream::new(outbound_rx)));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -662,6 +665,45 @@ async fn yellowstone_reconnect_budget_exhaustion_is_terminal_and_safe() {
|
|||||||
server.stop().await;
|
server.stop().await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn yellowstone_snapshot_source_observes_replay_attempt_before_successful_reconnect() {
|
||||||
|
let server = FixtureServer::start(FixtureMode::ReconnectReplay).await;
|
||||||
|
let defaults = crate::YellowstoneGrpcSessionSettings::default();
|
||||||
|
let settings = fixture_settings_with_reconnect(
|
||||||
|
server.endpoint_url.as_str(),
|
||||||
|
8,
|
||||||
|
8,
|
||||||
|
defaults.max_inbound_message_size_bytes(),
|
||||||
|
defaults.max_outbound_message_size_bytes(),
|
||||||
|
crate::YellowstoneGrpcReconnectSettings::new(3, std::time::Duration::from_millis(5), std::time::Duration::from_millis(20)),
|
||||||
|
);
|
||||||
|
let channel = crate::YellowstoneGrpcChannel::connect(&settings).await.expect("fixture channel must connect");
|
||||||
|
let mut session = channel.open_standard_subscribe(initial_request()).await.expect("fixture Subscribe stream must open");
|
||||||
|
let mut snapshots = session.snapshot_source();
|
||||||
|
assert_eq!(snapshots.current().state(), crate::YellowstoneGrpcSubscribeState::Active);
|
||||||
|
let _ = session.next_update().await.expect("first slot must decode").expect("first slot must be present");
|
||||||
|
let replaying = tokio::time::timeout(std::time::Duration::from_secs(1), async {
|
||||||
|
loop {
|
||||||
|
let snapshot = match snapshots.wait_for_change().await {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
if snapshot.replay_attempt_count() > 0 && snapshot.reconnect_count() == 0 {
|
||||||
|
return std::option::Option::Some(snapshot);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.expect("snapshot source must observe reconnect progress")
|
||||||
|
.expect("snapshot source must stay open during reconnect");
|
||||||
|
assert_eq!(replaying.state(), crate::YellowstoneGrpcSubscribeState::Reconnecting);
|
||||||
|
assert_eq!(replaying.replay_attempt_count(), 1);
|
||||||
|
assert_eq!(replaying.reconnect_count(), 0);
|
||||||
|
let _ = session.next_update().await.expect("replayed duplicate must decode").expect("replayed duplicate must be present");
|
||||||
|
session.close().await.expect("reconnected stream must close cleanly");
|
||||||
|
server.stop().await;
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test(flavor = "current_thread")]
|
#[tokio::test(flavor = "current_thread")]
|
||||||
async fn yellowstone_shutdown_interrupts_reconnect_backoff_and_mutation_is_rejected_during_reconnect() {
|
async fn yellowstone_shutdown_interrupts_reconnect_backoff_and_mutation_is_rejected_during_reconnect() {
|
||||||
let server = FixtureServer::start(FixtureMode::ReconnectReplay).await;
|
let server = FixtureServer::start(FixtureMode::ReconnectReplay).await;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs
|
// file: crates/ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs
|
||||||
// version: 5
|
// version: 6
|
||||||
|
|
||||||
fn filter_name(value: &str) -> crate::YellowstoneSubscribeFilterName {
|
fn filter_name(value: &str) -> crate::YellowstoneSubscribeFilterName {
|
||||||
return crate::YellowstoneSubscribeFilterName::new(value).expect("fixture filter name must validate");
|
return crate::YellowstoneSubscribeFilterName::new(value).expect("fixture filter name must validate");
|
||||||
@@ -138,6 +138,37 @@ fn yellowstone_subscribe_common_bounds_reject_before_wire_conversion() {
|
|||||||
assert!(filters.insert_account_filter(excess_name, crate::YellowstoneSubscribeAccountFilter::new()).is_err());
|
assert!(filters.insert_account_filter(excess_name, crate::YellowstoneSubscribeAccountFilter::new()).is_err());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn yellowstone_subscribe_request_identity_is_order_stable_exact_and_debug_redacted() {
|
||||||
|
let mut first = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
let mut second = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
let mut changed = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
let mut filter_a = crate::YellowstoneSubscribeTransactionFilter::new();
|
||||||
|
filter_a.set_failed(std::option::Option::Some(false));
|
||||||
|
let mut filter_b = crate::YellowstoneSubscribeTransactionFilter::new();
|
||||||
|
filter_b.set_vote(std::option::Option::Some(false));
|
||||||
|
assert!(first.insert_transaction_filter(filter_name("identity-filter-beta-canary"), filter_b.clone()).is_ok());
|
||||||
|
assert!(first.insert_transaction_filter(filter_name("identity-filter-alpha-canary"), filter_a.clone()).is_ok());
|
||||||
|
assert!(second.insert_transaction_filter(filter_name("identity-filter-alpha-canary"), filter_a.clone()).is_ok());
|
||||||
|
assert!(second.insert_transaction_filter(filter_name("identity-filter-beta-canary"), filter_b.clone()).is_ok());
|
||||||
|
filter_a.set_failed(std::option::Option::Some(true));
|
||||||
|
assert!(changed.insert_transaction_filter(filter_name("identity-filter-alpha-canary"), filter_a).is_ok());
|
||||||
|
assert!(changed.insert_transaction_filter(filter_name("identity-filter-beta-canary"), filter_b).is_ok());
|
||||||
|
first.set_commitment(std::option::Option::Some(crate::SolanaCommitment::Confirmed));
|
||||||
|
second.set_commitment(std::option::Option::Some(crate::SolanaCommitment::Confirmed));
|
||||||
|
changed.set_commitment(std::option::Option::Some(crate::SolanaCommitment::Confirmed));
|
||||||
|
let first_identity = first.identity().expect("first request identity must build");
|
||||||
|
let second_identity = second.identity().expect("second request identity must build");
|
||||||
|
let changed_identity = changed.identity().expect("changed request identity must build");
|
||||||
|
assert_eq!(first_identity, second_identity);
|
||||||
|
assert_ne!(first_identity, changed_identity);
|
||||||
|
let debug = std::format!("{first_identity:?}");
|
||||||
|
assert!(debug.contains("byte_len"));
|
||||||
|
assert!(!debug.contains("identity-filter-alpha-canary"));
|
||||||
|
assert!(!debug.contains("identity-filter-beta-canary"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn yellowstone_subscribe_debug_omits_filter_names_and_future_payloads() {
|
fn yellowstone_subscribe_debug_omits_filter_names_and_future_payloads() {
|
||||||
let mut request = crate::YellowstoneSubscribeRequest::new();
|
let mut request = crate::YellowstoneSubscribeRequest::new();
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/unit_tests/rpc_blocks.rs
|
// file: crates/ksp-onchain-transport-lib/unit_tests/rpc_blocks.rs
|
||||||
// version: 6
|
// version: 7
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn transaction_details_and_get_block_config_preserve_all_modern_options() {
|
fn transaction_details_and_get_block_config_preserve_all_modern_options() {
|
||||||
@@ -188,6 +188,35 @@ fn serve_once(body: &'static str) -> (std::string::String, std::thread::JoinHand
|
|||||||
return (format!("http://{address}"), handle);
|
return (format!("http://{address}"), handle);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn serve_status_and_count(status_line: &'static str) -> (std::string::String, std::thread::JoinHandle<(usize, std::string::String)>) {
|
||||||
|
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("fixture listener must bind");
|
||||||
|
let address = listener.local_addr().expect("fixture listener address must resolve");
|
||||||
|
let handle = std::thread::spawn(move || {
|
||||||
|
let (mut stream, _) = listener.accept().expect("fixture server must accept first request");
|
||||||
|
let first_request = read_request(&mut stream);
|
||||||
|
let response = format!("HTTP/1.1 {status_line}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n");
|
||||||
|
std::io::Write::write_all(&mut stream, response.as_bytes()).expect("fixture response must write");
|
||||||
|
let mut count = 1_usize;
|
||||||
|
listener.set_nonblocking(true).expect("fixture listener must become nonblocking");
|
||||||
|
let deadline = std::time::Instant::now() + std::time::Duration::from_millis(120);
|
||||||
|
while std::time::Instant::now() < deadline {
|
||||||
|
match listener.accept() {
|
||||||
|
std::result::Result::Ok((mut retry_stream, _)) => {
|
||||||
|
let _ = read_request(&mut retry_stream);
|
||||||
|
std::io::Write::write_all(&mut retry_stream, response.as_bytes()).expect("fixture retry response must write");
|
||||||
|
count = count.saturating_add(1);
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {
|
||||||
|
std::thread::sleep(std::time::Duration::from_millis(5));
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => panic!("fixture listener failed while counting retries: {error}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return (count, first_request);
|
||||||
|
});
|
||||||
|
return (format!("http://{address}"), handle);
|
||||||
|
}
|
||||||
|
|
||||||
fn read_request(stream: &mut std::net::TcpStream) -> std::string::String {
|
fn read_request(stream: &mut std::net::TcpStream) -> std::string::String {
|
||||||
let mut bytes = std::vec::Vec::new();
|
let mut bytes = std::vec::Vec::new();
|
||||||
let mut buffer = [0_u8; 1024];
|
let mut buffer = [0_u8; 1024];
|
||||||
@@ -784,3 +813,92 @@ async fn typed_get_block_rejects_processed_commitment_before_io() {
|
|||||||
assert_eq!(error.context()[1].key(), "commitment");
|
assert_eq!(error.context()[1].key(), "commitment");
|
||||||
assert_eq!(error.context()[1].value(), "processed");
|
assert_eq!(error.context()[1].value(), "processed");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_get_block_observed_reports_actual_winner_after_retry_reroute() {
|
||||||
|
let (first_url, first_handle) = serve_status_and_count("429 Too Many Requests");
|
||||||
|
let (winner_url, winner_handle) = serve_once(include_str!("../fixtures/http/get_block.observed_material.success.json"));
|
||||||
|
let urls = [(first_url.as_str(), "first-endpoint", "first-provider"), (winner_url.as_str(), "winner-endpoint", "winner-provider")];
|
||||||
|
let mut endpoints = std::vec::Vec::with_capacity(urls.len());
|
||||||
|
for (url, endpoint_name, provider) in urls {
|
||||||
|
let role = crate::HttpEndpointRoleSettings::new(
|
||||||
|
crate::HttpRoleName::new("default"),
|
||||||
|
true,
|
||||||
|
std::vec![crate::HttpRequestKind::wildcard()],
|
||||||
|
10,
|
||||||
|
crate::HttpRoleLimits::new(std::option::Option::None, std::option::Option::None, std::option::Option::None, std::option::Option::None),
|
||||||
|
);
|
||||||
|
endpoints.push(crate::HttpEndpointSettings::new(
|
||||||
|
endpoint_name,
|
||||||
|
true,
|
||||||
|
crate::HttpProviderName::new(provider),
|
||||||
|
crate::HttpClusterName::new("local"),
|
||||||
|
crate::HttpEndpointUrl::parse(url).expect("fixture URL must parse"),
|
||||||
|
std::time::Duration::from_millis(100),
|
||||||
|
std::time::Duration::from_secs(1),
|
||||||
|
std::option::Option::Some(1),
|
||||||
|
std::vec![role],
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let pool = crate::HttpTransportPool::new(crate::HttpTransportSettings::new(
|
||||||
|
endpoints,
|
||||||
|
crate::HttpRetrySettings::new(1, std::time::Duration::from_millis(1), std::time::Duration::from_millis(2)),
|
||||||
|
))
|
||||||
|
.expect("observed fixture pool must build");
|
||||||
|
let config = crate::SolanaGetBlockConfig::new(
|
||||||
|
std::option::Option::Some(crate::SolanaCommitment::Confirmed),
|
||||||
|
std::option::Option::Some(crate::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(crate::SolanaTransactionDetails::Full),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
std::option::Option::Some(false),
|
||||||
|
);
|
||||||
|
let observed = pool
|
||||||
|
.get_block_observed(&crate::HttpRoleName::new("default"), 430_000_123, std::option::Option::Some(&config))
|
||||||
|
.await
|
||||||
|
.expect("retry-safe observed getBlock must succeed on the second endpoint");
|
||||||
|
assert_eq!(observed.endpoint_name(), "winner-endpoint");
|
||||||
|
assert_eq!(observed.provider().as_str(), "winner-provider");
|
||||||
|
let block = observed.value().as_ref().expect("winning response must contain a block");
|
||||||
|
let transactions = block.transactions().value().expect("winning block must contain transactions");
|
||||||
|
assert_eq!(transactions.len(), 1);
|
||||||
|
assert!(matches!(
|
||||||
|
transactions[0].transaction(),
|
||||||
|
crate::SolanaEncodedTransaction::Binary { encoding: crate::SolanaTransactionBinaryEncoding::Base64, .. }
|
||||||
|
));
|
||||||
|
let rendered = format!("{observed:?}");
|
||||||
|
assert!(rendered.contains("winner-endpoint"));
|
||||||
|
assert!(rendered.contains("winner-provider"));
|
||||||
|
assert!(rendered.contains("<available>"));
|
||||||
|
assert!(!rendered.contains("AQAAAAAAAA"));
|
||||||
|
let (first_count, first_request) = first_handle.join().expect("first fixture server must join");
|
||||||
|
assert_eq!(first_count, 1);
|
||||||
|
assert_eq!(request_body(first_request.as_str())["method"], serde_json::json!("getBlock"));
|
||||||
|
let winner_request = winner_handle.join().expect("winner fixture server must join");
|
||||||
|
assert_eq!(request_body(winner_request.as_str())["method"], serde_json::json!("getBlock"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_get_block_observed_preserves_null_and_reuses_get_block_validation() {
|
||||||
|
let (url, handle) = serve_once(include_str!("../fixtures/http/get_block.null.json"));
|
||||||
|
let pool = pool_for_url(url.as_str());
|
||||||
|
let observed = pool
|
||||||
|
.get_block_observed(&crate::HttpRoleName::new("default"), 430_000_123, std::option::Option::None)
|
||||||
|
.await
|
||||||
|
.expect("observed null getBlock must succeed");
|
||||||
|
assert!(observed.value().is_none());
|
||||||
|
assert_eq!(observed.endpoint_name(), "fixture");
|
||||||
|
assert_eq!(observed.provider().as_str(), "fixture");
|
||||||
|
handle.join().expect("fixture server must join");
|
||||||
|
let processed = crate::SolanaGetBlockConfig::new(
|
||||||
|
std::option::Option::Some(crate::SolanaCommitment::Processed),
|
||||||
|
std::option::Option::Some(crate::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(crate::SolanaTransactionDetails::Full),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
std::option::Option::Some(false),
|
||||||
|
);
|
||||||
|
let result = pool.get_block_observed(&crate::HttpRoleName::new("default"), 1, std::option::Option::Some(&processed)).await;
|
||||||
|
let error = result.expect_err("processed observed getBlock commitment must reject before I/O");
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_INVALID_RPC_PARAMETERS);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
18
crates/ksp-raw-transaction-lib/Cargo.toml
Normal file
18
crates/ksp-raw-transaction-lib/Cargo.toml
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
# file: crates/ksp-raw-transaction-lib/Cargo.toml
|
||||||
|
# version: 2
|
||||||
|
|
||||||
|
[package]
|
||||||
|
name = "ksp-raw-transaction-lib"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
base64.workspace = true
|
||||||
|
ksp-core-lib = { path = "../ksp-core-lib" }
|
||||||
|
ksp-store-api = { path = "../ksp-store-api" }
|
||||||
|
serde_json = { workspace = true }
|
||||||
|
sha2 = { workspace = true }
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
98
crates/ksp-raw-transaction-lib/README.md
Normal file
98
crates/ksp-raw-transaction-lib/README.md
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
<!-- file: crates/ksp-raw-transaction-lib/README.md -->
|
||||||
|
<!-- version: 2 -->
|
||||||
|
|
||||||
|
# ksp-raw-transaction-lib
|
||||||
|
|
||||||
|
`ksp-raw-transaction-lib` fournit la lower-layer KSP source-neutral commune aux producteurs de `RawTransaction`.
|
||||||
|
|
||||||
|
La crate transforme un matériau transactionnel complet déjà acquis en représentation RAW canonique KSP, fournit les primitives de parsing de signature et de sérialisation wire Solana nécessaires aux voies qualifiées, puis permet d'associer la transaction canonique à une observation/provenance possédée par le producer.
|
||||||
|
|
||||||
|
Elle ne possède aucun Transport, Config, Job, Worker, runtime async, Store runtime ou backend physique.
|
||||||
|
|
||||||
|
## Responsabilités
|
||||||
|
|
||||||
|
La façade crate-root expose quatre familles de contrats.
|
||||||
|
|
||||||
|
### Canonicalisation RAW Transaction
|
||||||
|
|
||||||
|
- `RawTransactionMaterial` ;
|
||||||
|
- `RawTransactionWireField<T>` pour préserver distinctement omission, `null` explicite et valeur ;
|
||||||
|
- `RawTransactionVersion` ;
|
||||||
|
- `canonicalize_raw_transaction` ;
|
||||||
|
- `RAW_TRANSACTION_FORMAT_ID` et `RAW_TRANSACTION_FORMAT_VERSION`.
|
||||||
|
|
||||||
|
La canonicalisation produit un `ksp_store_api::RawTransaction` avec payload KSP déterministe et SHA-256 de contenu. Le producer reste responsable de fournir un matériau source-neutral complet ; la crate n'effectue aucune I/O réseau.
|
||||||
|
|
||||||
|
### Signature Solana
|
||||||
|
|
||||||
|
- `parse_raw_transaction_signature` valide et décode une signature Base58 bornée vers exactement 64 bytes ;
|
||||||
|
- `format_raw_transaction_signature` encode exactement 64 bytes canoniques vers la forme Base58 bornée correspondante ;
|
||||||
|
- `extract_raw_transaction_signature_from_binary_base64` extrait la première signature canonique d'un wire transactionnel Base64 complet ;
|
||||||
|
- les bornes textuelles publiques permettent l'admission avant décodage.
|
||||||
|
|
||||||
|
Les diagnostics ne recopient pas la signature hostile.
|
||||||
|
|
||||||
|
### Wire transactionnel source-neutral
|
||||||
|
|
||||||
|
La surface `RawSolana*` représente et sérialise les messages transactionnels Solana Legacy, V0 et V1 nécessaires à la common RAW :
|
||||||
|
|
||||||
|
- `RawSolanaMessageVersion` ;
|
||||||
|
- `RawSolanaMessageHeader` ;
|
||||||
|
- `RawSolanaCompiledInstruction` ;
|
||||||
|
- `RawSolanaAddressTableLookup` ;
|
||||||
|
- `RawSolanaTransactionConfig` ;
|
||||||
|
- `RawSolanaTransactionMessage` ;
|
||||||
|
- `RawSolanaTransactionWire` ;
|
||||||
|
- `serialize_solana_transaction_wire` et `serialize_solana_transaction_wire_base64`.
|
||||||
|
|
||||||
|
Cette surface est un contrat de wire commun, pas un decoder Program et pas une façade Transport.
|
||||||
|
|
||||||
|
### Acquisition canonique + observation
|
||||||
|
|
||||||
|
`assemble_raw_transaction_acquisition` associe une transaction canonique à :
|
||||||
|
|
||||||
|
- une `RawObservationKey` déterministe possédée par le producer ;
|
||||||
|
- une `RawAcquisitionProvenance` sûre possédée par le producer.
|
||||||
|
|
||||||
|
Le résultat `RawTransactionAcquisition` contient exactement une entité canonique et une observation liée. Il ne persiste rien lui-même.
|
||||||
|
|
||||||
|
## Frontières
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport / fixture / import / producer
|
||||||
|
|
|
||||||
|
v
|
||||||
|
matériau source-neutral
|
||||||
|
|
|
||||||
|
v
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
| |
|
||||||
|
| +-> exact Solana wire helpers
|
||||||
|
v
|
||||||
|
RawTransaction + RawTransactionObservation
|
||||||
|
|
|
||||||
|
v
|
||||||
|
producer runtime -> ksp-store-lib / capability Store appropriée
|
||||||
|
```
|
||||||
|
|
||||||
|
Interdictions structurelles :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib -X-> ksp-onchain-transport-lib
|
||||||
|
ksp-raw-transaction-lib -X-> ksp-config-lib
|
||||||
|
ksp-raw-transaction-lib -X-> ksp-job-api / ksp-job-backfill-lib
|
||||||
|
ksp-raw-transaction-lib -X-> ksp-worker-api / worker concret
|
||||||
|
ksp-raw-transaction-lib -X-> ksp-store-lib / backend physique
|
||||||
|
```
|
||||||
|
|
||||||
|
Le graphe normal reste limité à `ksp-core-lib`, `ksp-store-api`, `base64`, `serde_json` et `sha2`.
|
||||||
|
|
||||||
|
## Relation avec STRUCTURAL
|
||||||
|
|
||||||
|
Cette crate appartient à **RAW**. Elle ne réalise pas la décomposition Store D2 `RAW -> STRUCTURAL`. La future couche STRUCTURAL consommera le RAW persistant et le décomposera en unités Solana génériques plus fines avant tout décodage Program.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [`USAGE.md`](USAGE.md) — utilisation durable de la façade publique ;
|
||||||
|
- [`../../docs/architecture/008-DATA_MATERIALIZATION_AND_STORE.md`](../../docs/architecture/008-DATA_MATERIALIZATION_AND_STORE.md) — frontières RAW / STRUCTURAL / DECODED / DOMAIN ;
|
||||||
|
- [`../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`](../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md) — qualification des sources `RawTransaction`.
|
||||||
163
crates/ksp-raw-transaction-lib/USAGE.md
Normal file
163
crates/ksp-raw-transaction-lib/USAGE.md
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
<!-- file: crates/ksp-raw-transaction-lib/USAGE.md -->
|
||||||
|
<!-- version: 2 -->
|
||||||
|
|
||||||
|
# Utilisation de ksp-raw-transaction-lib
|
||||||
|
|
||||||
|
Cette page décrit la façade publique durable de `ksp-raw-transaction-lib`. Les consumers utilisent uniquement les exports du crate-root.
|
||||||
|
|
||||||
|
## Parser une signature transactionnelle
|
||||||
|
|
||||||
|
Lorsqu'une source fournit une signature Base58 textuelle, la convertir avec le parser commun plutôt que de dupliquer le décodage :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn parse_signature(value: &str) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
|
return ksp_raw_transaction_lib::parse_raw_transaction_signature(value);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le résultat contient exactement 64 bytes canoniques. Les entrées vides, hors bornes, Base58 invalides ou de longueur décodée incorrecte sont rejetées sans recopier la valeur hostile dans l'erreur.
|
||||||
|
|
||||||
|
Lorsqu'un consumer possède déjà les 64 bytes canoniques et doit appeler une API textuelle Solana, utiliser l'encodeur commun plutôt que d'implémenter Base58 localement :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn format_signature(value: &ksp_store_api::RawTransactionSignature) -> std::string::String {
|
||||||
|
return ksp_raw_transaction_lib::format_raw_transaction_signature(value);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le texte produit respecte les bornes publiques de signature et effectue un round-trip exact avec `parse_raw_transaction_signature`.
|
||||||
|
|
||||||
|
Lorsqu'un wire Base64 complet contient déjà son tableau de signatures, utiliser :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn embedded_signature(value: &str) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
|
return ksp_raw_transaction_lib::extract_raw_transaction_signature_from_binary_base64(value);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Construire et sérialiser un wire Solana source-neutral
|
||||||
|
|
||||||
|
Le producer peut projeter son DTO Transport/protobuf vers les types `RawSolana*`, puis utiliser le sérialiseur commun. Exemple V1 minimal :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn serialize_v1() -> ksp_core_lib::Result<std::string::String> {
|
||||||
|
let message = ksp_raw_transaction_lib::RawSolanaTransactionMessage::new(
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageVersion::V1,
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageHeader::new(1, 0, 0),
|
||||||
|
vec![[1_u8; 32]],
|
||||||
|
[2_u8; 32],
|
||||||
|
vec![ksp_raw_transaction_lib::RawSolanaCompiledInstruction::new(0, vec![0], vec![3])],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::Some(ksp_raw_transaction_lib::RawSolanaTransactionConfig::default()),
|
||||||
|
);
|
||||||
|
let wire = ksp_raw_transaction_lib::RawSolanaTransactionWire::new(vec![[4_u8; 64]], message);
|
||||||
|
return ksp_raw_transaction_lib::serialize_solana_transaction_wire_base64(&wire);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour Legacy ou V0, choisir `RawSolanaMessageVersion::Legacy` ou `V0` et fournir uniquement les éléments admis par cette version. Le sérialiseur valide les invariants structurels avant de produire les bytes.
|
||||||
|
|
||||||
|
`RawSolanaAddressTableLookup` représente les lookups V0. `RawSolanaTransactionConfig` représente la configuration inline V1 ; le cas explicite où toutes ses options valent `None` reste sémantiquement présent.
|
||||||
|
|
||||||
|
## Construire le matériau RAW canonique
|
||||||
|
|
||||||
|
Une fois le wire transactionnel Base64 et les métadonnées source-neutral disponibles, construire `RawTransactionMaterial`. Les états wire doivent conserver la différence entre champ absent, `null` explicite et valeur :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn material(
|
||||||
|
network: ksp_store_api::RawNetworkId,
|
||||||
|
signature: ksp_store_api::RawTransactionSignature,
|
||||||
|
transaction_base64: std::string::String,
|
||||||
|
) -> ksp_raw_transaction_lib::RawTransactionMaterial {
|
||||||
|
return ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::None,
|
||||||
|
transaction_base64,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(ksp_raw_transaction_lib::RawTransactionVersion::Legacy),
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour une source où la signature doit être extraite du wire lui-même, `RawTransactionMaterial::binary_base64_with_embedded_signature` évite un second parser producer-owned.
|
||||||
|
|
||||||
|
## Canonicaliser vers RawTransaction
|
||||||
|
|
||||||
|
La canonicalisation est déterministe pour un même matériau complet :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn canonicalize(
|
||||||
|
material: ksp_raw_transaction_lib::RawTransactionMaterial,
|
||||||
|
) -> ksp_core_lib::Result<ksp_store_api::RawTransaction> {
|
||||||
|
return ksp_raw_transaction_lib::canonicalize_raw_transaction(material);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate :
|
||||||
|
|
||||||
|
- canonicalise les sous-arbres JSON significatifs ;
|
||||||
|
- conserve les états omission / `null` / valeur ;
|
||||||
|
- construit le payload au format KSP RAW Transaction ;
|
||||||
|
- calcule son SHA-256 ;
|
||||||
|
- applique les bornes communes avant construction du modèle Store.
|
||||||
|
|
||||||
|
Elle ne persiste pas la transaction et ne choisit aucune politique de retry, endpoint ou source.
|
||||||
|
|
||||||
|
## Associer l'observation du producer
|
||||||
|
|
||||||
|
Le producer construit sa provenance avec les types Store API, puis assemble l'entité et l'observation :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn acquisition(
|
||||||
|
transaction: ksp_store_api::RawTransaction,
|
||||||
|
) -> ksp_core_lib::Result<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
let provider = match ksp_store_api::RawProvenanceCode::new("example") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_api::RawProvenanceCode::new("solana.http") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let method = match ksp_store_api::RawProvenanceCode::new("get_transaction") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_api::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_api::RawAcquisitionProvenance::new(
|
||||||
|
provider,
|
||||||
|
protocol,
|
||||||
|
method,
|
||||||
|
ksp_store_api::RawAcquisitionOrigin::Live,
|
||||||
|
received_at,
|
||||||
|
);
|
||||||
|
return std::result::Result::Ok(ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(
|
||||||
|
transaction,
|
||||||
|
ksp_store_api::RawObservationKey::new([7_u8; 32]),
|
||||||
|
provenance,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La clé d'observation doit être déterministe selon la politique du producer. Plusieurs sources peuvent produire des observations différentes pour la même identité canonique `(network, signature)`.
|
||||||
|
|
||||||
|
## Persister depuis un Job ou un Worker
|
||||||
|
|
||||||
|
La common crate ne dépend pas du Store runtime. Le lifecycle host concret récupère les deux parties puis appelle la façade Store appropriée :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let (transaction, observation) = acquisition.into_parts();
|
||||||
|
// Le Job/Worker concret transmet ensuite transaction + observation à ksp-store-lib.
|
||||||
|
```
|
||||||
|
|
||||||
|
Ne pas introduire `ksp-store-lib`, Transport, Config, Job ou Worker dans `ksp-raw-transaction-lib` pour simplifier cet appel. La direction de dépendance reste du producer vers la common RAW, puis du producer vers le Store runtime.
|
||||||
|
|
||||||
|
## Frontière avec STRUCTURAL
|
||||||
|
|
||||||
|
Ne pas utiliser cette crate pour décoder des Programs ou pour produire directement des unités STRUCTURAL. Son résultat reste D1 RAW. La décomposition `RAW -> STRUCTURAL` appartient à une couche ultérieure et réutilise le RAW durable comme input.
|
||||||
49
crates/ksp-raw-transaction-lib/src/acquisition.rs
Normal file
49
crates/ksp-raw-transaction-lib/src/acquisition.rs
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/src/acquisition.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Complete source-neutral RAW transaction acquisition containing one canonical entity and one producer-owned observation.
|
||||||
|
pub struct RawTransactionAcquisition {
|
||||||
|
observation: ksp_store_api::RawTransactionObservation,
|
||||||
|
transaction: ksp_store_api::RawTransaction,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionAcquisition {
|
||||||
|
/// Returns the producer-owned acquisition observation linked to the canonical transaction.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation(&self) -> &ksp_store_api::RawTransactionObservation {
|
||||||
|
return &self.observation;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the canonical source-independent RAW transaction.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn transaction(&self) -> &ksp_store_api::RawTransaction {
|
||||||
|
return &self.transaction;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Consumes this acquisition into its canonical transaction and observation.
|
||||||
|
#[must_use]
|
||||||
|
pub fn into_parts(self) -> (ksp_store_api::RawTransaction, ksp_store_api::RawTransactionObservation) {
|
||||||
|
return (self.transaction, self.observation);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionAcquisition {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.write_str("RawTransactionAcquisition(..)");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Assembles one canonical transaction with a producer-owned observation key and safe acquisition provenance.
|
||||||
|
#[must_use]
|
||||||
|
pub fn assemble_raw_transaction_acquisition(
|
||||||
|
transaction: ksp_store_api::RawTransaction,
|
||||||
|
observation_key: ksp_store_api::RawObservationKey,
|
||||||
|
provenance: ksp_store_api::RawAcquisitionProvenance,
|
||||||
|
) -> crate::RawTransactionAcquisition {
|
||||||
|
let observation = ksp_store_api::RawTransactionObservation::new(observation_key, transaction.reference().clone(), provenance);
|
||||||
|
return crate::RawTransactionAcquisition { observation, transaction };
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/acquisition.rs"]
|
||||||
|
mod tests;
|
||||||
325
crates/ksp-raw-transaction-lib/src/canonical.rs
Normal file
325
crates/ksp-raw-transaction-lib/src/canonical.rs
Normal file
@@ -0,0 +1,325 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/src/canonical.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
|
/// KSP-owned source-independent RAW transaction format identifier.
|
||||||
|
pub const RAW_TRANSACTION_FORMAT_ID: &str = "ksp.solana.raw_transaction";
|
||||||
|
/// Frozen KSP-owned RAW transaction format version.
|
||||||
|
pub const RAW_TRANSACTION_FORMAT_VERSION: u32 = 1;
|
||||||
|
|
||||||
|
/// Source-wire presence state preserving omitted, explicit null and concrete values independently.
|
||||||
|
#[non_exhaustive]
|
||||||
|
pub enum RawTransactionWireField<T> {
|
||||||
|
/// The source field was omitted entirely.
|
||||||
|
Omitted,
|
||||||
|
/// The source field was explicitly present as JSON/protocol null.
|
||||||
|
Null,
|
||||||
|
/// The source field carried one concrete value.
|
||||||
|
Value(T),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<T> std::fmt::Debug for crate::RawTransactionWireField<T> {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return match self {
|
||||||
|
Self::Omitted => formatter.write_str("Omitted"),
|
||||||
|
Self::Null => formatter.write_str("Null"),
|
||||||
|
Self::Value(_) => formatter.write_str("Value(..)"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Canonical Solana transaction version admitted by RAW transaction format v1.
|
||||||
|
#[non_exhaustive]
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub enum RawTransactionVersion {
|
||||||
|
/// Legacy unversioned transaction representation.
|
||||||
|
Legacy,
|
||||||
|
/// Explicit versioned transaction number admitted by the current Solana wire contract.
|
||||||
|
Number(u8),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete source-neutral material required to construct one canonical RAW transaction.
|
||||||
|
pub struct RawTransactionMaterial {
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
meta: crate::RawTransactionWireField<serde_json::Value>,
|
||||||
|
network: ksp_store_api::RawNetworkId,
|
||||||
|
signature: ksp_store_api::RawTransactionSignature,
|
||||||
|
slot: u64,
|
||||||
|
transaction_data: std::string::String,
|
||||||
|
transaction_index: crate::RawTransactionWireField<u32>,
|
||||||
|
version: crate::RawTransactionWireField<crate::RawTransactionVersion>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionMaterial {
|
||||||
|
/// Creates complete source-neutral material for the frozen RAW v1 binary Base64 body path.
|
||||||
|
#[must_use]
|
||||||
|
pub fn binary_base64(
|
||||||
|
network: ksp_store_api::RawNetworkId,
|
||||||
|
signature: ksp_store_api::RawTransactionSignature,
|
||||||
|
slot: u64,
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
transaction_data: impl std::convert::Into<std::string::String>,
|
||||||
|
meta: crate::RawTransactionWireField<serde_json::Value>,
|
||||||
|
version: crate::RawTransactionWireField<crate::RawTransactionVersion>,
|
||||||
|
transaction_index: crate::RawTransactionWireField<u32>,
|
||||||
|
) -> Self {
|
||||||
|
return Self {
|
||||||
|
block_time,
|
||||||
|
meta,
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
slot,
|
||||||
|
transaction_data: transaction_data.into(),
|
||||||
|
transaction_index,
|
||||||
|
version,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates complete source-neutral RAW v1 material from a Base64 transaction wire that embeds its own canonical signature array.
|
||||||
|
pub fn binary_base64_with_embedded_signature(
|
||||||
|
network: ksp_store_api::RawNetworkId,
|
||||||
|
slot: u64,
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
transaction_data: impl std::convert::Into<std::string::String>,
|
||||||
|
meta: crate::RawTransactionWireField<serde_json::Value>,
|
||||||
|
version: crate::RawTransactionWireField<crate::RawTransactionVersion>,
|
||||||
|
transaction_index: crate::RawTransactionWireField<u32>,
|
||||||
|
) -> ksp_core_lib::Result<Self> {
|
||||||
|
let transaction_data = transaction_data.into();
|
||||||
|
let signature = crate::extract_raw_transaction_signature_from_binary_base64(transaction_data.as_str());
|
||||||
|
return match signature {
|
||||||
|
std::result::Result::Ok(signature) => {
|
||||||
|
std::result::Result::Ok(Self { block_time, meta, network, signature, slot, transaction_data, transaction_index, version })
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionMaterial {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionMaterial")
|
||||||
|
.field("network", &self.network)
|
||||||
|
.field("slot", &self.slot)
|
||||||
|
.field("block_time", &self.block_time)
|
||||||
|
.field("transaction_data_len", &self.transaction_data.len())
|
||||||
|
.field("meta", &wire_field_state(&self.meta))
|
||||||
|
.field("version", &wire_field_state(&self.version))
|
||||||
|
.field("transaction_index", &wire_field_state(&self.transaction_index))
|
||||||
|
.finish_non_exhaustive();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Canonicalizes complete source-neutral transaction material into the frozen KSP RAW transaction v1 representation.
|
||||||
|
pub fn canonicalize_raw_transaction(material: crate::RawTransactionMaterial) -> ksp_core_lib::Result<ksp_store_api::RawTransaction> {
|
||||||
|
let block_time = convert_block_time(material.block_time);
|
||||||
|
let block_time = match block_time {
|
||||||
|
std::result::Result::Ok(block_time) => block_time,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let bytes = canonical_payload_bytes(&material);
|
||||||
|
let bytes = match bytes {
|
||||||
|
std::result::Result::Ok(bytes) => bytes,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if bytes.is_empty() || bytes.len() > ksp_store_api::MAX_RAW_PAYLOAD_BYTES {
|
||||||
|
return std::result::Result::Err(crate::canonicalization_error("payload.bytes"));
|
||||||
|
}
|
||||||
|
let hash: [u8; 32] = sha2::Sha256::digest(bytes.as_slice()).into();
|
||||||
|
let format_id = ksp_store_api::RawFormatId::new(crate::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
let format_id = match format_id {
|
||||||
|
std::result::Result::Ok(format_id) => format_id,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::canonicalization_error("payload.format_id")),
|
||||||
|
};
|
||||||
|
let payload = ksp_store_api::RawPayload::try_new(
|
||||||
|
format_id,
|
||||||
|
crate::RAW_TRANSACTION_FORMAT_VERSION,
|
||||||
|
bytes.into_boxed_slice(),
|
||||||
|
ksp_store_api::RawContentHash::new(hash),
|
||||||
|
);
|
||||||
|
let payload = match payload {
|
||||||
|
std::result::Result::Ok(payload) => payload,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::canonicalization_error("payload")),
|
||||||
|
};
|
||||||
|
let reference = ksp_store_api::RawTransactionReference::new(material.network, material.signature);
|
||||||
|
return std::result::Result::Ok(ksp_store_api::RawTransaction::new(reference, material.slot, block_time, payload));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_canonical_json(output: &mut std::vec::Vec<u8>, value: &serde_json::Value) -> ksp_core_lib::Result<()> {
|
||||||
|
return match value {
|
||||||
|
serde_json::Value::Null => {
|
||||||
|
output.extend_from_slice(b"null");
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::Bool(value) => {
|
||||||
|
if *value {
|
||||||
|
output.extend_from_slice(b"true");
|
||||||
|
} else {
|
||||||
|
output.extend_from_slice(b"false");
|
||||||
|
}
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::Number(value) => {
|
||||||
|
output.extend_from_slice(value.to_string().as_bytes());
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::String(value) => append_json_string(output, value.as_str()),
|
||||||
|
serde_json::Value::Array(values) => {
|
||||||
|
output.push(b'[');
|
||||||
|
for (index, item) in values.iter().enumerate() {
|
||||||
|
if index != 0 {
|
||||||
|
output.push(b',');
|
||||||
|
}
|
||||||
|
let item_result = append_canonical_json(output, item);
|
||||||
|
if let std::result::Result::Err(error) = item_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output.push(b']');
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::Object(values) => {
|
||||||
|
output.push(b'{');
|
||||||
|
let mut keys = values.keys().collect::<std::vec::Vec<_>>();
|
||||||
|
keys.sort_unstable();
|
||||||
|
for (index, key) in keys.iter().enumerate() {
|
||||||
|
if index != 0 {
|
||||||
|
output.push(b',');
|
||||||
|
}
|
||||||
|
let key_result = append_json_string(output, key.as_str());
|
||||||
|
if let std::result::Result::Err(error) = key_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.push(b':');
|
||||||
|
let item = values.get(key.as_str());
|
||||||
|
let item = match item {
|
||||||
|
std::option::Option::Some(item) => item,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::canonicalization_error("payload.meta")),
|
||||||
|
};
|
||||||
|
let item_result = append_canonical_json(output, item);
|
||||||
|
if let std::result::Result::Err(error) = item_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output.push(b'}');
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_json_string(output: &mut std::vec::Vec<u8>, value: &str) -> ksp_core_lib::Result<()> {
|
||||||
|
let encoded = serde_json::to_vec(value);
|
||||||
|
return match encoded {
|
||||||
|
std::result::Result::Ok(encoded) => {
|
||||||
|
output.extend_from_slice(encoded.as_slice());
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(crate::canonicalization_error("payload.json")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_wire_value<T, F>(
|
||||||
|
output: &mut std::vec::Vec<u8>,
|
||||||
|
key: &str,
|
||||||
|
field: &crate::RawTransactionWireField<T>,
|
||||||
|
mut append_value: F,
|
||||||
|
) -> ksp_core_lib::Result<()>
|
||||||
|
where
|
||||||
|
F: FnMut(&mut std::vec::Vec<u8>, &T) -> ksp_core_lib::Result<()>,
|
||||||
|
{
|
||||||
|
return match field {
|
||||||
|
crate::RawTransactionWireField::Omitted => std::result::Result::Ok(()),
|
||||||
|
crate::RawTransactionWireField::Null => {
|
||||||
|
output.push(b',');
|
||||||
|
let key_result = append_json_string(output, key);
|
||||||
|
if let std::result::Result::Err(error) = key_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.extend_from_slice(b":null");
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
crate::RawTransactionWireField::Value(value) => {
|
||||||
|
output.push(b',');
|
||||||
|
let key_result = append_json_string(output, key);
|
||||||
|
if let std::result::Result::Err(error) = key_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.push(b':');
|
||||||
|
append_value(output, value)
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonical_payload_bytes(material: &crate::RawTransactionMaterial) -> ksp_core_lib::Result<std::vec::Vec<u8>> {
|
||||||
|
let mut bytes = std::vec::Vec::new();
|
||||||
|
bytes.extend_from_slice(b"{\"transaction\":[");
|
||||||
|
let transaction_result = append_json_string(&mut bytes, material.transaction_data.as_str());
|
||||||
|
if let std::result::Result::Err(error) = transaction_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
bytes.push(b',');
|
||||||
|
let encoding_result = append_json_string(&mut bytes, "base64");
|
||||||
|
if let std::result::Result::Err(error) = encoding_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
bytes.push(b']');
|
||||||
|
let meta_result = append_wire_value(&mut bytes, "meta", &material.meta, |output, value| return append_canonical_json(output, value));
|
||||||
|
if let std::result::Result::Err(error) = meta_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let version_result = append_wire_value(&mut bytes, "version", &material.version, |output, value| {
|
||||||
|
return match value {
|
||||||
|
crate::RawTransactionVersion::Legacy => append_json_string(output, "legacy"),
|
||||||
|
crate::RawTransactionVersion::Number(number) => {
|
||||||
|
output.extend_from_slice(number.to_string().as_bytes());
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
};
|
||||||
|
});
|
||||||
|
if let std::result::Result::Err(error) = version_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let transaction_index_result = append_wire_value(&mut bytes, "transactionIndex", &material.transaction_index, |output, value| {
|
||||||
|
output.extend_from_slice(value.to_string().as_bytes());
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
if let std::result::Result::Err(error) = transaction_index_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
bytes.push(b'}');
|
||||||
|
return std::result::Result::Ok(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn convert_block_time(value: std::option::Option<i64>) -> ksp_core_lib::Result<std::option::Option<ksp_store_api::RawTimestamp>> {
|
||||||
|
let seconds = match value {
|
||||||
|
std::option::Option::Some(seconds) => seconds,
|
||||||
|
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
||||||
|
};
|
||||||
|
let seconds = match u64::try_from(seconds) {
|
||||||
|
std::result::Result::Ok(seconds) => seconds,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::material_error("block_time")),
|
||||||
|
};
|
||||||
|
let millis = match seconds.checked_mul(1_000) {
|
||||||
|
std::option::Option::Some(millis) => millis,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::material_error("block_time")),
|
||||||
|
};
|
||||||
|
let timestamp = ksp_store_api::RawTimestamp::from_unix_millis(millis);
|
||||||
|
return match timestamp {
|
||||||
|
std::result::Result::Ok(timestamp) => std::result::Result::Ok(std::option::Option::Some(timestamp)),
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(crate::material_error("block_time")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wire_field_state<T>(field: &crate::RawTransactionWireField<T>) -> &'static str {
|
||||||
|
return match field {
|
||||||
|
crate::RawTransactionWireField::Omitted => "omitted",
|
||||||
|
crate::RawTransactionWireField::Null => "null",
|
||||||
|
crate::RawTransactionWireField::Value(_) => "value",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/canonical.rs"]
|
||||||
|
mod tests;
|
||||||
27
crates/ksp-raw-transaction-lib/src/error.rs
Normal file
27
crates/ksp-raw-transaction-lib/src/error.rs
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/src/error.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
/// Error code used when RAW v1 canonical payload construction cannot preserve the frozen format contract.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_CANONICALIZATION_INVALID: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("raw_transaction", "canonicalization_invalid");
|
||||||
|
/// Error code used when source-neutral RAW transaction material violates a bounded semantic invariant.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_MATERIAL_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("raw_transaction", "material_invalid");
|
||||||
|
/// Error code used when Solana transaction signature material cannot resolve to exactly 64 canonical bytes.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("raw_transaction", "signature_invalid");
|
||||||
|
|
||||||
|
/// Creates a safe canonicalization error without copying source payload material.
|
||||||
|
pub(crate) fn canonicalization_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_CANONICALIZATION_INVALID, "invalid canonical RAW transaction representation")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates a safe source-material error without copying source payload material.
|
||||||
|
pub(crate) fn material_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_MATERIAL_INVALID, "invalid source-neutral RAW transaction material")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates a safe signature error without copying source signature or transaction material.
|
||||||
|
pub(crate) fn signature_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID, "invalid canonical Solana transaction signature");
|
||||||
|
}
|
||||||
77
crates/ksp-raw-transaction-lib/src/lib.rs
Normal file
77
crates/ksp-raw-transaction-lib/src/lib.rs
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/src/lib.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
#![deny(unreachable_pub)]
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
|
//! Source-neutral canonicalization of complete Solana transactions into KSP RAW transaction v1.
|
||||||
|
//!
|
||||||
|
//! This lower layer owns the frozen RAW transaction format, textual signature parsing/formatting, canonical
|
||||||
|
//! JSON construction, content hashing and assembly of one canonical transaction with one
|
||||||
|
//! producer-owned observation. It owns no Transport, Config, Job, Worker, async runtime or Store
|
||||||
|
//! backend behavior.
|
||||||
|
|
||||||
|
mod acquisition;
|
||||||
|
mod canonical;
|
||||||
|
mod error;
|
||||||
|
mod signature;
|
||||||
|
mod wire;
|
||||||
|
|
||||||
|
/// Complete source-neutral RAW transaction acquisition containing one canonical entity and one producer-owned observation.
|
||||||
|
pub use self::acquisition::RawTransactionAcquisition;
|
||||||
|
/// Assembles one canonical transaction with a producer-owned observation key and safe acquisition provenance.
|
||||||
|
pub use self::acquisition::assemble_raw_transaction_acquisition;
|
||||||
|
/// KSP-owned source-independent RAW transaction format identifier.
|
||||||
|
pub use self::canonical::RAW_TRANSACTION_FORMAT_ID;
|
||||||
|
/// Frozen KSP-owned RAW transaction format version.
|
||||||
|
pub use self::canonical::RAW_TRANSACTION_FORMAT_VERSION;
|
||||||
|
/// Complete source-neutral material required to construct one canonical RAW transaction.
|
||||||
|
pub use self::canonical::RawTransactionMaterial;
|
||||||
|
/// Canonical Solana transaction version admitted by RAW transaction format v1.
|
||||||
|
pub use self::canonical::RawTransactionVersion;
|
||||||
|
/// Source-wire presence state preserving omitted, explicit null and concrete values independently.
|
||||||
|
pub use self::canonical::RawTransactionWireField;
|
||||||
|
/// Canonicalizes complete source-neutral transaction material into the frozen KSP RAW transaction v1 representation.
|
||||||
|
pub use self::canonical::canonicalize_raw_transaction;
|
||||||
|
/// Error code used when RAW v1 canonical payload construction cannot preserve the frozen format contract.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_CANONICALIZATION_INVALID;
|
||||||
|
/// Error code used when source-neutral RAW transaction material violates a bounded semantic invariant.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_MATERIAL_INVALID;
|
||||||
|
/// Error code used when Solana transaction signature material cannot resolve to exactly 64 canonical bytes.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID;
|
||||||
|
/// Maximum UTF-8 byte length admitted for one textual Base58 Solana transaction signature.
|
||||||
|
pub use self::signature::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES;
|
||||||
|
/// Minimum UTF-8 byte length admitted for one textual Base58 Solana transaction signature.
|
||||||
|
pub use self::signature::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES;
|
||||||
|
/// Extracts the first canonical 64-byte Solana signature from one complete Base64 transaction wire.
|
||||||
|
pub use self::signature::extract_raw_transaction_signature_from_binary_base64;
|
||||||
|
/// Encodes one canonical 64-byte Solana transaction signature as bounded Base58 text.
|
||||||
|
pub use self::signature::format_raw_transaction_signature;
|
||||||
|
/// Parses one bounded Base58 Solana transaction signature to exactly 64 canonical bytes.
|
||||||
|
pub use self::signature::parse_raw_transaction_signature;
|
||||||
|
/// One source-neutral v0 address-table lookup.
|
||||||
|
pub use self::wire::RawSolanaAddressTableLookup;
|
||||||
|
/// One source-neutral compiled Solana instruction.
|
||||||
|
pub use self::wire::RawSolanaCompiledInstruction;
|
||||||
|
/// Source-neutral Solana message header.
|
||||||
|
pub use self::wire::RawSolanaMessageHeader;
|
||||||
|
/// Source-neutral Solana transaction message version.
|
||||||
|
pub use self::wire::RawSolanaMessageVersion;
|
||||||
|
/// Optional source-neutral Transaction V1 inline configuration.
|
||||||
|
pub use self::wire::RawSolanaTransactionConfig;
|
||||||
|
/// Complete source-neutral Solana transaction message.
|
||||||
|
pub use self::wire::RawSolanaTransactionMessage;
|
||||||
|
/// Complete source-neutral Solana transaction wire material.
|
||||||
|
pub use self::wire::RawSolanaTransactionWire;
|
||||||
|
/// Serializes one source-neutral Solana transaction to exact canonical wire bytes.
|
||||||
|
pub use self::wire::serialize_solana_transaction_wire;
|
||||||
|
/// Serializes one source-neutral Solana transaction to canonical padded standard Base64.
|
||||||
|
pub use self::wire::serialize_solana_transaction_wire_base64;
|
||||||
|
|
||||||
|
/// Creates a safe canonicalization error without copying source payload material.
|
||||||
|
pub(crate) use self::error::canonicalization_error;
|
||||||
|
/// Creates a safe source-material error without copying source payload material.
|
||||||
|
pub(crate) use self::error::material_error;
|
||||||
|
/// Creates a safe signature error without copying textual signature material.
|
||||||
|
pub(crate) use self::error::signature_error;
|
||||||
319
crates/ksp-raw-transaction-lib/src/signature.rs
Normal file
319
crates/ksp-raw-transaction-lib/src/signature.rs
Normal file
@@ -0,0 +1,319 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/src/signature.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
use base64::Engine; // rust-rules: trait-import
|
||||||
|
|
||||||
|
/// Maximum UTF-8 byte length admitted for one textual Base58 Solana transaction signature.
|
||||||
|
pub const MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES: usize = 88;
|
||||||
|
/// Minimum UTF-8 byte length admitted for one textual Base58 Solana transaction signature.
|
||||||
|
pub const MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES: usize = 64;
|
||||||
|
|
||||||
|
/// Encodes one canonical 64-byte Solana transaction signature as bounded Base58 text.
|
||||||
|
#[must_use]
|
||||||
|
pub fn format_raw_transaction_signature(signature: &ksp_store_api::RawTransactionSignature) -> std::string::String {
|
||||||
|
const ALPHABET: &[u8; 58] = b"123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";
|
||||||
|
let bytes = signature.as_bytes();
|
||||||
|
let leading_zeroes = bytes.iter().take_while(|byte| return **byte == 0).count();
|
||||||
|
let mut digits = std::vec::Vec::with_capacity(crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES);
|
||||||
|
for byte in bytes {
|
||||||
|
let mut carry = u32::from(*byte);
|
||||||
|
for digit in &mut digits {
|
||||||
|
let expanded = (u32::from(*digit) * 256) + carry;
|
||||||
|
*digit = (expanded % 58) as u8;
|
||||||
|
carry = expanded / 58;
|
||||||
|
}
|
||||||
|
while carry != 0 {
|
||||||
|
digits.push((carry % 58) as u8);
|
||||||
|
carry /= 58;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut output = std::string::String::with_capacity(leading_zeroes + digits.len());
|
||||||
|
for _ in 0..leading_zeroes {
|
||||||
|
output.push('1');
|
||||||
|
}
|
||||||
|
for digit in digits.iter().rev() {
|
||||||
|
output.push(char::from(ALPHABET[usize::from(*digit)]));
|
||||||
|
}
|
||||||
|
return output;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parses one bounded Base58 Solana transaction signature to exactly 64 canonical bytes.
|
||||||
|
pub fn parse_raw_transaction_signature(value: &str) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
|
let text = value.as_bytes();
|
||||||
|
if text.len() < crate::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES || text.len() > crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES {
|
||||||
|
return std::result::Result::Err(
|
||||||
|
crate::signature_error()
|
||||||
|
.with_context("actual_len", text.len().to_string())
|
||||||
|
.with_context("minimum_len", crate::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES.to_string())
|
||||||
|
.with_context("maximum_len", crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES.to_string()),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let mut decoded = [0_u8; 64];
|
||||||
|
let mut leading_zeroes = 0_usize;
|
||||||
|
for byte in text {
|
||||||
|
if *byte != b'1' {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
leading_zeroes += 1;
|
||||||
|
}
|
||||||
|
for byte in text {
|
||||||
|
let digit = match base58_digit(*byte) {
|
||||||
|
std::option::Option::Some(digit) => digit,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let mut carry = u32::from(digit);
|
||||||
|
for output in decoded.iter_mut().rev() {
|
||||||
|
let expanded = (u32::from(*output) * 58) + carry;
|
||||||
|
*output = (expanded & 0xff) as u8;
|
||||||
|
carry = expanded >> 8;
|
||||||
|
}
|
||||||
|
if carry != 0 {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let significant_len = match decoded.iter().position(|byte| return *byte != 0) {
|
||||||
|
std::option::Option::Some(index) => decoded.len() - index,
|
||||||
|
std::option::Option::None => 0,
|
||||||
|
};
|
||||||
|
if leading_zeroes + significant_len != decoded.len() {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(ksp_store_api::RawTransactionSignature::new(decoded));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extracts the first canonical 64-byte Solana signature from one complete Base64-encoded transaction wire.
|
||||||
|
///
|
||||||
|
/// The compact signature-count prefix must use its canonical short-vector representation, contain at least one signature,
|
||||||
|
/// and the decoded transaction must retain message bytes after the declared signature array.
|
||||||
|
pub fn extract_raw_transaction_signature_from_binary_base64(value: &str) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
|
if value.is_empty() || value.len() > ksp_store_api::MAX_RAW_PAYLOAD_BYTES {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
let decoded = base64::engine::general_purpose::STANDARD.decode(value);
|
||||||
|
let decoded = match decoded {
|
||||||
|
std::result::Result::Ok(decoded) => decoded,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
if base64::engine::general_purpose::STANDARD.encode(decoded.as_slice()) != value {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
if decoded.first() == std::option::Option::Some(&0x81) {
|
||||||
|
return extract_v1_signature(decoded.as_slice());
|
||||||
|
}
|
||||||
|
return extract_legacy_or_v0_signature(decoded.as_slice());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn extract_legacy_or_v0_signature(decoded: &[u8]) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
|
let prefix = decode_signature_count(decoded);
|
||||||
|
let (signature_count, prefix_len) = match prefix {
|
||||||
|
std::result::Result::Ok(prefix) => prefix,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if signature_count == 0 {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
let signatures_len = match signature_count.checked_mul(64) {
|
||||||
|
std::option::Option::Some(length) => length,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let message_offset = match prefix_len.checked_add(signatures_len) {
|
||||||
|
std::option::Option::Some(offset) => offset,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
if message_offset >= decoded.len() {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
return copy_signature(decoded, prefix_len);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn extract_v1_signature(decoded: &[u8]) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
|
if decoded.len() > 4_096 || decoded.len() < 42 {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
let required_signatures = match decoded.get(1) {
|
||||||
|
std::option::Option::Some(value) => usize::from(*value),
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let readonly_signed = match decoded.get(2) {
|
||||||
|
std::option::Option::Some(value) => usize::from(*value),
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let readonly_unsigned = match decoded.get(3) {
|
||||||
|
std::option::Option::Some(value) => usize::from(*value),
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
if required_signatures == 0 || required_signatures > 12 || readonly_signed >= required_signatures {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
let mask_bytes = match decoded.get(4..8) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let mask = u32::from_le_bytes([mask_bytes[0], mask_bytes[1], mask_bytes[2], mask_bytes[3]]);
|
||||||
|
if mask & !0x1f != 0 || matches!(mask & 0b11, 1 | 2) {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
let instruction_count = match decoded.get(40) {
|
||||||
|
std::option::Option::Some(value) => usize::from(*value),
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let address_count = match decoded.get(41) {
|
||||||
|
std::option::Option::Some(value) => usize::from(*value),
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
if instruction_count > 64 || address_count > 64 || address_count < required_signatures + readonly_unsigned {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
let addresses_len = match address_count.checked_mul(32) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let mut offset = match 42_usize.checked_add(addresses_len) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let config_len = config_value_length(mask);
|
||||||
|
offset = match offset.checked_add(config_len) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let headers_len = match instruction_count.checked_mul(4) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let headers_end = match offset.checked_add(headers_len) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
if headers_end > decoded.len() {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
let mut payload_len = 0_usize;
|
||||||
|
for index in 0..instruction_count {
|
||||||
|
let header_offset = offset + (index * 4);
|
||||||
|
let program_index = usize::from(decoded[header_offset]);
|
||||||
|
let account_index_count = usize::from(decoded[header_offset + 1]);
|
||||||
|
let data_len = u16::from_le_bytes([decoded[header_offset + 2], decoded[header_offset + 3]]);
|
||||||
|
if program_index >= address_count {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
payload_len = match payload_len.checked_add(account_index_count).and_then(|value| return value.checked_add(usize::from(data_len))) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
let payload_end = match headers_end.checked_add(payload_len) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let payload = match decoded.get(headers_end..payload_end) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let mut payload_offset = 0_usize;
|
||||||
|
for index in 0..instruction_count {
|
||||||
|
let header_offset = offset + (index * 4);
|
||||||
|
let account_index_count = usize::from(decoded[header_offset + 1]);
|
||||||
|
let data_len = usize::from(u16::from_le_bytes([decoded[header_offset + 2], decoded[header_offset + 3]]));
|
||||||
|
let account_end = match payload_offset.checked_add(account_index_count) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let account_indexes = match payload.get(payload_offset..account_end) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
if account_indexes.iter().any(|value| return usize::from(*value) >= address_count) {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
payload_offset = match account_end.checked_add(data_len) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
let signatures_len = match required_signatures.checked_mul(64) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let expected_end = match payload_end.checked_add(signatures_len) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
if expected_end != decoded.len() {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
return copy_signature(decoded, payload_end);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn config_value_length(mask: u32) -> usize {
|
||||||
|
let mut length = 0_usize;
|
||||||
|
if mask & 0b11 == 0b11 {
|
||||||
|
length += 8;
|
||||||
|
}
|
||||||
|
if mask & (1_u32 << 2) != 0 {
|
||||||
|
length += 4;
|
||||||
|
}
|
||||||
|
if mask & (1_u32 << 3) != 0 {
|
||||||
|
length += 4;
|
||||||
|
}
|
||||||
|
if mask & (1_u32 << 4) != 0 {
|
||||||
|
length += 4;
|
||||||
|
}
|
||||||
|
return length;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn copy_signature(decoded: &[u8], offset: usize) -> ksp_core_lib::Result<ksp_store_api::RawTransactionSignature> {
|
||||||
|
let first_end = match offset.checked_add(64) {
|
||||||
|
std::option::Option::Some(end) => end,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let first = match decoded.get(offset..first_end) {
|
||||||
|
std::option::Option::Some(first) => first,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let mut signature = [0_u8; 64];
|
||||||
|
signature.copy_from_slice(first);
|
||||||
|
return std::result::Result::Ok(ksp_store_api::RawTransactionSignature::new(signature));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn decode_signature_count(bytes: &[u8]) -> ksp_core_lib::Result<(usize, usize)> {
|
||||||
|
let mut value = 0_usize;
|
||||||
|
for index in 0..3_usize {
|
||||||
|
let byte = match bytes.get(index) {
|
||||||
|
std::option::Option::Some(byte) => *byte,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::signature_error()),
|
||||||
|
};
|
||||||
|
let payload = usize::from(byte & 0x7f);
|
||||||
|
let shift = index * 7;
|
||||||
|
value |= payload << shift;
|
||||||
|
if byte & 0x80 == 0 {
|
||||||
|
if index != 0 {
|
||||||
|
let minimum = 1_usize << shift;
|
||||||
|
if value < minimum {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if value > usize::from(u16::MAX) {
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok((value, index + 1));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(crate::signature_error());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn base58_digit(byte: u8) -> std::option::Option<u8> {
|
||||||
|
return match byte {
|
||||||
|
b'1'..=b'9' => std::option::Option::Some(byte - b'1'),
|
||||||
|
b'A'..=b'H' => std::option::Option::Some((byte - b'A') + 9),
|
||||||
|
b'J'..=b'N' => std::option::Option::Some((byte - b'J') + 17),
|
||||||
|
b'P'..=b'Z' => std::option::Option::Some((byte - b'P') + 22),
|
||||||
|
b'a'..=b'k' => std::option::Option::Some((byte - b'a') + 33),
|
||||||
|
b'm'..=b'z' => std::option::Option::Some((byte - b'm') + 44),
|
||||||
|
_ => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/signature.rs"]
|
||||||
|
mod tests;
|
||||||
545
crates/ksp-raw-transaction-lib/src/wire.rs
Normal file
545
crates/ksp-raw-transaction-lib/src/wire.rs
Normal file
@@ -0,0 +1,545 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/src/wire.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
use base64::Engine; // rust-rules: trait-import
|
||||||
|
|
||||||
|
const MAX_LEGACY_SHORT_VECTOR_VALUE: usize = u16::MAX as usize;
|
||||||
|
const MAX_V1_ADDRESS_COUNT: usize = 64;
|
||||||
|
const MAX_V1_INSTRUCTION_COUNT: usize = 64;
|
||||||
|
const MAX_V1_SIGNATURE_COUNT: usize = 12;
|
||||||
|
const MAX_V1_TRANSACTION_BYTES: usize = 4_096;
|
||||||
|
|
||||||
|
/// Source-neutral Solana transaction message version used by the exact wire serializer.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub enum RawSolanaMessageVersion {
|
||||||
|
/// Legacy Solana message without a version prefix.
|
||||||
|
Legacy,
|
||||||
|
/// Versioned transaction message v0, prefixed with `0x80`.
|
||||||
|
V0,
|
||||||
|
/// Transaction V1 / SIMD-0385 message, prefixed with `0x81`.
|
||||||
|
V1,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Exact three-byte Solana message header.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub struct RawSolanaMessageHeader {
|
||||||
|
num_required_signatures: u8,
|
||||||
|
num_readonly_signed_accounts: u8,
|
||||||
|
num_readonly_unsigned_accounts: u8,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawSolanaMessageHeader {
|
||||||
|
/// Creates an exact Solana message header.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(num_required_signatures: u8, num_readonly_signed_accounts: u8, num_readonly_unsigned_accounts: u8) -> Self {
|
||||||
|
return Self { num_required_signatures, num_readonly_signed_accounts, num_readonly_unsigned_accounts };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the required signature count.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn num_required_signatures(self) -> u8 {
|
||||||
|
return self.num_required_signatures;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the readonly signed-account count.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn num_readonly_signed_accounts(self) -> u8 {
|
||||||
|
return self.num_readonly_signed_accounts;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the readonly unsigned-account count.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn num_readonly_unsigned_accounts(self) -> u8 {
|
||||||
|
return self.num_readonly_unsigned_accounts;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One source-neutral compiled Solana instruction.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawSolanaCompiledInstruction {
|
||||||
|
program_id_index: u8,
|
||||||
|
accounts: std::vec::Vec<u8>,
|
||||||
|
data: std::vec::Vec<u8>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawSolanaCompiledInstruction {
|
||||||
|
/// Creates one compiled instruction while preserving exact ordered account indexes and data bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub fn new(program_id_index: u8, accounts: std::vec::Vec<u8>, data: std::vec::Vec<u8>) -> Self {
|
||||||
|
return Self { program_id_index, accounts, data };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the program account index.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn program_id_index(&self) -> u8 {
|
||||||
|
return self.program_id_index;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns ordered account indexes.
|
||||||
|
#[must_use]
|
||||||
|
pub fn accounts(&self) -> &[u8] {
|
||||||
|
return self.accounts.as_slice();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns exact instruction data bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub fn data(&self) -> &[u8] {
|
||||||
|
return self.data.as_slice();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawSolanaCompiledInstruction {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawSolanaCompiledInstruction")
|
||||||
|
.field("program_id_index", &self.program_id_index)
|
||||||
|
.field("account_index_count", &self.accounts.len())
|
||||||
|
.field("data_length", &self.data.len())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One source-neutral v0 address-table lookup.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawSolanaAddressTableLookup {
|
||||||
|
account_key: [u8; 32],
|
||||||
|
writable_indexes: std::vec::Vec<u8>,
|
||||||
|
readonly_indexes: std::vec::Vec<u8>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawSolanaAddressTableLookup {
|
||||||
|
/// Creates one exact v0 address-table lookup.
|
||||||
|
#[must_use]
|
||||||
|
pub fn new(account_key: [u8; 32], writable_indexes: std::vec::Vec<u8>, readonly_indexes: std::vec::Vec<u8>) -> Self {
|
||||||
|
return Self { account_key, writable_indexes, readonly_indexes };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the lookup-table account key bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn account_key(&self) -> &[u8; 32] {
|
||||||
|
return &self.account_key;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns ordered writable lookup indexes.
|
||||||
|
#[must_use]
|
||||||
|
pub fn writable_indexes(&self) -> &[u8] {
|
||||||
|
return self.writable_indexes.as_slice();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns ordered readonly lookup indexes.
|
||||||
|
#[must_use]
|
||||||
|
pub fn readonly_indexes(&self) -> &[u8] {
|
||||||
|
return self.readonly_indexes.as_slice();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawSolanaAddressTableLookup {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawSolanaAddressTableLookup")
|
||||||
|
.field("writable_index_count", &self.writable_indexes.len())
|
||||||
|
.field("readonly_index_count", &self.readonly_indexes.len())
|
||||||
|
.finish_non_exhaustive();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Optional inline budget configuration carried by Solana Transaction V1.
|
||||||
|
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
|
||||||
|
pub struct RawSolanaTransactionConfig {
|
||||||
|
priority_fee: std::option::Option<u64>,
|
||||||
|
compute_unit_limit: std::option::Option<u32>,
|
||||||
|
loaded_accounts_data_size_limit: std::option::Option<u32>,
|
||||||
|
heap_size: std::option::Option<u32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawSolanaTransactionConfig {
|
||||||
|
/// Creates an explicit Transaction V1 configuration, including the meaningful all-`None` case.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(
|
||||||
|
priority_fee: std::option::Option<u64>,
|
||||||
|
compute_unit_limit: std::option::Option<u32>,
|
||||||
|
loaded_accounts_data_size_limit: std::option::Option<u32>,
|
||||||
|
heap_size: std::option::Option<u32>,
|
||||||
|
) -> Self {
|
||||||
|
return Self { priority_fee, compute_unit_limit, loaded_accounts_data_size_limit, heap_size };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the optional priority fee.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn priority_fee(self) -> std::option::Option<u64> {
|
||||||
|
return self.priority_fee;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the optional compute-unit limit.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn compute_unit_limit(self) -> std::option::Option<u32> {
|
||||||
|
return self.compute_unit_limit;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the optional loaded-account-data-size limit.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn loaded_accounts_data_size_limit(self) -> std::option::Option<u32> {
|
||||||
|
return self.loaded_accounts_data_size_limit;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the optional heap-size override.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn heap_size(self) -> std::option::Option<u32> {
|
||||||
|
return self.heap_size;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete source-neutral Solana transaction message required for exact wire serialization.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawSolanaTransactionMessage {
|
||||||
|
version: crate::RawSolanaMessageVersion,
|
||||||
|
header: crate::RawSolanaMessageHeader,
|
||||||
|
account_keys: std::vec::Vec<[u8; 32]>,
|
||||||
|
recent_blockhash: [u8; 32],
|
||||||
|
instructions: std::vec::Vec<crate::RawSolanaCompiledInstruction>,
|
||||||
|
address_table_lookups: std::vec::Vec<crate::RawSolanaAddressTableLookup>,
|
||||||
|
config: std::option::Option<crate::RawSolanaTransactionConfig>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawSolanaTransactionMessage {
|
||||||
|
/// Creates one exact source-neutral message.
|
||||||
|
#[must_use]
|
||||||
|
pub fn new(
|
||||||
|
version: crate::RawSolanaMessageVersion,
|
||||||
|
header: crate::RawSolanaMessageHeader,
|
||||||
|
account_keys: std::vec::Vec<[u8; 32]>,
|
||||||
|
recent_blockhash: [u8; 32],
|
||||||
|
instructions: std::vec::Vec<crate::RawSolanaCompiledInstruction>,
|
||||||
|
address_table_lookups: std::vec::Vec<crate::RawSolanaAddressTableLookup>,
|
||||||
|
config: std::option::Option<crate::RawSolanaTransactionConfig>,
|
||||||
|
) -> Self {
|
||||||
|
return Self { version, header, account_keys, recent_blockhash, instructions, address_table_lookups, config };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the message version.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn version(&self) -> crate::RawSolanaMessageVersion {
|
||||||
|
return self.version;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the message header.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn header(&self) -> crate::RawSolanaMessageHeader {
|
||||||
|
return self.header;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns ordered static account key bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub fn account_keys(&self) -> &[[u8; 32]] {
|
||||||
|
return self.account_keys.as_slice();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the recent blockhash bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn recent_blockhash(&self) -> &[u8; 32] {
|
||||||
|
return &self.recent_blockhash;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns ordered compiled instructions.
|
||||||
|
#[must_use]
|
||||||
|
pub fn instructions(&self) -> &[crate::RawSolanaCompiledInstruction] {
|
||||||
|
return self.instructions.as_slice();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns ordered v0 address-table lookups.
|
||||||
|
#[must_use]
|
||||||
|
pub fn address_table_lookups(&self) -> &[crate::RawSolanaAddressTableLookup] {
|
||||||
|
return self.address_table_lookups.as_slice();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns optional V1 inline configuration.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn config(&self) -> std::option::Option<crate::RawSolanaTransactionConfig> {
|
||||||
|
return self.config;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawSolanaTransactionMessage {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawSolanaTransactionMessage")
|
||||||
|
.field("version", &self.version)
|
||||||
|
.field("header", &self.header)
|
||||||
|
.field("account_key_count", &self.account_keys.len())
|
||||||
|
.field("instruction_count", &self.instructions.len())
|
||||||
|
.field("address_table_lookup_count", &self.address_table_lookups.len())
|
||||||
|
.field("has_config", &self.config.is_some())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete source-neutral Solana transaction wire material.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawSolanaTransactionWire {
|
||||||
|
signatures: std::vec::Vec<[u8; 64]>,
|
||||||
|
message: crate::RawSolanaTransactionMessage,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawSolanaTransactionWire {
|
||||||
|
/// Creates one complete transaction wire.
|
||||||
|
#[must_use]
|
||||||
|
pub fn new(signatures: std::vec::Vec<[u8; 64]>, message: crate::RawSolanaTransactionMessage) -> Self {
|
||||||
|
return Self { signatures, message };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns ordered signature bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub fn signatures(&self) -> &[[u8; 64]] {
|
||||||
|
return self.signatures.as_slice();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the exact source-neutral message.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn message(&self) -> &crate::RawSolanaTransactionMessage {
|
||||||
|
return &self.message;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawSolanaTransactionWire {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_struct("RawSolanaTransactionWire").field("signature_count", &self.signatures.len()).field("message", &self.message).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Serializes one source-neutral Solana transaction to exact canonical wire bytes.
|
||||||
|
pub fn serialize_solana_transaction_wire(transaction: &crate::RawSolanaTransactionWire) -> ksp_core_lib::Result<std::vec::Vec<u8>> {
|
||||||
|
if let std::result::Result::Err(error) = validate_transaction(transaction) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let mut output = std::vec::Vec::new();
|
||||||
|
match transaction.message().version() {
|
||||||
|
crate::RawSolanaMessageVersion::Legacy | crate::RawSolanaMessageVersion::V0 => {
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(transaction.signatures().len(), &mut output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
for signature in transaction.signatures() {
|
||||||
|
output.extend_from_slice(signature);
|
||||||
|
}
|
||||||
|
if transaction.message().version() == crate::RawSolanaMessageVersion::V0 {
|
||||||
|
output.push(0x80);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = encode_legacy_or_v0_message(transaction.message(), &mut output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
crate::RawSolanaMessageVersion::V1 => {
|
||||||
|
if let std::result::Result::Err(error) = encode_v1_message(transaction, &mut output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if output.len() > ksp_store_api::MAX_RAW_PAYLOAD_BYTES {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire.bytes").with_context("actual_len", output.len().to_string()));
|
||||||
|
}
|
||||||
|
if transaction.message().version() == crate::RawSolanaMessageVersion::V1 && output.len() > MAX_V1_TRANSACTION_BYTES {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire.bytes").with_context("actual_len", output.len().to_string()));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(output);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Serializes one source-neutral Solana transaction and returns canonical padded standard Base64.
|
||||||
|
pub fn serialize_solana_transaction_wire_base64(transaction: &crate::RawSolanaTransactionWire) -> ksp_core_lib::Result<std::string::String> {
|
||||||
|
let bytes = match crate::serialize_solana_transaction_wire(transaction) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(base64::engine::general_purpose::STANDARD.encode(bytes));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_transaction(transaction: &crate::RawSolanaTransactionWire) -> ksp_core_lib::Result<()> {
|
||||||
|
let message = transaction.message();
|
||||||
|
let required_signatures = usize::from(message.header().num_required_signatures());
|
||||||
|
if required_signatures == 0 || transaction.signatures().len() != required_signatures {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
if usize::from(message.header().num_readonly_signed_accounts()) >= required_signatures {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
if message.account_keys().len() < required_signatures + usize::from(message.header().num_readonly_unsigned_accounts()) {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
let loaded_count = message.address_table_lookups().iter().fold(0_usize, |count, lookup| {
|
||||||
|
return count.saturating_add(lookup.writable_indexes().len()).saturating_add(lookup.readonly_indexes().len());
|
||||||
|
});
|
||||||
|
let account_count = message.account_keys().len().saturating_add(loaded_count);
|
||||||
|
for instruction in message.instructions() {
|
||||||
|
if usize::from(instruction.program_id_index()) >= account_count {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
for account_index in instruction.accounts() {
|
||||||
|
if usize::from(*account_index) >= account_count {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
match message.version() {
|
||||||
|
crate::RawSolanaMessageVersion::Legacy => {
|
||||||
|
if !message.address_table_lookups().is_empty() || message.config().is_some() {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
},
|
||||||
|
crate::RawSolanaMessageVersion::V0 => {
|
||||||
|
if message.config().is_some() {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
},
|
||||||
|
crate::RawSolanaMessageVersion::V1 => {
|
||||||
|
if message.config().is_none() || !message.address_table_lookups().is_empty() {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
if transaction.signatures().len() > MAX_V1_SIGNATURE_COUNT
|
||||||
|
|| message.account_keys().len() > MAX_V1_ADDRESS_COUNT
|
||||||
|
|| message.instructions().len() > MAX_V1_INSTRUCTION_COUNT
|
||||||
|
{
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
for (index, account) in message.account_keys().iter().enumerate() {
|
||||||
|
if message.account_keys()[..index].contains(account) {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for instruction in message.instructions() {
|
||||||
|
if instruction.accounts().len() > usize::from(u8::MAX) || instruction.data().len() > usize::from(u16::MAX) {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(heap_size) = message.config().and_then(crate::RawSolanaTransactionConfig::heap_size)
|
||||||
|
&& (!(32_768..=262_144).contains(&heap_size) || heap_size % 1_024 != 0)
|
||||||
|
{
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn encode_legacy_or_v0_message(message: &crate::RawSolanaTransactionMessage, output: &mut std::vec::Vec<u8>) -> ksp_core_lib::Result<()> {
|
||||||
|
encode_header(message.header(), output);
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(message.account_keys().len(), output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
for account_key in message.account_keys() {
|
||||||
|
output.extend_from_slice(account_key);
|
||||||
|
}
|
||||||
|
output.extend_from_slice(message.recent_blockhash());
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(message.instructions().len(), output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
for instruction in message.instructions() {
|
||||||
|
output.push(instruction.program_id_index());
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(instruction.accounts().len(), output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.extend_from_slice(instruction.accounts());
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(instruction.data().len(), output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.extend_from_slice(instruction.data());
|
||||||
|
}
|
||||||
|
if message.version() == crate::RawSolanaMessageVersion::V0 {
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(message.address_table_lookups().len(), output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
for lookup in message.address_table_lookups() {
|
||||||
|
output.extend_from_slice(lookup.account_key());
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(lookup.writable_indexes().len(), output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.extend_from_slice(lookup.writable_indexes());
|
||||||
|
if let std::result::Result::Err(error) = encode_short_vec(lookup.readonly_indexes().len(), output) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.extend_from_slice(lookup.readonly_indexes());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn encode_v1_message(transaction: &crate::RawSolanaTransactionWire, output: &mut std::vec::Vec<u8>) -> ksp_core_lib::Result<()> {
|
||||||
|
let message = transaction.message();
|
||||||
|
let config = match message.config() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::material_error("wire")),
|
||||||
|
};
|
||||||
|
output.push(0x81);
|
||||||
|
encode_header(message.header(), output);
|
||||||
|
let mut mask = 0_u32;
|
||||||
|
if config.priority_fee().is_some() {
|
||||||
|
mask |= 0b11;
|
||||||
|
}
|
||||||
|
if config.compute_unit_limit().is_some() {
|
||||||
|
mask |= 1_u32 << 2;
|
||||||
|
}
|
||||||
|
if config.loaded_accounts_data_size_limit().is_some() {
|
||||||
|
mask |= 1_u32 << 3;
|
||||||
|
}
|
||||||
|
if config.heap_size().is_some() {
|
||||||
|
mask |= 1_u32 << 4;
|
||||||
|
}
|
||||||
|
output.extend_from_slice(&mask.to_le_bytes());
|
||||||
|
output.extend_from_slice(message.recent_blockhash());
|
||||||
|
output.push(message.instructions().len() as u8);
|
||||||
|
output.push(message.account_keys().len() as u8);
|
||||||
|
for account_key in message.account_keys() {
|
||||||
|
output.extend_from_slice(account_key);
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(value) = config.priority_fee() {
|
||||||
|
output.extend_from_slice(&value.to_le_bytes());
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(value) = config.compute_unit_limit() {
|
||||||
|
output.extend_from_slice(&value.to_le_bytes());
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(value) = config.loaded_accounts_data_size_limit() {
|
||||||
|
output.extend_from_slice(&value.to_le_bytes());
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(value) = config.heap_size() {
|
||||||
|
output.extend_from_slice(&value.to_le_bytes());
|
||||||
|
}
|
||||||
|
for instruction in message.instructions() {
|
||||||
|
output.push(instruction.program_id_index());
|
||||||
|
output.push(instruction.accounts().len() as u8);
|
||||||
|
output.extend_from_slice(&(instruction.data().len() as u16).to_le_bytes());
|
||||||
|
}
|
||||||
|
for instruction in message.instructions() {
|
||||||
|
output.extend_from_slice(instruction.accounts());
|
||||||
|
output.extend_from_slice(instruction.data());
|
||||||
|
}
|
||||||
|
for signature in transaction.signatures() {
|
||||||
|
output.extend_from_slice(signature);
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn encode_header(header: crate::RawSolanaMessageHeader, output: &mut std::vec::Vec<u8>) {
|
||||||
|
output.push(header.num_required_signatures());
|
||||||
|
output.push(header.num_readonly_signed_accounts());
|
||||||
|
output.push(header.num_readonly_unsigned_accounts());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn encode_short_vec(value: usize, output: &mut std::vec::Vec<u8>) -> ksp_core_lib::Result<()> {
|
||||||
|
if value > MAX_LEGACY_SHORT_VECTOR_VALUE {
|
||||||
|
return std::result::Result::Err(crate::material_error("wire"));
|
||||||
|
}
|
||||||
|
let mut remaining = value;
|
||||||
|
loop {
|
||||||
|
let mut byte = (remaining & 0x7f) as u8;
|
||||||
|
remaining >>= 7;
|
||||||
|
if remaining != 0 {
|
||||||
|
byte |= 0x80;
|
||||||
|
}
|
||||||
|
output.push(byte);
|
||||||
|
if remaining == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/wire.rs"]
|
||||||
|
mod tests;
|
||||||
55
crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
Normal file
55
crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
//! Dependency-boundary canaries for the common RAW Transaction foundation.
|
||||||
|
|
||||||
|
fn manifest() -> std::string::String {
|
||||||
|
return std::fs::read_to_string(std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("Cargo.toml")).unwrap_or_default();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn production_sources() -> std::string::String {
|
||||||
|
let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
||||||
|
let mut combined = std::string::String::new();
|
||||||
|
for name in ["acquisition.rs", "canonical.rs", "error.rs", "lib.rs", "signature.rs", "wire.rs"] {
|
||||||
|
let source = std::fs::read_to_string(root.join(name)).unwrap_or_default();
|
||||||
|
combined.push_str(source.as_str());
|
||||||
|
}
|
||||||
|
return combined;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_manifest_adds_only_base64_to_the_exact_common_dependency_surface() {
|
||||||
|
let manifest = manifest();
|
||||||
|
assert!(manifest.contains("base64.workspace = true"));
|
||||||
|
assert!(manifest.contains("ksp-core-lib = { path = \"../ksp-core-lib\" }"));
|
||||||
|
assert!(manifest.contains("ksp-store-api = { path = \"../ksp-store-api\" }"));
|
||||||
|
assert!(manifest.contains("serde_json = { workspace = true }"));
|
||||||
|
assert!(manifest.contains("sha2 = { workspace = true }"));
|
||||||
|
for forbidden in
|
||||||
|
["ksp-store-lib", "ksp-onchain-transport-lib", "ksp-config-lib", "ksp-job-api", "ksp-job-backfill-lib", "ksp-worker-api", "tokio", "futures"]
|
||||||
|
{
|
||||||
|
assert!(!manifest.contains(forbidden));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_production_sources_keep_transport_runtime_job_worker_and_backend_out() {
|
||||||
|
let sources = production_sources();
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_store_lib",
|
||||||
|
"ksp_onchain_transport_lib",
|
||||||
|
"ksp_config_lib",
|
||||||
|
"ksp_job_api",
|
||||||
|
"ksp_job_backfill_lib",
|
||||||
|
"ksp_worker_api",
|
||||||
|
"ksp_store_postgres_lib",
|
||||||
|
"tokio::",
|
||||||
|
"futures::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
] {
|
||||||
|
assert!(!sources.contains(forbidden));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
135
crates/ksp-raw-transaction-lib/tests/public_api.rs
Normal file
135
crates/ksp-raw-transaction-lib/tests/public_api.rs
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
//! Integration canaries for the public common RAW Transaction crate-root surface.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_common_raw_contract_is_consumable_from_crate_root() {
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_ID, "ksp.solana.raw_transaction");
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_VERSION, 1);
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES, 64);
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES, 88);
|
||||||
|
let network = match ksp_store_api::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let signature = match ksp_raw_transaction_lib::parse_raw_transaction_signature("1".repeat(64).as_str()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
5,
|
||||||
|
std::option::Option::None,
|
||||||
|
"AQID",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Null,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(ksp_raw_transaction_lib::RawTransactionVersion::Legacy),
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let transaction = ksp_raw_transaction_lib::canonicalize_raw_transaction(material);
|
||||||
|
assert!(transaction.is_ok());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_acquisition_assembly_accepts_store_api_producer_owned_metadata() {
|
||||||
|
let network = match ksp_store_api::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let signature = match ksp_raw_transaction_lib::parse_raw_transaction_signature("1".repeat(64).as_str()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
5,
|
||||||
|
std::option::Option::None,
|
||||||
|
"AQID",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let transaction = match ksp_raw_transaction_lib::canonicalize_raw_transaction(material) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let provider = match ksp_store_api::RawProvenanceCode::new("fixture") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_api::RawProvenanceCode::new("fixture.protocol") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_api::RawProvenanceCode::new("fixture.method") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_api::RawTimestamp::from_unix_millis(1) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_api::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_api::RawAcquisitionOrigin::Import, received_at);
|
||||||
|
let acquisition = ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(transaction, ksp_store_api::RawObservationKey::new([9_u8; 32]), provenance);
|
||||||
|
assert_eq!(acquisition.transaction().reference(), acquisition.observation().transaction());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_embedded_signature_and_block_material_contract_are_available_from_crate_root() {
|
||||||
|
let encoded = "AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
|
||||||
|
let signature = ksp_raw_transaction_lib::extract_raw_transaction_signature_from_binary_base64(encoded);
|
||||||
|
assert!(signature.is_ok());
|
||||||
|
let network = ksp_store_api::RawNetworkId::new("devnet");
|
||||||
|
assert!(network.is_ok());
|
||||||
|
if let std::result::Result::Ok(network) = network {
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64_with_embedded_signature(
|
||||||
|
network,
|
||||||
|
1,
|
||||||
|
std::option::Option::None,
|
||||||
|
encoded,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Value(0),
|
||||||
|
);
|
||||||
|
assert!(material.is_ok());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_source_neutral_wire_contract_is_available_from_crate_root() {
|
||||||
|
let message = ksp_raw_transaction_lib::RawSolanaTransactionMessage::new(
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageVersion::V1,
|
||||||
|
ksp_raw_transaction_lib::RawSolanaMessageHeader::new(1, 0, 0),
|
||||||
|
vec![[1_u8; 32]],
|
||||||
|
[2_u8; 32],
|
||||||
|
vec![ksp_raw_transaction_lib::RawSolanaCompiledInstruction::new(0, vec![0], vec![3])],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::Some(ksp_raw_transaction_lib::RawSolanaTransactionConfig::default()),
|
||||||
|
);
|
||||||
|
let wire = ksp_raw_transaction_lib::RawSolanaTransactionWire::new(vec![[4_u8; 64]], message);
|
||||||
|
let encoded = ksp_raw_transaction_lib::serialize_solana_transaction_wire_base64(&wire);
|
||||||
|
assert!(encoded.is_ok());
|
||||||
|
if let std::result::Result::Ok(encoded) = encoded {
|
||||||
|
assert!(encoded.starts_with("gQ"));
|
||||||
|
assert!(ksp_raw_transaction_lib::extract_raw_transaction_signature_from_binary_base64(encoded.as_str()).is_ok());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_signature_formatter_is_consumable_from_crate_root() {
|
||||||
|
let signature = ksp_store_api::RawTransactionSignature::new([0_u8; 64]);
|
||||||
|
let text = ksp_raw_transaction_lib::format_raw_transaction_signature(&signature);
|
||||||
|
assert_eq!(text, "1".repeat(64));
|
||||||
|
let parsed = ksp_raw_transaction_lib::parse_raw_transaction_signature(text.as_str());
|
||||||
|
assert!(parsed.is_ok());
|
||||||
|
if let std::result::Result::Ok(parsed) = parsed {
|
||||||
|
assert_eq!(parsed, signature);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
41
crates/ksp-raw-transaction-lib/tests/release_completeness.rs
Normal file
41
crates/ksp-raw-transaction-lib/tests/release_completeness.rs
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/tests/release_completeness.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
//! Release-completeness canaries for the common RAW Transaction foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_production_module_inventory_is_exact() {
|
||||||
|
let source_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
||||||
|
let entries = match std::fs::read_dir(source_root) {
|
||||||
|
std::result::Result::Ok(entries) => entries,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries.flatten() {
|
||||||
|
let path = entry.path();
|
||||||
|
if path.extension().and_then(std::ffi::OsStr::to_str) != std::option::Option::Some("rs") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let std::option::Option::Some(name) = path.file_name().and_then(std::ffi::OsStr::to_str) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
names.push(name.to_owned());
|
||||||
|
}
|
||||||
|
names.sort();
|
||||||
|
let expected = ["acquisition.rs", "canonical.rs", "error.rs", "lib.rs", "signature.rs", "wire.rs"];
|
||||||
|
assert_eq!(names.len(), expected.len());
|
||||||
|
for (actual, expected_name) in names.iter().zip(expected.iter()) {
|
||||||
|
assert_eq!(actual.as_str(), *expected_name);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_common_surface_contains_no_worker_job_or_transport_dependency_or_public_module() {
|
||||||
|
let lib = std::fs::read_to_string(std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src/lib.rs")).unwrap_or_default();
|
||||||
|
assert!(!lib.contains("pub mod "));
|
||||||
|
for forbidden in ["ksp_worker", "ksp_job", "ksp_onchain_transport", "ksp_config", "ksp_store_lib", "ksp_store_postgres"] {
|
||||||
|
assert!(!lib.contains(forbidden));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
28
crates/ksp-raw-transaction-lib/tests/security_hardening.rs
Normal file
28
crates/ksp-raw-transaction-lib/tests/security_hardening.rs
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/tests/security_hardening.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! Adversarial canaries for the common RAW Transaction foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_signature_errors_never_echo_hostile_input() {
|
||||||
|
let hostile = format!("{}0", "A".repeat(63));
|
||||||
|
let result = ksp_raw_transaction_lib::parse_raw_transaction_signature(hostile.as_str());
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), ksp_raw_transaction_lib::ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID);
|
||||||
|
assert!(!std::format!("{error:?}").contains(hostile.as_str()));
|
||||||
|
assert!(!error.to_string().contains(hostile.as_str()));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_public_error_codes_are_stable_and_domain_scoped() {
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::ERROR_CODE_RAW_TRANSACTION_CANONICALIZATION_INVALID.domain(), "raw_transaction");
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::ERROR_CODE_RAW_TRANSACTION_CANONICALIZATION_INVALID.code(), "canonicalization_invalid");
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::ERROR_CODE_RAW_TRANSACTION_MATERIAL_INVALID.domain(), "raw_transaction");
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::ERROR_CODE_RAW_TRANSACTION_MATERIAL_INVALID.code(), "material_invalid");
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID.domain(), "raw_transaction");
|
||||||
|
assert_eq!(ksp_raw_transaction_lib::ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID.code(), "signature_invalid");
|
||||||
|
return;
|
||||||
|
}
|
||||||
73
crates/ksp-raw-transaction-lib/unit_tests/acquisition.rs
Normal file
73
crates/ksp-raw-transaction-lib/unit_tests/acquisition.rs
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/unit_tests/acquisition.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
fn canonical_transaction() -> std::option::Option<ksp_store_api::RawTransaction> {
|
||||||
|
let network = match ksp_store_api::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = match crate::parse_raw_transaction_signature("1".repeat(64).as_str()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let material = crate::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::None,
|
||||||
|
"AQID",
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
return match crate::canonicalize_raw_transaction(material) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn provenance() -> std::option::Option<ksp_store_api::RawAcquisitionProvenance> {
|
||||||
|
let provider = match ksp_store_api::RawProvenanceCode::new("fixture") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_api::RawProvenanceCode::new("fixture.protocol") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_api::RawProvenanceCode::new("fixture.method") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_api::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_store_api::RawAcquisitionProvenance::new(
|
||||||
|
provider,
|
||||||
|
protocol,
|
||||||
|
method,
|
||||||
|
ksp_store_api::RawAcquisitionOrigin::Import,
|
||||||
|
received_at,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_acquisition_preserves_producer_key_and_exact_transaction_reference() {
|
||||||
|
let transaction = match canonical_transaction() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let key = ksp_store_api::RawObservationKey::new([7_u8; 32]);
|
||||||
|
let acquisition = crate::assemble_raw_transaction_acquisition(transaction, key, provenance);
|
||||||
|
assert_eq!(acquisition.observation().observation_key(), key);
|
||||||
|
assert_eq!(acquisition.observation().transaction(), acquisition.transaction().reference());
|
||||||
|
assert_eq!(std::format!("{acquisition:?}"), "RawTransactionAcquisition(..)");
|
||||||
|
let (transaction, observation) = acquisition.into_parts();
|
||||||
|
assert_eq!(observation.transaction(), transaction.reference());
|
||||||
|
return;
|
||||||
|
}
|
||||||
247
crates/ksp-raw-transaction-lib/unit_tests/canonical.rs
Normal file
247
crates/ksp-raw-transaction-lib/unit_tests/canonical.rs
Normal file
@@ -0,0 +1,247 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/unit_tests/canonical.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn network() -> std::option::Option<ksp_store_api::RawNetworkId> {
|
||||||
|
return match ksp_store_api::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn signature() -> std::option::Option<ksp_store_api::RawTransactionSignature> {
|
||||||
|
let result = crate::parse_raw_transaction_signature("1".repeat(64).as_str());
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn material(
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
transaction_data: std::string::String,
|
||||||
|
meta: crate::RawTransactionWireField<serde_json::Value>,
|
||||||
|
version: crate::RawTransactionWireField<crate::RawTransactionVersion>,
|
||||||
|
transaction_index: crate::RawTransactionWireField<u32>,
|
||||||
|
) -> std::option::Option<crate::RawTransactionMaterial> {
|
||||||
|
let network = match network() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = match signature() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(crate::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
123,
|
||||||
|
block_time,
|
||||||
|
transaction_data,
|
||||||
|
meta,
|
||||||
|
version,
|
||||||
|
transaction_index,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_canonical_raw_v1_golden_bytes_and_hash_are_exact() {
|
||||||
|
let material = match material(
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
"AQID".to_owned(),
|
||||||
|
crate::RawTransactionWireField::Value(serde_json::json!({"z": 1, "a": {"y": true, "x": null}})),
|
||||||
|
crate::RawTransactionWireField::Value(crate::RawTransactionVersion::Legacy),
|
||||||
|
crate::RawTransactionWireField::Value(7_u32),
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let transaction = crate::canonicalize_raw_transaction(material);
|
||||||
|
assert!(transaction.is_ok());
|
||||||
|
let transaction = match transaction {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let expected = b"{\"transaction\":[\"AQID\",\"base64\"],\"meta\":{\"a\":{\"x\":null,\"y\":true},\"z\":1},\"version\":\"legacy\",\"transactionIndex\":7}";
|
||||||
|
assert_eq!(transaction.payload().bytes(), expected);
|
||||||
|
assert_eq!(transaction.payload().byte_len(), 112);
|
||||||
|
assert_eq!(transaction.payload().format_id().as_str(), crate::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
assert_eq!(transaction.payload().format_version(), crate::RAW_TRANSACTION_FORMAT_VERSION);
|
||||||
|
assert_eq!(
|
||||||
|
transaction.payload().content_hash().as_bytes(),
|
||||||
|
&[
|
||||||
|
34, 7, 146, 210, 177, 93, 38, 47, 218, 36, 44, 178, 32, 119, 78, 233, 221, 239, 254, 191, 4, 220, 250, 218, 188, 248, 239, 118, 169, 177, 167, 195
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert_eq!(transaction.slot(), 123);
|
||||||
|
assert_eq!(transaction.block_time().map(|value| return value.unix_millis()), std::option::Option::Some(1_700_000_000_000));
|
||||||
|
assert_eq!(transaction.reference().network().as_str(), "devnet");
|
||||||
|
assert_eq!(transaction.reference().signature().as_bytes(), &[0_u8; 64]);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_wire_omitted_null_and_value_states_remain_distinct() {
|
||||||
|
let omitted = match material(
|
||||||
|
std::option::Option::None,
|
||||||
|
"AQID".to_owned(),
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let omitted = super::canonical_payload_bytes(&omitted);
|
||||||
|
assert!(omitted.is_ok());
|
||||||
|
if let std::result::Result::Ok(omitted) = omitted {
|
||||||
|
assert_eq!(omitted, b"{\"transaction\":[\"AQID\",\"base64\"]}");
|
||||||
|
}
|
||||||
|
let nulls = match material(
|
||||||
|
std::option::Option::None,
|
||||||
|
"AQID".to_owned(),
|
||||||
|
crate::RawTransactionWireField::Null,
|
||||||
|
crate::RawTransactionWireField::Null,
|
||||||
|
crate::RawTransactionWireField::Null,
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let nulls = super::canonical_payload_bytes(&nulls);
|
||||||
|
assert!(nulls.is_ok());
|
||||||
|
if let std::result::Result::Ok(nulls) = nulls {
|
||||||
|
assert_eq!(nulls, b"{\"transaction\":[\"AQID\",\"base64\"],\"meta\":null,\"version\":null,\"transactionIndex\":null}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_canonical_json_sorts_objects_preserves_arrays_and_escapes_strings() {
|
||||||
|
let meta = serde_json::json!({
|
||||||
|
"z": [3, 2, 1],
|
||||||
|
"quote": "a\"b\\c\n",
|
||||||
|
"a": {"n": -12.5, "b": false, "a": true}
|
||||||
|
});
|
||||||
|
let material = match material(
|
||||||
|
std::option::Option::None,
|
||||||
|
"A\"B".to_owned(),
|
||||||
|
crate::RawTransactionWireField::Value(meta),
|
||||||
|
crate::RawTransactionWireField::Value(crate::RawTransactionVersion::Number(0)),
|
||||||
|
crate::RawTransactionWireField::Value(u32::MAX),
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let bytes = super::canonical_payload_bytes(&material);
|
||||||
|
assert!(bytes.is_ok());
|
||||||
|
if let std::result::Result::Ok(bytes) = bytes {
|
||||||
|
let expected = b"{\"transaction\":[\"A\\\"B\",\"base64\"],\"meta\":{\"a\":{\"a\":true,\"b\":false,\"n\":-12.5},\"quote\":\"a\\\"b\\\\c\\n\",\"z\":[3,2,1]},\"version\":0,\"transactionIndex\":4294967295}";
|
||||||
|
assert_eq!(bytes, expected);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_block_time_rejects_negative_overflow_and_store_maximum_overrun() {
|
||||||
|
for block_time in [-1_i64, i64::MAX, 253_402_300_800_i64] {
|
||||||
|
let material = match material(
|
||||||
|
std::option::Option::Some(block_time),
|
||||||
|
"AQID".to_owned(),
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::canonicalize_raw_transaction(material);
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_MATERIAL_INVALID);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let maximum = match material(
|
||||||
|
std::option::Option::Some(253_402_300_799_i64),
|
||||||
|
"AQID".to_owned(),
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let maximum = crate::canonicalize_raw_transaction(maximum);
|
||||||
|
assert!(maximum.is_ok());
|
||||||
|
if let std::result::Result::Ok(maximum) = maximum {
|
||||||
|
assert_eq!(maximum.block_time().map(|value| return value.unix_millis()), std::option::Option::Some(253_402_300_799_000));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_payload_bound_is_enforced_by_common_before_store_construction() {
|
||||||
|
let material = match material(
|
||||||
|
std::option::Option::None,
|
||||||
|
"A".repeat(ksp_store_api::MAX_RAW_PAYLOAD_BYTES),
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
crate::RawTransactionWireField::Omitted,
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::canonicalize_raw_transaction(material);
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_CANONICALIZATION_INVALID);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_material_and_wire_debug_do_not_render_transaction_or_meta_material() {
|
||||||
|
let secret_transaction = "SECRET_TRANSACTION_BODY";
|
||||||
|
let secret_meta = "SECRET_META_VALUE";
|
||||||
|
let material = match material(
|
||||||
|
std::option::Option::None,
|
||||||
|
secret_transaction.to_owned(),
|
||||||
|
crate::RawTransactionWireField::Value(serde_json::json!({"secret": secret_meta})),
|
||||||
|
crate::RawTransactionWireField::Value(crate::RawTransactionVersion::Legacy),
|
||||||
|
crate::RawTransactionWireField::Value(1_u32),
|
||||||
|
) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let debug = std::format!("{material:?}");
|
||||||
|
assert!(!debug.contains(secret_transaction));
|
||||||
|
assert!(!debug.contains(secret_meta));
|
||||||
|
assert_eq!(std::format!("{:?}", crate::RawTransactionWireField::Value(secret_meta)), "Value(..)");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_embedded_signature_material_preserves_block_fields_and_canonicalizes() {
|
||||||
|
let network = match ksp_store_api::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(network) => network,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let material = crate::RawTransactionMaterial::binary_base64_with_embedded_signature(
|
||||||
|
network,
|
||||||
|
430_000_123,
|
||||||
|
std::option::Option::Some(1_787_072_400),
|
||||||
|
"AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
|
||||||
|
crate::RawTransactionWireField::Value(serde_json::json!({"err":null,"fee":5000})),
|
||||||
|
crate::RawTransactionWireField::Value(crate::RawTransactionVersion::Legacy),
|
||||||
|
crate::RawTransactionWireField::Value(0_u32),
|
||||||
|
);
|
||||||
|
assert!(material.is_ok());
|
||||||
|
let transaction = match material.and_then(crate::canonicalize_raw_transaction) {
|
||||||
|
std::result::Result::Ok(transaction) => transaction,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(transaction.reference().signature().as_bytes(), &[0_u8; 64]);
|
||||||
|
assert_eq!(transaction.slot(), 430_000_123);
|
||||||
|
assert_eq!(transaction.block_time().map(|value| return value.unix_millis()), std::option::Option::Some(1_787_072_400_000));
|
||||||
|
assert!(transaction.payload().bytes().starts_with(b"{\"transaction\":[\"AQAAAA"));
|
||||||
|
assert!(transaction.payload().bytes().ends_with(b"\"version\":\"legacy\",\"transactionIndex\":0}"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
97
crates/ksp-raw-transaction-lib/unit_tests/signature.rs
Normal file
97
crates/ksp-raw-transaction-lib/unit_tests/signature.rs
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/unit_tests/signature.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
use base64::Engine; // rust-rules: trait-import
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_signature_parser_accepts_exact_sixty_four_zero_bytes() {
|
||||||
|
let text = "1".repeat(crate::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES);
|
||||||
|
let signature = crate::parse_raw_transaction_signature(text.as_str());
|
||||||
|
assert!(signature.is_ok());
|
||||||
|
if let std::result::Result::Ok(signature) = signature {
|
||||||
|
assert_eq!(signature.as_bytes(), &[0_u8; 64]);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_signature_parser_rejects_text_bounds_invalid_base58_and_noncanonical_lengths() {
|
||||||
|
for text in [
|
||||||
|
"1".repeat(crate::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES - 1),
|
||||||
|
"1".repeat(crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES + 1),
|
||||||
|
format!("{}0", "1".repeat(crate::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES - 1)),
|
||||||
|
"1".repeat(65),
|
||||||
|
"z".repeat(crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES),
|
||||||
|
] {
|
||||||
|
let result = crate::parse_raw_transaction_signature(text.as_str());
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID);
|
||||||
|
let debug = std::format!("{error:?}");
|
||||||
|
assert!(!debug.contains(text.as_str()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_embedded_signature_extracts_first_signature_from_canonical_transaction_base64() {
|
||||||
|
let encoded = "AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
|
||||||
|
let signature = crate::extract_raw_transaction_signature_from_binary_base64(encoded);
|
||||||
|
assert!(signature.is_ok());
|
||||||
|
if let std::result::Result::Ok(signature) = signature {
|
||||||
|
assert_eq!(signature.as_bytes(), &[0_u8; 64]);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_embedded_signature_rejects_invalid_base64_noncanonical_short_vec_and_truncation() {
|
||||||
|
for encoded in [
|
||||||
|
"not-base64",
|
||||||
|
"gQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
|
||||||
|
"AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
|
||||||
|
"AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
|
||||||
|
] {
|
||||||
|
let result = crate::extract_raw_transaction_signature_from_binary_base64(encoded);
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID);
|
||||||
|
assert!(!error.to_string().contains(encoded));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_embedded_signature_extracts_terminal_v1_signature_and_rejects_trailing_bytes() {
|
||||||
|
let encoded = "gQEAAR8AAABRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUQECQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQgcAAAAAAAAACAAAAAkAAAAAgAAAAQEDAAABAgOqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq";
|
||||||
|
let signature = crate::extract_raw_transaction_signature_from_binary_base64(encoded);
|
||||||
|
assert!(signature.is_ok());
|
||||||
|
if let std::result::Result::Ok(signature) = signature {
|
||||||
|
assert_eq!(signature.as_bytes(), &[0xaa_u8; 64]);
|
||||||
|
}
|
||||||
|
let decoded = base64::engine::general_purpose::STANDARD.decode(encoded);
|
||||||
|
assert!(decoded.is_ok());
|
||||||
|
if let std::result::Result::Ok(mut decoded) = decoded {
|
||||||
|
decoded.push(0);
|
||||||
|
let trailing = base64::engine::general_purpose::STANDARD.encode(decoded);
|
||||||
|
assert!(crate::extract_raw_transaction_signature_from_binary_base64(trailing.as_str()).is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_base58_formatter_round_trips_canonical_signatures() {
|
||||||
|
for bytes in [[0_u8; 64], [1_u8; 64], [0xAB_u8; 64]] {
|
||||||
|
let signature = ksp_store_api::RawTransactionSignature::new(bytes);
|
||||||
|
let text = crate::format_raw_transaction_signature(&signature);
|
||||||
|
assert!((crate::MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES..=crate::MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES).contains(&text.len()));
|
||||||
|
let parsed = crate::parse_raw_transaction_signature(text.as_str());
|
||||||
|
assert!(parsed.is_ok());
|
||||||
|
if let std::result::Result::Ok(parsed) = parsed {
|
||||||
|
assert_eq!(parsed, signature);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(crate::format_raw_transaction_signature(&ksp_store_api::RawTransactionSignature::new([0_u8; 64])), "1".repeat(64));
|
||||||
|
return;
|
||||||
|
}
|
||||||
122
crates/ksp-raw-transaction-lib/unit_tests/wire.rs
Normal file
122
crates/ksp-raw-transaction-lib/unit_tests/wire.rs
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
// file: crates/ksp-raw-transaction-lib/unit_tests/wire.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
const LEGACY_GOLDEN_BASE64: &str = "AREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREBAAECISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIjExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExAQEBAAKquw==";
|
||||||
|
const V0_GOLDEN_BASE64: &str = "ARISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhISEhKAAQAAASMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIBAAEBAcwBJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQBAAA=";
|
||||||
|
const V1_EMPTY_CONFIG_GOLDEN_BASE64: &str = "gQEAAAAAAABSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUgABQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0Orq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6urq6ur";
|
||||||
|
const V1_GOLDEN_BASE64: &str = "gQEAAR8AAABRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUQECQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQgcAAAAAAAAACAAAAAkAAAAAgAAAAQEDAAABAgOqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq";
|
||||||
|
|
||||||
|
fn legacy_wire() -> crate::RawSolanaTransactionWire {
|
||||||
|
let message = crate::RawSolanaTransactionMessage::new(
|
||||||
|
crate::RawSolanaMessageVersion::Legacy,
|
||||||
|
crate::RawSolanaMessageHeader::new(1, 0, 1),
|
||||||
|
vec![[0x21_u8; 32], [0x22_u8; 32]],
|
||||||
|
[0x31_u8; 32],
|
||||||
|
vec![crate::RawSolanaCompiledInstruction::new(1, vec![0], vec![0xaa, 0xbb])],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
return crate::RawSolanaTransactionWire::new(vec![[0x11_u8; 64]], message);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn v0_wire() -> crate::RawSolanaTransactionWire {
|
||||||
|
let message = crate::RawSolanaTransactionMessage::new(
|
||||||
|
crate::RawSolanaMessageVersion::V0,
|
||||||
|
crate::RawSolanaMessageHeader::new(1, 0, 0),
|
||||||
|
vec![[0x23_u8; 32]],
|
||||||
|
[0x32_u8; 32],
|
||||||
|
vec![crate::RawSolanaCompiledInstruction::new(0, vec![1], vec![0xcc])],
|
||||||
|
vec![crate::RawSolanaAddressTableLookup::new([0x24_u8; 32], vec![0], vec![])],
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
return crate::RawSolanaTransactionWire::new(vec![[0x12_u8; 64]], message);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn v1_wire() -> crate::RawSolanaTransactionWire {
|
||||||
|
let message = crate::RawSolanaTransactionMessage::new(
|
||||||
|
crate::RawSolanaMessageVersion::V1,
|
||||||
|
crate::RawSolanaMessageHeader::new(1, 0, 1),
|
||||||
|
vec![[0x41_u8; 32], [0x42_u8; 32]],
|
||||||
|
[0x51_u8; 32],
|
||||||
|
vec![crate::RawSolanaCompiledInstruction::new(1, vec![0], vec![1, 2, 3])],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::Some(crate::RawSolanaTransactionConfig::new(
|
||||||
|
std::option::Option::Some(7),
|
||||||
|
std::option::Option::Some(8),
|
||||||
|
std::option::Option::Some(9),
|
||||||
|
std::option::Option::Some(32_768),
|
||||||
|
)),
|
||||||
|
);
|
||||||
|
return crate::RawSolanaTransactionWire::new(vec![[0xaa_u8; 64]], message);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn v1_empty_config_wire() -> crate::RawSolanaTransactionWire {
|
||||||
|
let message = crate::RawSolanaTransactionMessage::new(
|
||||||
|
crate::RawSolanaMessageVersion::V1,
|
||||||
|
crate::RawSolanaMessageHeader::new(1, 0, 0),
|
||||||
|
vec![[0x43_u8; 32]],
|
||||||
|
[0x52_u8; 32],
|
||||||
|
vec![],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::Some(crate::RawSolanaTransactionConfig::default()),
|
||||||
|
);
|
||||||
|
return crate::RawSolanaTransactionWire::new(vec![[0xab_u8; 64]], message);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_legacy_v0_and_v1_wire_goldens_are_exact() {
|
||||||
|
let legacy = crate::serialize_solana_transaction_wire_base64(&legacy_wire());
|
||||||
|
let v0 = crate::serialize_solana_transaction_wire_base64(&v0_wire());
|
||||||
|
let v1 = crate::serialize_solana_transaction_wire_base64(&v1_wire());
|
||||||
|
let v1_empty = crate::serialize_solana_transaction_wire_base64(&v1_empty_config_wire());
|
||||||
|
assert!(matches!(legacy.as_deref(), std::result::Result::Ok(value) if value == LEGACY_GOLDEN_BASE64));
|
||||||
|
assert!(matches!(v0.as_deref(), std::result::Result::Ok(value) if value == V0_GOLDEN_BASE64));
|
||||||
|
assert!(matches!(v1.as_deref(), std::result::Result::Ok(value) if value == V1_GOLDEN_BASE64));
|
||||||
|
assert!(matches!(v1_empty.as_deref(), std::result::Result::Ok(value) if value == V1_EMPTY_CONFIG_GOLDEN_BASE64));
|
||||||
|
let v1_bytes = crate::serialize_solana_transaction_wire(&v1_wire());
|
||||||
|
assert!(v1_bytes.is_ok());
|
||||||
|
if let std::result::Result::Ok(v1_bytes) = v1_bytes {
|
||||||
|
assert_eq!(v1_bytes.len(), 198);
|
||||||
|
assert_eq!(v1_bytes.first(), std::option::Option::Some(&0x81));
|
||||||
|
assert_eq!(v1_bytes.get(v1_bytes.len() - 64..), std::option::Option::Some([0xaa_u8; 64].as_slice()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_v1_structural_guards_reject_alt_duplicates_and_missing_config() {
|
||||||
|
let mut missing_config = v1_wire();
|
||||||
|
missing_config.message.config = std::option::Option::None;
|
||||||
|
assert!(crate::serialize_solana_transaction_wire(&missing_config).is_err());
|
||||||
|
let mut with_lookup = v1_wire();
|
||||||
|
with_lookup.message.address_table_lookups.push(crate::RawSolanaAddressTableLookup::new([9_u8; 32], vec![], vec![]));
|
||||||
|
assert!(crate::serialize_solana_transaction_wire(&with_lookup).is_err());
|
||||||
|
let mut duplicate = v1_wire();
|
||||||
|
duplicate.message.account_keys[1] = duplicate.message.account_keys[0];
|
||||||
|
assert!(crate::serialize_solana_transaction_wire(&duplicate).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_v1_program_index_zero_is_valid_and_bounds_remain_enforced() {
|
||||||
|
let message = crate::RawSolanaTransactionMessage::new(
|
||||||
|
crate::RawSolanaMessageVersion::V1,
|
||||||
|
crate::RawSolanaMessageHeader::new(1, 0, 0),
|
||||||
|
vec![[1_u8; 32]],
|
||||||
|
[2_u8; 32],
|
||||||
|
vec![crate::RawSolanaCompiledInstruction::new(0, vec![0], vec![])],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::Some(crate::RawSolanaTransactionConfig::default()),
|
||||||
|
);
|
||||||
|
let wire = crate::RawSolanaTransactionWire::new(vec![[3_u8; 64]], message);
|
||||||
|
assert!(crate::serialize_solana_transaction_wire(&wire).is_ok());
|
||||||
|
let too_many_addresses = crate::RawSolanaTransactionMessage::new(
|
||||||
|
crate::RawSolanaMessageVersion::V1,
|
||||||
|
crate::RawSolanaMessageHeader::new(1, 0, 0),
|
||||||
|
(0_u8..65_u8).map(|value| return [value; 32]).collect(),
|
||||||
|
[2_u8; 32],
|
||||||
|
vec![],
|
||||||
|
vec![],
|
||||||
|
std::option::Option::Some(crate::RawSolanaTransactionConfig::default()),
|
||||||
|
);
|
||||||
|
let wire = crate::RawSolanaTransactionWire::new(vec![[3_u8; 64]], too_many_addresses);
|
||||||
|
assert!(crate::serialize_solana_transaction_wire(&wire).is_err());
|
||||||
|
}
|
||||||
21
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
Normal file
21
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
# file: crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
# version: 2
|
||||||
|
|
||||||
|
[package]
|
||||||
|
name = "ksp-worker-raw-transaction-ingest-lib"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
ksp-core-lib = { path = "../ksp-core-lib" }
|
||||||
|
ksp-logging-lib = { path = "../ksp-logging-lib" }
|
||||||
|
ksp-onchain-transport-lib = { path = "../ksp-onchain-transport-lib" }
|
||||||
|
ksp-raw-transaction-lib = { path = "../ksp-raw-transaction-lib" }
|
||||||
|
ksp-store-lib = { path = "../ksp-store-lib", default-features = false }
|
||||||
|
ksp-worker-api = { path = "../ksp-worker-api" }
|
||||||
|
sha2 = { workspace = true }
|
||||||
|
tokio = { workspace = true, features = ["macros", "rt", "sync", "time"] }
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
319
crates/ksp-worker-raw-transaction-ingest-lib/README.md
Normal file
319
crates/ksp-worker-raw-transaction-ingest-lib/README.md
Normal file
@@ -0,0 +1,319 @@
|
|||||||
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/README.md -->
|
||||||
|
<!-- version: 11 -->
|
||||||
|
|
||||||
|
# ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
|
`ksp-worker-raw-transaction-ingest-lib` est le Worker concret KSP chargé de l'alimentation continue de la couche RAW Transaction.
|
||||||
|
|
||||||
|
La crate possède deux niveaux publics complémentaires :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker::start
|
||||||
|
-> fondation source-neutral, sans source productive
|
||||||
|
|
||||||
|
RawTransactionIngestWorker::start_with_runtime_resources
|
||||||
|
-> même runtime + 1..32 sources productives supervisées simultanément
|
||||||
|
Yellowstone, WS standard logsSubscribe, WS standard blockSubscribe, Helius transactionSubscribe et/ou HTTP block polling
|
||||||
|
+ hydration HTTP getTransaction lorsque la source produit une référence
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker reste indépendant de Config et de tout backend Store physique. Le caller compose les ressources Transport et Store, puis les remet à la crate par ses façades publiques.
|
||||||
|
|
||||||
|
## Pipeline productif actuel
|
||||||
|
|
||||||
|
Les verticales live productives sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone standard subscribe
|
||||||
|
-> Transaction / TransactionStatus / Block
|
||||||
|
-> signal source-neutral (network, signature, slot, commitment, provenance sûre)
|
||||||
|
|
||||||
|
Solana standard WS logsSubscribe
|
||||||
|
-> context.slot + signature
|
||||||
|
-> signal source-neutral (network, signature, slot, commitment, provenance sûre)
|
||||||
|
-> coalescence bornée par (network, signature, commitment)
|
||||||
|
-> HTTP getTransaction observed, Base64, maxSupportedTransactionVersion=1
|
||||||
|
|
||||||
|
Solana standard WS blockSubscribe
|
||||||
|
-> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false
|
||||||
|
-> qualification explicite Legacy / V0 / V1
|
||||||
|
-> matériau Common RAW direct par transaction du bloc
|
||||||
|
|
||||||
|
Helius transactionSubscribe
|
||||||
|
-> Full + Base64 + maxSupportedTransactionVersion=1 + showRewards=false
|
||||||
|
-> signature + slot + transactionIndex uniquement dans le signal Worker
|
||||||
|
-> coalescence bornée par (network, signature, commitment)
|
||||||
|
-> HTTP getTransaction observed avant Common RAW
|
||||||
|
|
||||||
|
Solana HTTP live block polling
|
||||||
|
-> getSlot borne le run courant
|
||||||
|
-> getBlocksWithLimit découvre les blocs live
|
||||||
|
-> getBlock observed matérialise Full/Base64 Legacy/V0/V1
|
||||||
|
-> matériau Common RAW direct par transaction du bloc
|
||||||
|
|
||||||
|
les chemins productifs
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> admission centrale bornée
|
||||||
|
-> ksp-store-lib
|
||||||
|
-> RawTransaction + RawTransactionObservation
|
||||||
|
```
|
||||||
|
|
||||||
|
`BlockMeta` et `Slot` ne produisent pas de RAW directement. Ils servent uniquement à la projection de continuité du run.
|
||||||
|
|
||||||
|
La qualification reste conservative : même lorsqu'une update Yellowstone contient une transaction complète côté protobuf, le Worker hydrate actuellement les signaux transactionnels par HTTP `getTransaction` avant de construire le RAW canonique. Il n'existe donc pas de second canonicaliseur Yellowstone.
|
||||||
|
|
||||||
|
## Contrat de source Yellowstone + HTTP
|
||||||
|
|
||||||
|
`RawTransactionIngestYellowstoneSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
YellowstoneGrpcChannel
|
||||||
|
YellowstoneSubscribeRequest
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O et refuse notamment :
|
||||||
|
|
||||||
|
- une requête Yellowstone invalide ;
|
||||||
|
- l'absence de famille porteuse d'ingestion ;
|
||||||
|
- un commitment `Processed` ou implicite ;
|
||||||
|
- un réseau Yellowstone non représentable ;
|
||||||
|
- une provenance provider/endpoint non représentable ;
|
||||||
|
- l'absence d'une route HTTP compatible pour `getTransaction` sur le même réseau.
|
||||||
|
|
||||||
|
Le runtime-resource aggregate public accepte une collection validée de 1 à 32 sources logiques et les démarre simultanément sous un supervisor privé. La collection est validée entièrement avant spawn ; aucun sous-ensemble silencieux, source primaire implicite ou standby n'est choisi. La collection interne, les `source_key`, les URLs, les filtres et les clients inférieurs ne sont pas exposés.
|
||||||
|
|
||||||
|
Le supervisor possède toutes les tâches source. Une source qui échoue est terminale pour le Worker et déclenche l'arrêt coopératif puis le join des autres sources, car cette release ne suppose aucune équivalence de coverage. Une fermeture propre d'une source alors que le Worker n'est pas en arrêt est également traitée comme une perte de source configurée et devient terminale.
|
||||||
|
|
||||||
|
Un inventaire privé `source_key -> latest processing/source state`, borné à 32 entrées, agrège la projection run-local. La frontier agrégée reste conservative : elle n'expose un `processing_frontier_slot` que lorsque toutes les sources en possèdent un, choisit le minimum des frontiers connus et le plus ancien pending. Les sources reference-bearing partagent en plus un registre global d'hydration borné : une même clé `(network, signature, commitment)` ne déclenche qu'un leader HTTP, puis chaque signal source conserve sa propre provenance lors de la finalisation.
|
||||||
|
|
||||||
|
## Contrat de source Standard Logs + HTTP
|
||||||
|
|
||||||
|
`RawTransactionIngestStandardLogsSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaLogsSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O. Elle refuse un endpoint WS invalide ou non standard, `Processed`, un réseau/provenance non représentable et l'absence de route HTTP `getTransaction` compatible sur le même réseau. Le filtre `All`, `AllWithVotes` ou `Mentions(pubkey)` participe uniquement à une empreinte privée ; le pubkey d'un filtre `Mentions` n'est pas recopié dans `Debug`, snapshot ou provenance textuelle.
|
||||||
|
|
||||||
|
Au runtime, le Worker ouvre `SolanaStandardWsSession::connect`, puis `logs_subscribe`. Les lignes de logs et `err` restent dans Transport et ne sont jamais stockées dans le signal Worker. Seuls `context.slot` et `signature` sont projetés vers l'hydration commune. Reconnect, resubscribe et backpressure de la subscription restent possédés par Transport.
|
||||||
|
|
||||||
|
## Contrat de source Standard Block direct RAW
|
||||||
|
|
||||||
|
`RawTransactionIngestStandardBlockSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind solana_standard
|
||||||
|
SolanaBlockSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
```
|
||||||
|
|
||||||
|
Le runtime ouvre la `SolanaStandardWsSession` existante et demande exactement `Base64`, `Full`, `maxSupportedTransactionVersion = 1` et `showRewards = false`. Aucun `HttpTransportPool` n'est attaché à cette source : les transactions dont la version est explicitement `Legacy`, `0` ou `1` sont transformées directement en `RawTransactionMaterial` à partir du wire Base64 du bloc, avec signature embarquée, slot, block time, meta, version et index de transaction.
|
||||||
|
|
||||||
|
La qualification est fermée : une version omise/nulle ou supérieure à `1`, une transaction non Base64, `block: null`, une erreur distante de notification, un slot de contexte incohérent ou un champ transactions absent/nul termine la source par une erreur sûre. Aucun de ces cas n'est converti en bloc vide, en succès silencieux ou en progression artificielle de frontier.
|
||||||
|
|
||||||
|
Pour un bloc multi-transaction, le slot n'est projeté settled qu'après l'admission réussie de toutes ses transactions. Cette voie RAW-direct n'incrémente pas `hydration_pending`; un stop ou une erreur au milieu du bloc ne produit aucune progression artificielle. Reconnect, resubscribe et backpressure WebSocket restent possédés par Transport.
|
||||||
|
|
||||||
|
## Contrat de source Helius Transaction + HTTP
|
||||||
|
|
||||||
|
`RawTransactionIngestHeliusTransactionSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
WsEndpointSettings kind helius_laserstream
|
||||||
|
HeliusTransactionSubscribeFilter
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName d'hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O. Elle réutilise le contrat Transport existant et impose `Full`, `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`. Le Worker ne lit pas Config, ne lit pas `KSP_SECRET_HELIUS_API_KEY` et ne code aucun tier provider ; l'URL résolue et le credential restent dans l'endpoint Transport fourni par le caller.
|
||||||
|
|
||||||
|
Au runtime, `HeliusLaserStreamWsSession::connect` puis `transaction_subscribe` sont utilisés. Seule une notification `Full` conforme au mode demandé est admise ; une forme `Signature`, `Unknown` ou future devient une faute source sûre. Le payload Helius `transaction` n'est jamais copié dans l'état Worker : la projection conserve uniquement signature, slot et transaction index, puis réutilise le coordinateur d'hydration commun `getTransaction observed`.
|
||||||
|
|
||||||
|
La clé logique Helius inclut réseau, identités provider/endpoint sûres, commitment et empreinte privée du filtre. Les listes de pubkeys du filtre sont normalisées avant hash afin que leur ordre ne crée pas artificiellement deux sources logiques ; le rôle HTTP d'hydration reste exclu de l'identité live.
|
||||||
|
|
||||||
|
## Contrat de source HTTP Block Polling
|
||||||
|
|
||||||
|
`RawTransactionIngestHttpBlockPollingSource::new` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HttpTransportPool
|
||||||
|
HttpRoleName de polling
|
||||||
|
SolanaCommitment Confirmed ou Finalized
|
||||||
|
```
|
||||||
|
|
||||||
|
La construction est sans I/O et vérifie que le rôle HTTP possède, sur un seul réseau, les capacités `getSlot`, `getBlocksWithLimit` et `getBlock`. La cadence est bornée entre 100 ms et 30 s, avec 1 s par défaut ; la découverte est bornée entre 1 et 1024 blocs par cycle, avec 128 par défaut. Ces réglages de cadence ne font pas partie de l'identité logique de la source.
|
||||||
|
|
||||||
|
Au démarrage, le premier `getSlot` fixe la borne inférieure du run. Le Worker ne demande aucun slot antérieur. Chaque cycle relit le tip, découvre les blocs disponibles avec `getBlocksWithLimit`, puis matérialise chaque slot listé par `getBlock observed` en `Full + Base64 + maxSupportedTransactionVersion = 1 + showRewards = false`. Seules les transactions Legacy/V0/V1 explicitement qualifiées entrent directement dans Common RAW.
|
||||||
|
|
||||||
|
Un slot listé dont `getBlock` retourne `null` reste la tête de reprise du cycle suivant ; il n'est ni considéré vide ni marqué settled. Les slots absents de la liste de découverte sont traités comme non produits/skipped pour ce run. Le slot n'est settled qu'après admission réussie de toutes ses transactions. Le polling reste run-local : aucun checkpoint durable, aucun scan avant la borne initiale et aucun Backfill implicite ne sont créés. Les retries/reroutages HTTP restent possédés par `ksp-onchain-transport-lib`.
|
||||||
|
|
||||||
|
## Runtime et lifecycle
|
||||||
|
|
||||||
|
Le Worker s'exécute sur le runtime Tokio courant du caller. Il ne crée pas de runtime global et n'expose aucun `JoinHandle` public.
|
||||||
|
|
||||||
|
`RawTransactionIngestHandle` permet de :
|
||||||
|
|
||||||
|
- demander un stop coopératif et idempotent ;
|
||||||
|
- lire une source de snapshots concrets latest-value ;
|
||||||
|
- utiliser la même source via `WorkerSnapshotSource` ;
|
||||||
|
- attendre le terminal après drain et join des tâches possédées.
|
||||||
|
|
||||||
|
Le shutdown est borné par `shutdown_drain_timeout`. Le supervisor multi-source relaie le stop à toutes les sources et les rejoint avant de rendre son résultat au supervisor Worker ; les tâches source, hydration et persistence possédées sont ensuite drainées ou abort+join avant publication terminale. Si la deadline expire, l'abort du wrapper source détruit aussi son `JoinSet` interne et annule ses tâches imbriquées avant le terminal. Une faute déjà observée n'est pas remplacée par un stop concurrent, sauf le `drain_timeout` terminal lorsqu'une récupération bornée dépasse sa deadline. L'abandon terminal d'une hydration retire son pending run-local sans le convertir artificiellement en travail `settled`.
|
||||||
|
|
||||||
|
## Admission, coalescence et backpressure
|
||||||
|
|
||||||
|
La queue centrale est un `tokio::sync::mpsc` privé borné par `admission_queue_capacity`. Les sources internes subissent la backpressure ; aucune queue non bornée ni silent drop n'est autorisé.
|
||||||
|
|
||||||
|
Le Worker possède des coordinateurs source-neutral Yellowstone, Standard Logs et Helius Transaction branchés sur un registre global d'hydration partagé. Standard Block et HTTP Block Polling n'entrent pas dans ce registre lorsqu'une transaction est direct-qualified.
|
||||||
|
|
||||||
|
Les sources reference-bearing reçoivent des quotas déterministes dont la somme reste exactement dans les bornes techniques configurées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
somme pending source signals <= admission_queue_capacity
|
||||||
|
somme hydration tasks in flight <= persistence_concurrency
|
||||||
|
source reference-bearing active => quota pending >= 1 et quota in-flight >= 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour garantir simultanément ces bornes et l'absence de starvation structurelle, le démarrage échoue avant spawn si le nombre de sources reference-bearing dépasse `admission_queue_capacity` ou `persistence_concurrency`. Un sémaphore global protège en plus l'ouverture effective des hydrations HTTP.
|
||||||
|
|
||||||
|
Les signaux partageant le même `(network, signature, commitment)` sont coalescés cross-source avant le fan-out HTTP. La publication du résultat partagé notifie les followers sous le verrou de registry avant de retirer la clé : une nouvelle génération de leader ne peut donc pas s'intercaler entre retrait et notification. Après canonicalisation, une cache run-local bornée sérialise les acquisitions de même `(network, signature)` : la première passe par l'écriture atomique entity + observation, les suivantes de contenu canonique identique ajoutent uniquement leur observation déterministe. Une divergence de slot, block time, format ou hash canonique devient un content conflict terminal ; aucune majorité, préférence provider ou overwrite n'est appliqué. Le Store conserve son guard durable final.
|
||||||
|
|
||||||
|
Les retries/reroutages HTTP appartiennent à `ksp-onchain-transport-lib`. Le Worker ne possède pas une seconde boucle de retry autour de `getTransaction`.
|
||||||
|
|
||||||
|
## Processing frontier run-local
|
||||||
|
|
||||||
|
Le snapshot expose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydration_pending
|
||||||
|
processing_frontier_slot
|
||||||
|
oldest_pending_slot
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette frontier mesure uniquement le traitement des signaux réellement observés pendant le run courant. Elle n'est ni un checkpoint durable, ni une preuve de complétude blockchain, ni un curseur de Backfill.
|
||||||
|
|
||||||
|
Un signal transactionnel devient pending après validation de sa clé d'hydration et insertion dans le coordinateur. Il devient settled pour la source lorsque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
getTransaction -> Missing
|
||||||
|
ou
|
||||||
|
getTransaction -> Available puis ingress envoyé avec succès vers l'admission centrale
|
||||||
|
```
|
||||||
|
|
||||||
|
Un `BlockMeta` ou `Slot` continuity-only est settled localement sans produire de RAW. La frontier n'avance jamais à travers le plus ancien pending connu.
|
||||||
|
|
||||||
|
## Reconnect, replay et continuité
|
||||||
|
|
||||||
|
Le reconnect/replay Yellowstone appartient à Transport. Le Worker n'écrit pas `from_slot` et n'interprète pas directement `SubscribeReplayInfo`.
|
||||||
|
|
||||||
|
Le snapshot Worker projette uniquement des informations sûres et source-neutral :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_total
|
||||||
|
source_active
|
||||||
|
source_reconnecting
|
||||||
|
source_failed
|
||||||
|
source_state
|
||||||
|
source_failure_total
|
||||||
|
backpressure_wait_total
|
||||||
|
source_reconnect_total
|
||||||
|
source_replay_attempt_total
|
||||||
|
source_continuity_gap_total
|
||||||
|
```
|
||||||
|
|
||||||
|
Les quatre comptes de sources sont des gauges latest-value et ne contiennent aucune identité de source. `source_total` reste le nombre de sources logiques configurées pour le run. Tant que toutes les sources attendues ne sont pas `Active`, la health publique reste conservative : un reconnect transitoire projette `Degraded`, une source `Failed` projette `Unhealthy`, et le retour de toutes les sources attendues à `Active` permet de revenir à `Healthy`.
|
||||||
|
|
||||||
|
`RawTransactionIngestSourceState` distingue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Active
|
||||||
|
Reconnecting
|
||||||
|
Closing
|
||||||
|
Closed
|
||||||
|
Failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Un replay attempt n'est pas une preuve de replay réussi ni de continuité parfaite. Si Transport augmente son compteur de continuity gap parce que la borne de rétention prouve que le slot demandé n'est plus rejouable, le Worker classe la source en failure et termine avec `worker_raw_transaction_ingest.source_failed`.
|
||||||
|
|
||||||
|
Le Worker ne lance alors ni Job Backfill ni campagne historique automatique.
|
||||||
|
|
||||||
|
## Persistence Store
|
||||||
|
|
||||||
|
La persistance passe exclusivement par `ksp-store-lib` avec `default-features = false`. Aucun backend physique n'est importé directement.
|
||||||
|
|
||||||
|
L'écriture utilise le mode normal atomique `RawTransaction + RawTransactionObservation`. Les outcomes distingués incluent :
|
||||||
|
|
||||||
|
```text
|
||||||
|
entity inserted
|
||||||
|
entity already present
|
||||||
|
entity skipped purged
|
||||||
|
observation inserted
|
||||||
|
observation already present
|
||||||
|
observation not recorded for purged entity
|
||||||
|
content conflict
|
||||||
|
store failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Un content conflict est terminal et n'est jamais converti en succès idempotent.
|
||||||
|
|
||||||
|
## Snapshots et erreurs
|
||||||
|
|
||||||
|
`RawTransactionIngestSnapshotSource` est latest-value : les lecteurs peuvent rater des transitions intermédiaires mais récupèrent toujours la dernière projection complète et monotone.
|
||||||
|
|
||||||
|
Les codes Worker publics sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.settings_invalid
|
||||||
|
worker_raw_transaction_ingest.runtime_invalid
|
||||||
|
worker_raw_transaction_ingest.store_failed
|
||||||
|
worker_raw_transaction_ingest.content_conflict
|
||||||
|
worker_raw_transaction_ingest.counter_exhausted
|
||||||
|
worker_raw_transaction_ingest.source_failed
|
||||||
|
worker_raw_transaction_ingest.drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Les diagnostics et `Debug` ne recopient pas de payload RAW, signature, URL, credential, filtre provider, texte backend/provider arbitraire ou client inférieur. Les agrégations de compteurs de continuité multi-source utilisent une arithmétique vérifiée ; un overflow devient `counter_exhausted` au lieu d'être saturé silencieusement.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Les dépendances normales sont exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-logging-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
ksp-store-lib (default-features = false)
|
||||||
|
ksp-worker-api
|
||||||
|
sha2
|
||||||
|
tokio (macros, rt, sync, time)
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate ne dépend pas de Config, Job, `ksp-store-api` directement, backend Store concret, `reqwest`, `tonic`, `yellowstone-grpc-proto` ou Tauri.
|
||||||
|
|
||||||
|
## Hors périmètre
|
||||||
|
|
||||||
|
La verticale actuelle ne possède pas :
|
||||||
|
|
||||||
|
- sélection Config interne, reconfiguration dynamique ou policy de failover entre sources ;
|
||||||
|
- checkpoint persistent de processing frontier ;
|
||||||
|
- campagne de réparation historique automatique ;
|
||||||
|
- application Desk ou process autonome ;
|
||||||
|
- décodage STRUCTURAL/DECODED/DOMAIN.
|
||||||
|
|
||||||
|
Les futures extensions doivent conserver la séparation avec `ksp-job-backfill-lib` et réutiliser les mêmes contrats Common RAW/Store.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [`USAGE.md`](USAGE.md) — utilisation de la façade publique ;
|
||||||
|
- [`../ksp-worker-api/README.md`](../ksp-worker-api/README.md) — contrats Worker génériques ;
|
||||||
|
- [`../ksp-raw-transaction-lib/README.md`](../ksp-raw-transaction-lib/README.md) — canonicalisation RAW commune ;
|
||||||
|
- [`../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md`](../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md) — séparation Worker/Job ;
|
||||||
|
- [`../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`](../../docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md) — architecture d'acquisition RawTransaction.
|
||||||
422
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
Normal file
422
crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md
Normal file
@@ -0,0 +1,422 @@
|
|||||||
|
<!-- file: crates/ksp-worker-raw-transaction-ingest-lib/USAGE.md -->
|
||||||
|
<!-- version: 11 -->
|
||||||
|
|
||||||
|
# Utilisation de ksp-worker-raw-transaction-ingest-lib
|
||||||
|
|
||||||
|
Cette page décrit la façade publique de `ksp-worker-raw-transaction-ingest-lib`. Le caller possède la composition du runtime Tokio, du `Store` et des ressources Transport ; le Worker ne lit pas Config, ne construit pas un backend physique et ne lit pas de secret depuis l'environnement.
|
||||||
|
|
||||||
|
## Construire les settings
|
||||||
|
|
||||||
|
Les identités sont validées par leurs couches propriétaires :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn worker_settings() -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-0001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::with_defaults(network, worker_id));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker kind est fixe :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE, "raw_transaction_ingest");
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour des limites explicites :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let settings = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(
|
||||||
|
network,
|
||||||
|
worker_id,
|
||||||
|
512,
|
||||||
|
16,
|
||||||
|
std::time::Duration::from_secs(10),
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes publiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity 1 ..= 65_536 défaut 256
|
||||||
|
persistence_concurrency 1 ..= 64 défaut 8
|
||||||
|
shutdown_drain_timeout 100 ms ..= 30 s défaut 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
Une valeur hors borne retourne `worker_raw_transaction_ingest.settings_invalid` avec uniquement le nom stable du champ invalide.
|
||||||
|
|
||||||
|
## Choisir le mode de démarrage
|
||||||
|
|
||||||
|
### Fondation sans source productive
|
||||||
|
|
||||||
|
`RawTransactionIngestWorker::start(settings, store)` démarre la fondation runtime sans source Transport. Ce mode reste utile aux tests/compositions qui veulent uniquement le lifecycle, les snapshots et le contrat de shutdown.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start(settings, store) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### Source Yellowstone productive
|
||||||
|
|
||||||
|
Pour l'ingestion live, le caller compose d'abord les ressources via les crates propriétaires, puis construit :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn runtime_resources(
|
||||||
|
yellowstone_channel: ksp_onchain_transport_lib::YellowstoneGrpcChannel,
|
||||||
|
subscribe_request: ksp_onchain_transport_lib::YellowstoneSubscribeRequest,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource::new(
|
||||||
|
yellowstone_channel,
|
||||||
|
subscribe_request,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::new(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Puis :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let handle = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start_with_runtime_resources(
|
||||||
|
settings,
|
||||||
|
store,
|
||||||
|
runtime_resources,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deux entrées exigent un runtime Tokio courant et un `Store` portant exactement le même `RawNetworkId` que les settings. Le démarrage avec ressources exige également que chaque source composée cible ce même réseau. L'agrégat accepte 1 à 32 sources logiques et les lance simultanément ; les doublons d'identité et les mélanges de réseaux sont refusés avant spawn. Il n'existe pas de source primaire, standby ou fallback implicite : toute source configurée fait partie du run.
|
||||||
|
|
||||||
|
### Composer plusieurs sources simultanées
|
||||||
|
|
||||||
|
Une fois un premier `RawTransactionIngestRuntimeResources` construit, le caller ajoute les autres sources avec les méthodes `try_push_*` correspondant à leur capability. Exemple conceptuel :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let mut resources = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::new(yellowstone_source);
|
||||||
|
if let std::result::Result::Err(error) = resources.try_push_standard_logs_source(standard_logs_source) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = resources.try_push_http_block_polling_source(http_polling_source) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La validation est transactionnelle à chaque ajout : limite globale 32, réseau unique et `source_key` logique unique. Au démarrage, toutes les sources présentes sont supervisées ensemble. La défaillance d'une source est terminale pour le Worker ; les autres sources sont arrêtées et jointes, car le Worker ne suppose pas qu'elles couvrent les mêmes filtres ou le même univers de transactions.
|
||||||
|
|
||||||
|
### Source Standard Logs productive
|
||||||
|
|
||||||
|
Une source standard Solana WS se compose ainsi :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn standard_logs_runtime_resources(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource::new(
|
||||||
|
ws_endpoint,
|
||||||
|
filter,
|
||||||
|
commitment,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_logs_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`ws_endpoint` doit être un endpoint Transport valide de kind `solana_standard`. Le commitment doit être explicitement `Confirmed` ou `Finalized`. Le pool HTTP doit exposer `getTransaction` via le rôle indiqué sur le même réseau. `All`, `AllWithVotes` et `Mentions(pubkey)` sont acceptés par le contrat Transport ; la valeur du filtre reste privée dans le Worker.
|
||||||
|
|
||||||
|
### Source Standard Block productive
|
||||||
|
|
||||||
|
Une source `blockSubscribe` standard se compose sans pool HTTP :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn standard_block_runtime_resources(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::SolanaBlockSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource::new(ws_endpoint, filter, commitment) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_block_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker demande `Base64`, `Full`, `maxSupportedTransactionVersion = 1` et `showRewards = false`. Une transaction n'est RAW-direct que si sa version est explicitement `Legacy`, `0` ou `1`. Une version omise/nulle ou supérieure, `block: null`, une erreur de bloc, un champ transactions absent/nul ou une transaction non Base64 provoque une faute sûre ; ces cas ne sont jamais assimilés à une progression vide.
|
||||||
|
|
||||||
|
### Source Helius Transaction productive
|
||||||
|
|
||||||
|
Une source Helius LaserStream `transactionSubscribe` se compose avec le même pattern caller-owned :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn helius_transaction_runtime_resources(
|
||||||
|
ws_endpoint: ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
filter: ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
hydration_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new(
|
||||||
|
ws_endpoint,
|
||||||
|
filter,
|
||||||
|
commitment,
|
||||||
|
http_pool,
|
||||||
|
hydration_role,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`ws_endpoint` doit être un endpoint Transport de kind `helius_laserstream`. Le caller supérieur résout éventuellement `KSP_SECRET_HELIUS_API_KEY` via Config avant de construire l'endpoint ; le Worker ne lit jamais l'environnement ni Config. Le commitment est limité à `Confirmed`/`Finalized` et la route HTTP doit supporter `getTransaction` sur le même réseau.
|
||||||
|
|
||||||
|
Le Worker demande la forme Helius `Full` avec `Base64`, `showRewards = false` et `maxSupportedTransactionVersion = 1`, mais ne fait pas confiance au nested payload pour construire directement le Common RAW. Il conserve seulement signature/slot/index et hydrate par `getTransaction observed`. Une notification d'une autre forme est fail-closed.
|
||||||
|
|
||||||
|
### Source HTTP Block Polling productive
|
||||||
|
|
||||||
|
Une source HTTP live peut être composée sans WebSocket ni gRPC :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn http_block_polling_runtime_resources(
|
||||||
|
http_pool: ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
polling_role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
commitment: ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources> {
|
||||||
|
let source = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource::new(
|
||||||
|
http_pool,
|
||||||
|
polling_role,
|
||||||
|
commitment,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_http_block_polling_source(source));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rôle HTTP doit disposer, sur un même réseau, de `getSlot`, `getBlocksWithLimit` et `getBlock`. Le commitment est limité à `Confirmed`/`Finalized`. Par défaut, le Worker interroge toutes les secondes et borne la découverte à 128 blocs par cycle. `new_with_limits` permet de choisir une cadence entre 100 ms et 30 s et une limite entre 1 et 1024 blocs par cycle ; les limites de débit physiques restent celles de Transport.
|
||||||
|
|
||||||
|
Au démarrage du run, la première valeur `getSlot(commitment)` devient la borne inférieure stricte du poller. Il ne demande jamais de slot antérieur. `getBlocksWithLimit` détermine les slots réellement matérialisables, puis `getBlock observed` produit directement le Common RAW en Full/Base64 pour Legacy/V0/V1. Un slot listé dont `getBlock` retourne `null` reste la tête de reprise du prochain cycle et n'est jamais transformé en progression silencieuse.
|
||||||
|
|
||||||
|
## Préparer la source Yellowstone
|
||||||
|
|
||||||
|
La `YellowstoneSubscribeRequest` doit :
|
||||||
|
|
||||||
|
- être valide selon Transport ;
|
||||||
|
- contenir au moins une famille transaction-bearing admise par le Worker ;
|
||||||
|
- utiliser explicitement `Confirmed` ou `Finalized` ;
|
||||||
|
- rester compatible avec le réseau du `YellowstoneGrpcChannel`.
|
||||||
|
|
||||||
|
Le `HttpTransportPool` doit posséder au moins une route compatible avec le rôle d'hydration et `getTransaction` sur le même réseau. La construction de `RawTransactionIngestYellowstoneSource` vérifie ces invariants sans ouvrir la connexion réseau.
|
||||||
|
|
||||||
|
Le caller ne passe pas de signature, `program_id`, plage de slots ou limite historique au Worker. Ces paramètres appartiennent à un Job Backfill, pas au service continu.
|
||||||
|
|
||||||
|
## Comprendre la pipeline live
|
||||||
|
|
||||||
|
Les familles productives sont traitées ainsi :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone Transaction -> signal -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Yellowstone TransactionStatus -> signal -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Yellowstone Block -> un signal par transaction -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Standard WS logsSubscribe -> context.slot + signature -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
Standard WS blockSubscribe -> Full/Base64 Legacy|V0|V1 -> Common RAW direct par transaction -> admission
|
||||||
|
Helius transactionSubscribe -> Full envelope -> signature/slot/index -> HTTP getTransaction -> Common RAW -> admission
|
||||||
|
HTTP live block polling -> getSlot -> getBlocksWithLimit -> getBlock observed -> Common RAW direct Legacy|V0|V1 -> admission
|
||||||
|
Yellowstone BlockMeta -> continuity-only
|
||||||
|
Yellowstone Slot -> continuity-only
|
||||||
|
Yellowstone Account/Ping/Pong/Entry -> sans RAW Transaction dans cette verticale
|
||||||
|
```
|
||||||
|
|
||||||
|
Les signaux de même `(network, signature, commitment)` sont coalescés globalement avant l'hydration HTTP. Si plusieurs sources produisent ensuite la même transaction canonique, le Worker conserve une seule entité RAW et enregistre séparément les observations déterministes propres à chaque source. Le Worker ne possède pas une boucle de retry HTTP : reroutage/retry/backoff restent dans `ksp-onchain-transport-lib`.
|
||||||
|
|
||||||
|
Les sources qui nécessitent `getTransaction` partagent des quotas bornés et déterministes. Pour une composition valide :
|
||||||
|
|
||||||
|
```text
|
||||||
|
nombre de sources avec hydration <= admission_queue_capacity
|
||||||
|
nombre de sources avec hydration <= persistence_concurrency
|
||||||
|
somme de leurs pending <= admission_queue_capacity
|
||||||
|
somme de leurs tâches d'hydration actives <= persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces contraintes garantissent au moins une part à chaque source reference-bearing sans introduire de scheduler pondéré. Si les settings ne permettent pas cette répartition, le démarrage échoue avant spawn avec `runtime_invalid`; le caller doit augmenter la capacité concernée ou réduire le nombre de sources nécessitant une hydration.
|
||||||
|
|
||||||
|
Pour une même identité `(network, signature)`, une divergence canonique de slot, block time, format ou hash est un `content_conflict`. Le Worker ne choisit ni majorité ni provider préféré et n'écrase pas un contenu divergent ; le Store reste l'autorité durable finale du conflit.
|
||||||
|
|
||||||
|
## Observer le snapshot concret
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let current = source.current();
|
||||||
|
|
||||||
|
let sequence = current.worker_snapshot().sequence();
|
||||||
|
let state = current.worker_snapshot().state();
|
||||||
|
let queue_depth = current.admission_queue_depth();
|
||||||
|
let in_flight = current.in_flight_persistence();
|
||||||
|
let hydration_pending = current.hydration_pending();
|
||||||
|
let frontier = current.processing_frontier_slot();
|
||||||
|
let oldest_pending = current.oldest_pending_slot();
|
||||||
|
let source_total = current.source_total();
|
||||||
|
let source_active = current.source_active();
|
||||||
|
let source_reconnecting = current.source_reconnecting();
|
||||||
|
let source_failed = current.source_failed();
|
||||||
|
let source_state = current.source_state();
|
||||||
|
```
|
||||||
|
|
||||||
|
Pour attendre une valeur plus récente :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let observed = source.current().worker_snapshot().sequence();
|
||||||
|
let newer = source.wait_for_change(observed).await;
|
||||||
|
assert!(newer.worker_snapshot().sequence().is_after(observed));
|
||||||
|
```
|
||||||
|
|
||||||
|
Le flux est latest-value : les transitions intermédiaires peuvent être coalescées. Ne pas l'utiliser comme journal exhaustif.
|
||||||
|
|
||||||
|
## Utiliser la projection Worker API
|
||||||
|
|
||||||
|
La même source implémente `ksp_worker_api::WorkerSnapshotSource` :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let source = handle.worker_snapshot_source();
|
||||||
|
let current = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
let observed = current.sequence();
|
||||||
|
let newer = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, observed).await;
|
||||||
|
assert!(newer.sequence().is_after(observed));
|
||||||
|
```
|
||||||
|
|
||||||
|
La projection commune contient seulement les dimensions génériques Worker. Les compteurs et frontiers spécifiques restent sur `RawTransactionIngestSnapshot`.
|
||||||
|
|
||||||
|
## Lire les compteurs
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
|
||||||
|
let admitted = snapshot.admitted_total();
|
||||||
|
let canonicalized = snapshot.canonicalized_total();
|
||||||
|
let persisted = snapshot.persisted_total();
|
||||||
|
let inserted = snapshot.entity_inserted_total();
|
||||||
|
let existing = snapshot.entity_already_present_total();
|
||||||
|
let purged = snapshot.entity_skipped_purged_total();
|
||||||
|
let observations = snapshot.observation_inserted_total();
|
||||||
|
let conflicts = snapshot.content_conflict_total();
|
||||||
|
let store_failures = snapshot.store_failure_total();
|
||||||
|
let source_failures = snapshot.source_failure_total();
|
||||||
|
let backpressure = snapshot.backpressure_wait_total();
|
||||||
|
let reconnects = snapshot.source_reconnect_total();
|
||||||
|
let replay_attempts = snapshot.source_replay_attempt_total();
|
||||||
|
let proven_gaps = snapshot.source_continuity_gap_total();
|
||||||
|
```
|
||||||
|
|
||||||
|
`admission_queue_depth()`, `in_flight_persistence()`, `hydration_pending()`, `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` sont des gauges latest-value. Les compteurs cumulés et les agrégats multi-source ne wrapent ni ne saturent silencieusement ; l'épuisement est terminal avec `worker_raw_transaction_ingest.counter_exhausted`.
|
||||||
|
|
||||||
|
## Interpréter la processing frontier
|
||||||
|
|
||||||
|
`processing_frontier_slot()` est la plus haute slot de travail source réellement observé qui n'est pas bloquée par un pending plus ancien connu. `oldest_pending_slot()` expose ce plus ancien pending lorsqu'il existe.
|
||||||
|
|
||||||
|
Cette frontier est strictement run-local :
|
||||||
|
|
||||||
|
```text
|
||||||
|
elle ne prouve pas que toutes les transactions blockchain d'une slot ont été observées
|
||||||
|
elle ne prouve pas la persistence durable des ingress déjà envoyés à l'admission
|
||||||
|
elle n'est pas persistée entre deux runs
|
||||||
|
elle n'est pas un checkpoint Backfill
|
||||||
|
```
|
||||||
|
|
||||||
|
Un `Missing` HTTP règle le signal du point de vue source-processing sans créer de RAW. Un ingress `Available` n'est réglé qu'après envoi réussi vers l'admission centrale. Une hydration abandonnée au shutdown est retirée des pending sans faire avancer artificiellement la frontier.
|
||||||
|
|
||||||
|
## Interpréter reconnect et replay
|
||||||
|
|
||||||
|
`source_state()` peut retourner `Active`, `Reconnecting`, `Closing`, `Closed` ou `Failed` comme état agrégé source-neutral. Les gauges `source_total()`, `source_active()`, `source_reconnecting()` et `source_failed()` permettent d'interpréter une composition multi-source sans exposer provider, endpoint, filtre ou `source_key`.
|
||||||
|
|
||||||
|
La health commune est conservative : pendant `Running`, au moins une source `Failed` donne `Unhealthy`; au moins une source `Reconnecting`, ou une composition dont toutes les sources attendues ne sont pas encore `Active`, donne `Degraded`; toutes les sources attendues `Active` permettent `Healthy`. Une transition Worker `Faulted` reste `Unhealthy`.
|
||||||
|
|
||||||
|
Les compteurs ont des sémantiques distinctes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_reconnect_total reconnects automatiques réussis observés
|
||||||
|
source_replay_attempt_total tentatives de reconnect portant une demande de replay
|
||||||
|
source_continuity_gap_total gaps de rétention prouvés par Transport
|
||||||
|
```
|
||||||
|
|
||||||
|
Une tentative de replay n'est pas une preuve de continuité. Le Worker ne choisit pas `from_slot` et ne traite pas directement `SubscribeReplayInfo` ; ces mécanismes appartiennent à Transport.
|
||||||
|
|
||||||
|
Si `source_continuity_gap_total` augmente, le Worker fault avec `worker_raw_transaction_ingest.source_failed`. Il ne déclenche pas automatiquement `ksp-job-backfill-lib`.
|
||||||
|
|
||||||
|
## Demander un stop et attendre le terminal
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let accepted = handle.request_stop();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
|
||||||
|
match terminal {
|
||||||
|
std::result::Result::Ok(state) => {
|
||||||
|
assert!(state.is_terminal());
|
||||||
|
}
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
}
|
||||||
|
|
||||||
|
if accepted {
|
||||||
|
// La première demande de stop a été remise au runtime vivant.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`request_stop()` est idempotent. Le terminal n'est publié qu'après le drain borné et la récupération des tâches possédées. Si la deadline de drain expire, toutes les tâches source/persistence encore possédées sont abortées puis jointes avant publication terminale ; une persistence libérée après ce terminal ne peut donc pas produire une complétion tardive. Une faute source déjà observée reste prioritaire face à un stop concurrent, sauf si le drain lui-même expire et devient le terminal `drain_timeout`.
|
||||||
|
|
||||||
|
## Interpréter les faults
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings_invalid settings techniques hors contrat
|
||||||
|
runtime_invalid invariant runtime/lifecycle impossible
|
||||||
|
store_failed erreur Store non-conflict classifiée
|
||||||
|
content_conflict contenu canonique incompatible avec l'identité durable
|
||||||
|
counter_exhausted compteur/séquence monotone arrivé à sa borne
|
||||||
|
source_failed source/replay/hydration terminée par une erreur classifiée
|
||||||
|
drain_timeout drain de shutdown hors deadline
|
||||||
|
```
|
||||||
|
|
||||||
|
Un consumer doit traiter l'`ErrorCode` comme contrat stable et ne pas dépendre d'un texte backend/provider arbitraire.
|
||||||
|
|
||||||
|
## Composition supérieure
|
||||||
|
|
||||||
|
Le pattern attendu est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Config / application / service owner
|
||||||
|
-> résout endpoints, credentials et rôles
|
||||||
|
-> construit une source Transport Yellowstone, Standard Logs, Standard Block, Helius Transaction ou HTTP Block Polling
|
||||||
|
-> construit HttpTransportPool + rôle HTTP adapté à la source
|
||||||
|
-> construit le Store
|
||||||
|
-> construit RawTransactionIngestRuntimeResources
|
||||||
|
-> construit RawTransactionIngestSettings
|
||||||
|
-> démarre RawTransactionIngestWorker::start_with_runtime_resources
|
||||||
|
-> observe RawTransactionIngestSnapshotSource
|
||||||
|
-> demande stop lorsque nécessaire
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Worker ne reçoit pas de requête historique métier et ne dépend pas de Config. Une campagne `signature/program_id/plage/limite` appartient à `ksp-job-backfill-lib`.
|
||||||
110
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
Normal file
110
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
|
const RAW_TRANSACTION_INGEST_OBSERVATION_DOMAIN: &[u8] = b"ksp.raw_transaction_ingest.observation.v1\0";
|
||||||
|
|
||||||
|
/// Crate-private source-neutral ingress admitted by the central bounded Worker queue.
|
||||||
|
pub(crate) struct RawTransactionIngress {
|
||||||
|
/// Complete Common RAW material supplied by one private source task.
|
||||||
|
pub(crate) material: ksp_raw_transaction_lib::RawTransactionMaterial,
|
||||||
|
/// Logical network expected to match both Worker settings and canonical material.
|
||||||
|
pub(crate) network: ksp_store_lib::RawNetworkId,
|
||||||
|
/// Safe source-independent provenance attached to the acquisition observation.
|
||||||
|
pub(crate) provenance: ksp_store_lib::RawAcquisitionProvenance,
|
||||||
|
/// Opaque deterministic source-owned key material used only for Worker observation-key derivation.
|
||||||
|
pub(crate) source_key: [u8; 32],
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receiver side of the bounded central RAW transaction admission queue owned by the Worker supervisor.
|
||||||
|
pub(crate) struct RawTransactionAdmission {
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
capacity: usize,
|
||||||
|
receiver: tokio::sync::mpsc::Receiver<crate::RawTransactionIngress>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionAdmission {
|
||||||
|
/// Creates one bounded admission queue and returns its crate-private source sender.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn new(capacity: usize) -> (crate::RawTransactionAdmission, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) {
|
||||||
|
let (sender, receiver) = tokio::sync::mpsc::channel(capacity);
|
||||||
|
return (Self { backpressure_wait_observed: false, capacity, receiver }, sender);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest receiver-side queue depth without consuming an ingress entry.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn queue_depth(&self) -> usize {
|
||||||
|
return self.receiver.len();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Closes new admissions while preserving already queued ingress for deterministic drain.
|
||||||
|
pub(crate) fn close(&mut self) {
|
||||||
|
self.receiver.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Receives and converts one queued ingress into a common RAW transaction acquisition.
|
||||||
|
pub(crate) async fn receive(
|
||||||
|
&mut self,
|
||||||
|
expected_network: &ksp_store_lib::RawNetworkId,
|
||||||
|
) -> ksp_core_lib::Result<std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition>> {
|
||||||
|
self.backpressure_wait_observed = self.receiver.len() == self.capacity;
|
||||||
|
let ingress = match self.receiver.recv().await {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
||||||
|
};
|
||||||
|
let acquisition = canonicalize_ingress(expected_network, ingress);
|
||||||
|
return match acquisition {
|
||||||
|
std::result::Result::Ok(value) => std::result::Result::Ok(std::option::Option::Some(value)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Takes and clears the source-neutral signal that the previous dequeue observed the bounded queue at full capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn take_backpressure_wait_observed(&mut self) -> bool {
|
||||||
|
let observed = self.backpressure_wait_observed;
|
||||||
|
self.backpressure_wait_observed = false;
|
||||||
|
return observed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonicalize_ingress(
|
||||||
|
expected_network: &ksp_store_lib::RawNetworkId,
|
||||||
|
ingress: crate::RawTransactionIngress,
|
||||||
|
) -> ksp_core_lib::Result<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
if &ingress.network != expected_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("admission.network_mismatch"));
|
||||||
|
}
|
||||||
|
let transaction = ksp_raw_transaction_lib::canonicalize_raw_transaction(ingress.material);
|
||||||
|
let transaction = match transaction {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("admission.canonicalization_invalid")),
|
||||||
|
};
|
||||||
|
if transaction.reference().network() != expected_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("admission.material_network_mismatch"));
|
||||||
|
}
|
||||||
|
let observation_key = observation_key(transaction.reference(), &ingress.source_key);
|
||||||
|
return std::result::Result::Ok(ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(transaction, observation_key, ingress.provenance));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hash_bytes(hasher: &mut sha2::Sha256, value: &[u8]) {
|
||||||
|
hasher.update((value.len() as u64).to_be_bytes());
|
||||||
|
hasher.update(value);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn observation_key(reference: &ksp_store_lib::RawTransactionReference, source_key: &[u8; 32]) -> ksp_store_lib::RawObservationKey {
|
||||||
|
let mut hasher = sha2::Sha256::new();
|
||||||
|
hasher.update(RAW_TRANSACTION_INGEST_OBSERVATION_DOMAIN);
|
||||||
|
hash_bytes(&mut hasher, reference.network().as_str().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, reference.signature().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, source_key);
|
||||||
|
let bytes: [u8; 32] = hasher.finalize().into();
|
||||||
|
return ksp_store_lib::RawObservationKey::new(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/admission.rs"]
|
||||||
|
mod tests;
|
||||||
54
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
Normal file
54
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
/// Error code used when durable Store content conflicts with one admitted canonical RAW transaction.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "content_conflict");
|
||||||
|
/// Error code used when one monotone RAW transaction ingest Worker counter or snapshot sequence cannot advance without wrapping.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "counter_exhausted");
|
||||||
|
/// Error code used when private Worker tasks cannot drain before the configured shutdown deadline.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "drain_timeout");
|
||||||
|
/// Error code used when the RAW transaction ingest Worker reaches an invalid runtime or lifecycle condition.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "runtime_invalid");
|
||||||
|
/// Error code used when RAW transaction ingest Worker settings violate one bounded runtime invariant.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "settings_invalid");
|
||||||
|
/// Error code used when one private source task terminates with a classified failure.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "source_failed");
|
||||||
|
/// Error code used when Store persistence fails for one admitted canonical RAW transaction.
|
||||||
|
pub const ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "store_failed");
|
||||||
|
|
||||||
|
/// Creates one terminal counter-exhaustion error without exposing runtime material.
|
||||||
|
pub(crate) fn counter_exhausted_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED, "RAW transaction ingest Worker counter exhausted")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one terminal content-conflict error without copying conflicting material into diagnostics.
|
||||||
|
pub(crate) fn content_conflict_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT, "RAW transaction ingest Store content conflict");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one runtime-domain error carrying only one stable internal condition code.
|
||||||
|
pub(crate) fn runtime_error(condition: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID, "invalid RAW transaction ingest Worker runtime state")
|
||||||
|
.with_context("condition", condition);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one settings-domain error carrying only the stable invalid field name.
|
||||||
|
pub(crate) fn settings_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID, "invalid RAW transaction ingest Worker settings")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one terminal Store error while retaining only the already-safe lower-layer ErrorCode.
|
||||||
|
pub(crate) fn store_error(store_code: ksp_core_lib::ErrorCode) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED, "RAW transaction ingest Store persistence failed")
|
||||||
|
.with_context("store_domain", store_code.domain())
|
||||||
|
.with_context("store_code", store_code.code());
|
||||||
|
}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/identity.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Stable Worker kind code used by the continuous RAW transaction ingest vertical.
|
||||||
|
pub const RAW_TRANSACTION_INGEST_WORKER_KIND_CODE: &str = "raw_transaction_ingest";
|
||||||
140
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
Normal file
140
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
// version: 26
|
||||||
|
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
#![deny(unreachable_pub)]
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
|
//! Source-neutral runtime foundation for continuous KSP RAW transaction ingestion.
|
||||||
|
//! Owned source, hydration and persistence tasks are bounded and joined before terminal publication.
|
||||||
|
//!
|
||||||
|
//! This tranche owns the concrete Worker family identity, validated technical settings
|
||||||
|
//! and the caller-runtime-owned lifecycle with private child-task supervision. This tranche also
|
||||||
|
//! owns bounded source-neutral admission, common RAW canonicalization/assembly and backend-neutral
|
||||||
|
//! Store persistence in normal mode plus concrete latest-value snapshots projected onto Worker API. The bounded 1..32 caller-composed aggregate starts
|
||||||
|
//! all validated Yellowstone, Standard Logs, Standard Block, Helius Transaction and HTTP live block polling sources under one private supervisor. Reference-bearing
|
||||||
|
//! sources share bounded cross-source hydration and fairness budgets; direct-qualified Standard Block and HTTP polling transactions enter the existing central
|
||||||
|
//! admission path directly. Public snapshots expose only source-neutral aggregate counts/state and conservative Worker health. Yellowstone
|
||||||
|
//! Transaction/TransactionStatus/Block and standard logs notifications become signature/slot references; BlockMeta/Slot remain continuity-only signals.
|
||||||
|
//! Hydration is coalesced by network/signature/commitment under bounded in-flight and pending budgets. A bounded run-local processing frontier projects
|
||||||
|
//! hydration pending, oldest pending slot and highest unblocked actually observed slot. The productive source also projects safe Transport reconnect/replay
|
||||||
|
//! state and faults conservatively when Transport proves a replay-retention continuity gap; history remediation remains outside this crate.
|
||||||
|
|
||||||
|
mod admission;
|
||||||
|
mod error;
|
||||||
|
mod identity;
|
||||||
|
mod persistence;
|
||||||
|
mod runtime;
|
||||||
|
mod runtime_resources;
|
||||||
|
mod settings;
|
||||||
|
mod snapshot;
|
||||||
|
|
||||||
|
/// Error code used when durable Store content conflicts with one admitted canonical RAW transaction.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT;
|
||||||
|
/// Error code used when one monotone Worker counter or snapshot sequence cannot advance without wrapping.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED;
|
||||||
|
/// Error code used when private Worker tasks cannot drain before the configured shutdown deadline.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT;
|
||||||
|
/// Error code used when the RAW transaction ingest Worker reaches an invalid runtime or lifecycle condition.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID;
|
||||||
|
/// Error code used when RAW transaction ingest Worker settings violate one bounded runtime invariant.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID;
|
||||||
|
/// Error code used when one private source task terminates with a classified failure.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED;
|
||||||
|
/// Error code used when Store persistence fails for one admitted canonical RAW transaction.
|
||||||
|
pub use self::error::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED;
|
||||||
|
/// Stable Worker kind code used by the continuous RAW transaction ingest vertical.
|
||||||
|
pub use self::identity::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE;
|
||||||
|
/// Cloneable external control handle for one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime::RawTransactionIngestHandle;
|
||||||
|
/// Runtime-neutral boxed future resolving after one RAW transaction ingest Worker has fully reached a terminal lifecycle state.
|
||||||
|
pub use self::runtime::RawTransactionIngestTerminalFuture;
|
||||||
|
/// Entry point owning synchronous validation and task launch for one RAW transaction ingest Worker run.
|
||||||
|
pub use self::runtime::RawTransactionIngestWorker;
|
||||||
|
/// Default interval between HTTP live block polling cycles.
|
||||||
|
pub use self::runtime_resources::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL;
|
||||||
|
/// Default maximum number of confirmed blocks discovered during one HTTP live block polling cycle.
|
||||||
|
pub use self::runtime_resources::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE;
|
||||||
|
/// Maximum interval accepted between HTTP live block polling cycles.
|
||||||
|
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL;
|
||||||
|
/// Maximum number of confirmed blocks accepted during one HTTP live block polling cycle.
|
||||||
|
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE;
|
||||||
|
/// Maximum number of logical live sources accepted by one runtime-resource aggregate.
|
||||||
|
pub use self::runtime_resources::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES;
|
||||||
|
/// Minimum interval accepted between HTTP live block polling cycles.
|
||||||
|
pub use self::runtime_resources::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL;
|
||||||
|
/// Minimum number of confirmed blocks accepted during one HTTP live block polling cycle.
|
||||||
|
pub use self::runtime_resources::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE;
|
||||||
|
/// Validated Helius `transactionSubscribe` + HTTP hydration source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestHeliusTransactionSource;
|
||||||
|
/// Validated standard Solana HTTP live block polling source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestHttpBlockPollingSource;
|
||||||
|
/// Caller-composed bounded runtime resources for supported continuous RAW transaction live-source families.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestRuntimeResources;
|
||||||
|
/// Validated standard Solana `blockSubscribe` direct RAW source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestStandardBlockSource;
|
||||||
|
/// Validated standard Solana `logsSubscribe` + HTTP hydration source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestStandardLogsSource;
|
||||||
|
/// Validated Yellowstone + HTTP source contract owned by the continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::runtime_resources::RawTransactionIngestYellowstoneSource;
|
||||||
|
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Default number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Default cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Maximum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Maximum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Maximum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Minimum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY;
|
||||||
|
/// Minimum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY;
|
||||||
|
/// Minimum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT;
|
||||||
|
/// Validated source-neutral runtime settings for one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::settings::RawTransactionIngestSettings;
|
||||||
|
/// Complete safe latest-value snapshot of one continuous RAW transaction ingest Worker.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshot;
|
||||||
|
/// Runtime-neutral boxed future resolving to one newer concrete RAW transaction ingest Worker snapshot.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshotFuture;
|
||||||
|
/// Cloneable latest-value source exposing concrete and common Worker snapshots from one shared watch state.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSnapshotSource;
|
||||||
|
/// Source-neutral lifecycle state of the productive RAW transaction ingest source.
|
||||||
|
pub use self::snapshot::RawTransactionIngestSourceState;
|
||||||
|
|
||||||
|
/// Receiver-side owner of the private bounded RAW transaction admission queue.
|
||||||
|
pub(crate) use self::admission::RawTransactionAdmission;
|
||||||
|
/// Crate-private source-neutral ingress sent through the bounded central admission queue.
|
||||||
|
pub(crate) use self::admission::RawTransactionIngress;
|
||||||
|
/// Creates one terminal content-conflict error without copying conflicting material into diagnostics.
|
||||||
|
pub(crate) use self::error::content_conflict_error;
|
||||||
|
/// Creates one terminal counter-exhaustion error without exposing runtime material.
|
||||||
|
pub(crate) use self::error::counter_exhausted_error;
|
||||||
|
/// Creates one runtime-domain error without copying runtime/provider/Store values into diagnostics.
|
||||||
|
pub(crate) use self::error::runtime_error;
|
||||||
|
/// Creates one settings-domain error without copying caller-supplied values into diagnostics.
|
||||||
|
pub(crate) use self::error::settings_error;
|
||||||
|
/// Creates one terminal Store error while retaining only the already-safe lower-layer ErrorCode.
|
||||||
|
pub(crate) use self::error::store_error;
|
||||||
|
/// Canonical entity disposition produced by one successful Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestEntityPersistence;
|
||||||
|
/// Observation disposition produced by one successful Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestObservationPersistence;
|
||||||
|
/// Private bounded run-local cache serializing repeated canonical identities before Store writes.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistenceConvergence;
|
||||||
|
/// Classified successful outcome of one atomic Worker Store persistence attempt.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistenceOutcome;
|
||||||
|
/// Private backend-neutral Store persistence port used by the Worker and deterministic tests.
|
||||||
|
pub(crate) use self::persistence::RawTransactionIngestPersistencePort;
|
||||||
|
/// Persists one already-canonical Worker acquisition through the private Store port in `Normal` mode.
|
||||||
|
pub(crate) use self::persistence::persist_raw_transaction_ingest_acquisition;
|
||||||
|
/// Persists one canonical acquisition through the bounded cross-source convergence cache.
|
||||||
|
pub(crate) use self::persistence::persist_raw_transaction_ingest_converged_acquisition;
|
||||||
|
/// Private latest-value processing-frontier projection emitted by the productive source task.
|
||||||
|
pub(crate) use self::snapshot::RawTransactionIngestProcessingFrontierProjection;
|
||||||
|
/// Private latest-value publisher and checked counter owner shared by the Worker supervisor.
|
||||||
|
pub(crate) use self::snapshot::RawTransactionIngestSnapshotPublisher;
|
||||||
290
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
Normal file
290
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
Normal file
@@ -0,0 +1,290 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
/// Canonical entity disposition produced by one Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) enum RawTransactionIngestEntityPersistence {
|
||||||
|
/// The canonical RAW transaction was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// Identical canonical RAW transaction content was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// Normal persistence respected an existing purge tombstone.
|
||||||
|
SkippedPurged,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Observation disposition produced by one Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) enum RawTransactionIngestObservationPersistence {
|
||||||
|
/// The deterministic acquisition observation was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// The same deterministic acquisition observation was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// No observation was recorded because the canonical entity was intentionally skipped.
|
||||||
|
NotRecorded,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Classified successful outcome of one atomic Worker Store persistence attempt.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) struct RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
/// Returns the canonical entity persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) const fn entity(self) -> crate::RawTransactionIngestEntityPersistence {
|
||||||
|
return self.entity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the acquisition-observation persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) const fn observation(self) -> crate::RawTransactionIngestObservationPersistence {
|
||||||
|
return self.observation;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private backend-neutral Store persistence port used by the Worker and deterministic tests.
|
||||||
|
pub(crate) trait RawTransactionIngestPersistencePort: std::marker::Send + std::marker::Sync {
|
||||||
|
/// Returns whether the persistence target belongs to the requested logical network.
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool;
|
||||||
|
|
||||||
|
/// Executes one atomic canonical RAW transaction plus observation Store write.
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>>;
|
||||||
|
|
||||||
|
/// Records one additional observation for an already durable canonical RAW transaction.
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for ksp_store_lib::Store {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return self.runtime_snapshot().network() == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
return ksp_store_lib::RawTransactionWrite::persist_raw_transaction_acquisition(self, transaction, observation, mode);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>> {
|
||||||
|
return ksp_store_lib::RawTransactionObservationWrite::record_raw_transaction_observation(self, observation);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Eq, Ord, PartialEq, PartialOrd)]
|
||||||
|
struct RawTransactionIngestPersistenceKey {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
signature: ksp_store_lib::RawTransactionSignature,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
struct RawTransactionIngestCanonicalState {
|
||||||
|
block_time: std::option::Option<ksp_store_lib::RawTimestamp>,
|
||||||
|
content_hash: ksp_store_lib::RawContentHash,
|
||||||
|
format_id: ksp_store_lib::RawFormatId,
|
||||||
|
format_version: u32,
|
||||||
|
slot: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private bounded run-local cache serializing repeated canonical identities before Store writes.
|
||||||
|
pub(crate) struct RawTransactionIngestPersistenceConvergence {
|
||||||
|
max_entries: usize,
|
||||||
|
entries: std::sync::Mutex<
|
||||||
|
std::collections::BTreeMap<
|
||||||
|
RawTransactionIngestPersistenceKey,
|
||||||
|
std::sync::Arc<tokio::sync::Mutex<std::option::Option<RawTransactionIngestCanonicalState>>>,
|
||||||
|
>,
|
||||||
|
>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistenceConvergence {
|
||||||
|
/// Creates one bounded run-local convergence cache using the caller-provided effective runtime bound.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn new(max_entries: usize) -> Self {
|
||||||
|
return Self { max_entries, entries: std::sync::Mutex::new(std::collections::BTreeMap::new()) };
|
||||||
|
}
|
||||||
|
|
||||||
|
fn entry(
|
||||||
|
&self,
|
||||||
|
key: RawTransactionIngestPersistenceKey,
|
||||||
|
) -> std::option::Option<std::sync::Arc<tokio::sync::Mutex<std::option::Option<RawTransactionIngestCanonicalState>>>> {
|
||||||
|
let mut entries = match self.entries.lock() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(poisoned) => poisoned.into_inner(),
|
||||||
|
};
|
||||||
|
if let std::option::Option::Some(entry) = entries.get(&key) {
|
||||||
|
return std::option::Option::Some(std::sync::Arc::clone(entry));
|
||||||
|
}
|
||||||
|
if entries.len() >= self.max_entries {
|
||||||
|
let removable = entries.iter().find_map(|(candidate, entry)| {
|
||||||
|
if std::sync::Arc::strong_count(entry) == 1 {
|
||||||
|
return std::option::Option::Some(candidate.clone());
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
});
|
||||||
|
if let std::option::Option::Some(removable) = removable {
|
||||||
|
entries.remove(&removable);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if entries.len() >= self.max_entries {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
let entry = std::sync::Arc::new(tokio::sync::Mutex::new(std::option::Option::None));
|
||||||
|
entries.insert(key, std::sync::Arc::clone(&entry));
|
||||||
|
return std::option::Option::Some(entry);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Persists one already-canonical Worker acquisition through the private Store port in `Normal` mode.
|
||||||
|
pub(crate) async fn persist_raw_transaction_ingest_acquisition<P>(
|
||||||
|
port: &P,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>
|
||||||
|
where
|
||||||
|
P: crate::RawTransactionIngestPersistencePort + ?Sized,
|
||||||
|
{
|
||||||
|
let reference = acquisition.transaction().reference().clone();
|
||||||
|
if !port.network_matches(reference.network()) {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
if acquisition.observation().transaction() != &reference {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.acquisition_reference_mismatch"));
|
||||||
|
}
|
||||||
|
let (transaction, observation) = acquisition.into_parts();
|
||||||
|
let result = port.persist_acquisition(transaction, observation, ksp_store_lib::RawTransactionAcquisitionMode::Normal).await;
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(outcome) => map_store_outcome(outcome),
|
||||||
|
std::result::Result::Err(error) => map_store_error(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Persists one canonical acquisition through the bounded cross-source convergence cache.
|
||||||
|
pub(crate) async fn persist_raw_transaction_ingest_converged_acquisition<P>(
|
||||||
|
port: &P,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
convergence: &crate::RawTransactionIngestPersistenceConvergence,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>
|
||||||
|
where
|
||||||
|
P: crate::RawTransactionIngestPersistencePort + ?Sized,
|
||||||
|
{
|
||||||
|
let reference = acquisition.transaction().reference().clone();
|
||||||
|
if !port.network_matches(reference.network()) {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
if acquisition.observation().transaction() != &reference {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.acquisition_reference_mismatch"));
|
||||||
|
}
|
||||||
|
let canonical_state = canonical_state(acquisition.transaction());
|
||||||
|
let key = RawTransactionIngestPersistenceKey { network: reference.network().clone(), signature: reference.signature() };
|
||||||
|
let entry = convergence.entry(key);
|
||||||
|
let entry = match entry {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return crate::persist_raw_transaction_ingest_acquisition(port, acquisition).await,
|
||||||
|
};
|
||||||
|
let mut known_state = entry.lock().await;
|
||||||
|
if let std::option::Option::Some(known_state_value) = known_state.as_ref() {
|
||||||
|
if known_state_value != &canonical_state {
|
||||||
|
return std::result::Result::Err(crate::content_conflict_error());
|
||||||
|
}
|
||||||
|
let (_transaction, observation) = acquisition.into_parts();
|
||||||
|
let result = port.record_observation(observation).await;
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(outcome) => map_additional_observation_outcome(outcome),
|
||||||
|
std::result::Result::Err(error) => map_store_error(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
let outcome = crate::persist_raw_transaction_ingest_acquisition(port, acquisition).await;
|
||||||
|
let outcome = match outcome {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if outcome.entity() != crate::RawTransactionIngestEntityPersistence::SkippedPurged {
|
||||||
|
*known_state = std::option::Option::Some(canonical_state);
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(outcome);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonical_state(transaction: &ksp_store_lib::RawTransaction) -> RawTransactionIngestCanonicalState {
|
||||||
|
return RawTransactionIngestCanonicalState {
|
||||||
|
block_time: transaction.block_time(),
|
||||||
|
content_hash: transaction.payload().content_hash(),
|
||||||
|
format_id: transaction.payload().format_id().clone(),
|
||||||
|
format_version: transaction.payload().format_version(),
|
||||||
|
slot: transaction.slot(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_store_error(error: ksp_core_lib::Error) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
if error.code() == ksp_store_lib::ERROR_CODE_RAW_CONFLICT {
|
||||||
|
return std::result::Result::Err(crate::content_conflict_error());
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(crate::store_error(error.code()));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_store_outcome(outcome: ksp_store_lib::RawAcquisitionWriteOutcome) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
let entity = outcome.entity();
|
||||||
|
let observation = outcome.observation();
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::Inserted && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::Inserted,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::SkippedPurged && observation == ksp_store_lib::RawObservationWriteOutcome::NotRecorded {
|
||||||
|
return std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::SkippedPurged,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::NotRecorded,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(crate::runtime_error("persistence.store_outcome_invalid"));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_additional_observation_outcome(
|
||||||
|
outcome: ksp_store_lib::RawObservationWriteOutcome,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome> {
|
||||||
|
return match outcome {
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::Inserted => std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
}),
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent => std::result::Result::Ok(crate::RawTransactionIngestPersistenceOutcome {
|
||||||
|
entity: crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
observation: crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
}),
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::NotRecorded => {
|
||||||
|
std::result::Result::Err(crate::runtime_error("persistence.additional_observation_not_recorded"))
|
||||||
|
},
|
||||||
|
_ => std::result::Result::Err(crate::runtime_error("persistence.additional_observation_outcome_invalid")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/persistence.rs"]
|
||||||
|
mod tests;
|
||||||
662
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
Normal file
662
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
Normal file
@@ -0,0 +1,662 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
// version: 17
|
||||||
|
|
||||||
|
type PersistencePort = std::sync::Arc<dyn crate::RawTransactionIngestPersistencePort + 'static>;
|
||||||
|
type PersistenceTasks = tokio::task::JoinSet<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>>;
|
||||||
|
type ProcessingFrontierReceiver = tokio::sync::watch::Receiver<crate::RawTransactionIngestProcessingFrontierProjection>;
|
||||||
|
type SourceTasks = tokio::task::JoinSet<ksp_core_lib::Result<()>>;
|
||||||
|
|
||||||
|
/// Runtime-neutral boxed future resolving after one RAW transaction ingest Worker has fully reached a terminal lifecycle state.
|
||||||
|
pub type RawTransactionIngestTerminalFuture<'a> =
|
||||||
|
std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = ksp_core_lib::Result<ksp_worker_api::WorkerState>> + std::marker::Send + 'a>>;
|
||||||
|
|
||||||
|
/// Cloneable external control handle for one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct RawTransactionIngestHandle {
|
||||||
|
snapshots: crate::RawTransactionIngestSnapshotSource,
|
||||||
|
stop_sender: tokio::sync::watch::Sender<bool>,
|
||||||
|
stop_token: ksp_worker_api::WorkerStopToken,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestHandle {
|
||||||
|
/// Requests cooperative stop and returns `true` only for the first request accepted by the live runtime.
|
||||||
|
#[must_use]
|
||||||
|
pub fn request_stop(&self) -> bool {
|
||||||
|
if !self.stop_token.request_stop() {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return self.stop_sender.send(true).is_ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns an independent latest-value source for concrete RAW transaction ingest Worker snapshots.
|
||||||
|
#[must_use]
|
||||||
|
pub fn snapshot_source(&self) -> crate::RawTransactionIngestSnapshotSource {
|
||||||
|
return self.snapshots.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns an independent source implementing the common [`ksp_worker_api::WorkerSnapshotSource`] projection.
|
||||||
|
#[must_use]
|
||||||
|
pub fn worker_snapshot_source(&self) -> crate::RawTransactionIngestSnapshotSource {
|
||||||
|
return self.snapshots.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits until the private runtime task has published one terminal lifecycle state after draining owned work.
|
||||||
|
#[must_use]
|
||||||
|
pub fn wait_terminal(&self) -> crate::RawTransactionIngestTerminalFuture<'_> {
|
||||||
|
let source = self.snapshots.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = source.current();
|
||||||
|
let state = current.worker_snapshot().state();
|
||||||
|
if state.is_terminal() {
|
||||||
|
return std::result::Result::Ok(state);
|
||||||
|
}
|
||||||
|
let observed = current.worker_snapshot().sequence();
|
||||||
|
let changed = source.wait_for_change(observed).await;
|
||||||
|
let state = changed.worker_snapshot().state();
|
||||||
|
if state.is_terminal() {
|
||||||
|
return std::result::Result::Ok(state);
|
||||||
|
}
|
||||||
|
if changed.worker_snapshot().sequence() == observed && source.is_closed() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("terminal.closed_before_terminal"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestHandle {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let snapshot = self.snapshots.current();
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestHandle")
|
||||||
|
.field("stop_requested", &self.stop_token.is_stop_requested())
|
||||||
|
.field("sequence", &snapshot.worker_snapshot().sequence())
|
||||||
|
.field("state", &snapshot.worker_snapshot().state())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Entry point owning synchronous validation and task launch for one RAW transaction ingest Worker run.
|
||||||
|
pub struct RawTransactionIngestWorker;
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestWorker {
|
||||||
|
/// Starts one Worker on the caller-owned current Tokio runtime while retaining the caller-owned Store facade through an `Arc`.
|
||||||
|
pub fn start(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
store: std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
let runtime = match current_runtime_handle() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let store_snapshot = store.runtime_snapshot();
|
||||||
|
if let std::result::Result::Err(error) = validate_store_network(&settings, store_snapshot.network()) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
return start_foundation(settings, runtime, std::option::Option::Some(store));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Starts one Worker with caller-composed runtime resources.
|
||||||
|
///
|
||||||
|
/// Every validated source in the bounded runtime-resource aggregate is started concurrently under one private source supervisor. Any configured source
|
||||||
|
/// failure remains terminal for the Worker because this release does not infer equivalent coverage or failover between source families.
|
||||||
|
pub fn start_with_runtime_resources(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
store: std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
runtime_resources: crate::RawTransactionIngestRuntimeResources,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
let runtime = match current_runtime_handle() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let store_snapshot = store.runtime_snapshot();
|
||||||
|
if let std::result::Result::Err(error) = validate_store_network(&settings, store_snapshot.network()) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = runtime_resources.validate_network(settings.network()) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let source_total = runtime_resources.source_count();
|
||||||
|
let source_settings = settings.clone();
|
||||||
|
let (processing_frontier_sender, processing_frontier_receiver) =
|
||||||
|
tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty());
|
||||||
|
let port: PersistencePort = store;
|
||||||
|
return start_foundation_with_port_source_spawner_and_frontier(
|
||||||
|
settings,
|
||||||
|
runtime,
|
||||||
|
std::option::Option::Some(port),
|
||||||
|
std::option::Option::Some(processing_frontier_receiver),
|
||||||
|
source_total,
|
||||||
|
move |children, stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
return runtime_resources.run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender).await;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn begin_stopping(
|
||||||
|
lifecycle: &mut ksp_worker_api::WorkerLifecycle,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
if lifecycle.mark_stopping().is_err() {
|
||||||
|
return std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = snapshots.publish_state(lifecycle.state(), admission_queue_depth, in_flight_persistence) {
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn current_runtime_handle() -> ksp_core_lib::Result<tokio::runtime::Handle> {
|
||||||
|
return match tokio::runtime::Handle::try_current() {
|
||||||
|
std::result::Result::Ok(value) => std::result::Result::Ok(value),
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(crate::runtime_error("start.runtime_unavailable")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_children(
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
while let std::option::Option::Some(joined) = children.join_next().await {
|
||||||
|
let current = source_completion(joined, state, 0, 0, snapshots);
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_persistence(
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
admission: &crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
while let std::option::Option::Some(joined) = persistence.join_next().await {
|
||||||
|
let current = persistence_completion(joined, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots);
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_admission_and_persistence(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut fault = std::option::Option::None;
|
||||||
|
admission.close();
|
||||||
|
loop {
|
||||||
|
while persistence.len() >= settings.persistence_concurrency() {
|
||||||
|
let joined = persistence.join_next().await;
|
||||||
|
let current = match joined {
|
||||||
|
std::option::Option::Some(value) => persistence_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots),
|
||||||
|
std::option::Option::None => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
};
|
||||||
|
fault = merge_fault(fault, current);
|
||||||
|
}
|
||||||
|
let received = admission.receive(settings.network()).await;
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
match received {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(acquisition)) => {
|
||||||
|
let spawned = spawn_persistence(persistence, persistence_convergence, port, acquisition);
|
||||||
|
let published = snapshots.record_admission_success(lifecycle.state(), admission.queue_depth(), persistence.len(), backpressure_wait_observed);
|
||||||
|
if !spawned && fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = published
|
||||||
|
&& fault.is_none()
|
||||||
|
{
|
||||||
|
fault = std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => break,
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
let published = snapshots.record_admission_failure(lifecycle.state(), admission.queue_depth(), persistence.len(), backpressure_wait_observed);
|
||||||
|
match published {
|
||||||
|
std::result::Result::Ok(()) => {
|
||||||
|
if fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
if fault.is_none() {
|
||||||
|
fault = std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let persistence_fault = drain_persistence(lifecycle, admission, persistence, snapshots).await;
|
||||||
|
fault = merge_fault(fault, persistence_fault);
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn drain_owned_work(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let drain = async {
|
||||||
|
let mut fault = drain_admission_and_persistence(settings, lifecycle, admission, persistence, persistence_convergence, port, snapshots).await;
|
||||||
|
let child_fault = drain_children(lifecycle.state(), children, snapshots).await;
|
||||||
|
fault = merge_fault(fault, child_fault);
|
||||||
|
return fault;
|
||||||
|
};
|
||||||
|
let timed = tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await;
|
||||||
|
return match timed {
|
||||||
|
std::result::Result::Ok(fault) => fault,
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
admission.close();
|
||||||
|
persistence.abort_all();
|
||||||
|
children.abort_all();
|
||||||
|
while persistence.join_next().await.is_some() {}
|
||||||
|
while children.join_next().await.is_some() {}
|
||||||
|
std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT)
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finish_faulted(
|
||||||
|
lifecycle: &mut ksp_worker_api::WorkerLifecycle,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
code: ksp_core_lib::ErrorCode,
|
||||||
|
) {
|
||||||
|
if lifecycle.fault(code).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if snapshots.publish_state(lifecycle.state(), 0, 0).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finish_stopped(lifecycle: &mut ksp_worker_api::WorkerLifecycle, snapshots: &mut crate::RawTransactionIngestSnapshotPublisher) {
|
||||||
|
if lifecycle.mark_stopped().is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if snapshots.publish_state(lifecycle.state(), 0, 0).is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn merge_fault(
|
||||||
|
first: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
next: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
if first.is_some() {
|
||||||
|
return first;
|
||||||
|
}
|
||||||
|
return next;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persistence_completion(
|
||||||
|
joined: std::result::Result<ksp_core_lib::Result<crate::RawTransactionIngestPersistenceOutcome>, tokio::task::JoinError>,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
return match joined {
|
||||||
|
std::result::Result::Ok(std::result::Result::Ok(outcome)) => {
|
||||||
|
match snapshots.record_persistence_success(state, admission_queue_depth, in_flight_persistence, outcome) {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::None,
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(error)) => {
|
||||||
|
let code = error.code();
|
||||||
|
let published = snapshots.record_persistence_fault(state, admission_queue_depth, in_flight_persistence, code);
|
||||||
|
match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(code),
|
||||||
|
std::result::Result::Err(snapshot_error) => std::option::Option::Some(snapshot_error.code()),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn source_completion(
|
||||||
|
joined: std::result::Result<ksp_core_lib::Result<()>, tokio::task::JoinError>,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let source_code = match joined {
|
||||||
|
std::result::Result::Ok(std::result::Result::Ok(())) => return std::option::Option::None,
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(error)) if error.code() == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED => {
|
||||||
|
crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::result::Result::Err(_)) | std::result::Result::Err(_) => crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED,
|
||||||
|
};
|
||||||
|
let published = snapshots.record_source_failure(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
return match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(source_code),
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn run_supervisor<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
mut lifecycle: ksp_worker_api::WorkerLifecycle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
mut stop_receiver: tokio::sync::watch::Receiver<bool>,
|
||||||
|
mut snapshots: crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
mut processing_frontier_receiver: std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
if *stop_receiver.borrow() {
|
||||||
|
let stopping_fault = begin_stopping(&mut lifecycle, &mut snapshots, 0, 0);
|
||||||
|
match stopping_fault {
|
||||||
|
std::option::Option::Some(code) => finish_faulted(&mut lifecycle, &mut snapshots, code),
|
||||||
|
std::option::Option::None => finish_stopped(&mut lifecycle, &mut snapshots),
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if lifecycle.mark_running().is_err() {
|
||||||
|
snapshots.force_terminal(ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = snapshots.publish_state(lifecycle.state(), 0, 0) {
|
||||||
|
finish_faulted(&mut lifecycle, &mut snapshots, error.code());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let mut children = SourceTasks::new();
|
||||||
|
let mut persistence = PersistenceTasks::new();
|
||||||
|
let persistence_convergence = std::sync::Arc::new(crate::RawTransactionIngestPersistenceConvergence::new(
|
||||||
|
settings.admission_queue_capacity().max(settings.persistence_concurrency()),
|
||||||
|
));
|
||||||
|
let (source_stop_sender, source_stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (mut admission, admission_sender) = crate::RawTransactionAdmission::new(settings.admission_queue_capacity());
|
||||||
|
source_spawner(&mut children, source_stop_receiver, admission_sender);
|
||||||
|
let mut fault = supervise_until_stop(
|
||||||
|
&settings,
|
||||||
|
&lifecycle,
|
||||||
|
&source_stop_sender,
|
||||||
|
&mut stop_receiver,
|
||||||
|
&mut children,
|
||||||
|
&mut admission,
|
||||||
|
&mut persistence,
|
||||||
|
&persistence_convergence,
|
||||||
|
&port,
|
||||||
|
&mut snapshots,
|
||||||
|
&mut processing_frontier_receiver,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
let stopping_fault = begin_stopping(&mut lifecycle, &mut snapshots, admission.queue_depth(), persistence.len());
|
||||||
|
fault = merge_fault(fault, stopping_fault);
|
||||||
|
let drain_fault =
|
||||||
|
drain_owned_work(&settings, &lifecycle, &mut children, &mut admission, &mut persistence, &persistence_convergence, &port, &mut snapshots).await;
|
||||||
|
match drain_fault {
|
||||||
|
std::option::Option::Some(code) if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT => {
|
||||||
|
fault = std::option::Option::Some(code);
|
||||||
|
},
|
||||||
|
std::option::Option::Some(code) => {
|
||||||
|
fault = merge_fault(fault, std::option::Option::Some(code));
|
||||||
|
},
|
||||||
|
std::option::Option::None => {},
|
||||||
|
}
|
||||||
|
match fault {
|
||||||
|
std::option::Option::Some(code) => finish_faulted(&mut lifecycle, &mut snapshots, code),
|
||||||
|
std::option::Option::None => finish_stopped(&mut lifecycle, &mut snapshots),
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn spawn_persistence(
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
acquisition: ksp_raw_transaction_lib::RawTransactionAcquisition,
|
||||||
|
) -> bool {
|
||||||
|
let port = match port {
|
||||||
|
std::option::Option::Some(value) => std::sync::Arc::clone(value),
|
||||||
|
std::option::Option::None => return false,
|
||||||
|
};
|
||||||
|
let persistence_convergence = std::sync::Arc::clone(persistence_convergence);
|
||||||
|
let _abort_handle = persistence.spawn(async move {
|
||||||
|
return crate::persist_raw_transaction_ingest_converged_acquisition(port.as_ref(), acquisition, persistence_convergence.as_ref()).await;
|
||||||
|
});
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
store_guard: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle> {
|
||||||
|
return start_foundation_with_source_spawner(settings, runtime, store_guard, |_children, _stop_receiver, _admission_sender| {});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_port_and_source_spawner<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
return start_foundation_with_port_source_spawner_and_frontier(settings, runtime, port, std::option::Option::None, 0, source_spawner);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_port_source_spawner_and_frontier<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
port: std::option::Option<PersistencePort>,
|
||||||
|
processing_frontier_receiver: std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
source_total: usize,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
let kind = match ksp_worker_api::WorkerKindCode::new(crate::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::runtime_error("start.worker_kind_invalid")),
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_worker_api::WorkerLifecycle::new(settings.worker_id().clone(), kind);
|
||||||
|
if lifecycle.start().is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("start.lifecycle_invalid"));
|
||||||
|
}
|
||||||
|
let stop_token = ksp_worker_api::WorkerStopToken::new();
|
||||||
|
let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (snapshots, snapshot_source) = crate::RawTransactionIngestSnapshotPublisher::new(&settings, &lifecycle, source_total);
|
||||||
|
let handle = crate::RawTransactionIngestHandle { snapshots: snapshot_source, stop_sender, stop_token };
|
||||||
|
std::mem::drop(runtime.spawn(run_supervisor(settings, lifecycle, port, stop_receiver, snapshots, processing_frontier_receiver, source_spawner)));
|
||||||
|
return std::result::Result::Ok(handle);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn start_foundation_with_source_spawner<Spawner>(
|
||||||
|
settings: crate::RawTransactionIngestSettings,
|
||||||
|
runtime: tokio::runtime::Handle,
|
||||||
|
store_guard: std::option::Option<std::sync::Arc<ksp_store_lib::Store>>,
|
||||||
|
source_spawner: Spawner,
|
||||||
|
) -> ksp_core_lib::Result<crate::RawTransactionIngestHandle>
|
||||||
|
where
|
||||||
|
Spawner:
|
||||||
|
FnOnce(&mut SourceTasks, tokio::sync::watch::Receiver<bool>, tokio::sync::mpsc::Sender<crate::RawTransactionIngress>) + std::marker::Send + 'static,
|
||||||
|
{
|
||||||
|
let port = match store_guard {
|
||||||
|
std::option::Option::Some(store) => {
|
||||||
|
let port: PersistencePort = store;
|
||||||
|
std::option::Option::Some(port)
|
||||||
|
},
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
return start_foundation_with_port_and_source_spawner(settings, runtime, port, source_spawner);
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn supervise_until_stop(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
source_stop_sender: &tokio::sync::watch::Sender<bool>,
|
||||||
|
stop_receiver: &mut tokio::sync::watch::Receiver<bool>,
|
||||||
|
children: &mut SourceTasks,
|
||||||
|
admission: &mut crate::RawTransactionAdmission,
|
||||||
|
persistence: &mut PersistenceTasks,
|
||||||
|
persistence_convergence: &std::sync::Arc<crate::RawTransactionIngestPersistenceConvergence>,
|
||||||
|
port: &std::option::Option<PersistencePort>,
|
||||||
|
snapshots: &mut crate::RawTransactionIngestSnapshotPublisher,
|
||||||
|
processing_frontier_receiver: &mut std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
let mut admission_open = true;
|
||||||
|
loop {
|
||||||
|
if *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if children.is_empty() && !admission_open && persistence.is_empty() {
|
||||||
|
let changed = stop_receiver.changed().await;
|
||||||
|
if changed.is_err() || *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
tokio::select! {
|
||||||
|
biased;
|
||||||
|
changed = stop_receiver.changed() => {
|
||||||
|
if changed.is_err() || *stop_receiver.borrow() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
processing_frontier = wait_processing_frontier(processing_frontier_receiver) => {
|
||||||
|
match processing_frontier {
|
||||||
|
std::option::Option::Some(projection) => {
|
||||||
|
let published = snapshots.record_processing_frontier(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
projection,
|
||||||
|
);
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::option::Option::None => {
|
||||||
|
*processing_frontier_receiver = std::option::Option::None;
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
joined = children.join_next(), if !children.is_empty() => {
|
||||||
|
let source_fault = match joined {
|
||||||
|
std::option::Option::Some(value) => source_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots),
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
if source_fault.is_some() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return source_fault;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
joined = persistence.join_next(), if !persistence.is_empty() => {
|
||||||
|
match joined {
|
||||||
|
std::option::Option::Some(value) => {
|
||||||
|
let fault = persistence_completion(value, lifecycle.state(), admission.queue_depth(), persistence.len(), snapshots);
|
||||||
|
if fault.is_some() {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return fault;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::option::Option::None => {},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
received = admission.receive(settings.network()), if admission_open && persistence.len() < settings.persistence_concurrency() => {
|
||||||
|
match received {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(acquisition)) => {
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
let spawned = spawn_persistence(persistence, persistence_convergence, port, acquisition);
|
||||||
|
let published = snapshots.record_admission_success(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
backpressure_wait_observed,
|
||||||
|
);
|
||||||
|
if !spawned {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Ok(std::option::Option::None) => {
|
||||||
|
let _backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
admission_open = false;
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
let backpressure_wait_observed = admission.take_backpressure_wait_observed();
|
||||||
|
let published = snapshots.record_admission_failure(
|
||||||
|
lifecycle.state(),
|
||||||
|
admission.queue_depth(),
|
||||||
|
persistence.len(),
|
||||||
|
backpressure_wait_observed,
|
||||||
|
);
|
||||||
|
source_stop_sender.send_replace(true);
|
||||||
|
return match published {
|
||||||
|
std::result::Result::Ok(()) => std::option::Option::Some(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID),
|
||||||
|
std::result::Result::Err(error) => std::option::Option::Some(error.code()),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_processing_frontier(
|
||||||
|
receiver: &mut std::option::Option<ProcessingFrontierReceiver>,
|
||||||
|
) -> std::option::Option<crate::RawTransactionIngestProcessingFrontierProjection> {
|
||||||
|
let receiver = match receiver.as_mut() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => {
|
||||||
|
return std::future::pending::<std::option::Option<crate::RawTransactionIngestProcessingFrontierProjection>>().await;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
if receiver.changed().await.is_err() {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(*receiver.borrow_and_update());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_store_network(settings: &crate::RawTransactionIngestSettings, store_network: &ksp_store_lib::RawNetworkId) -> ksp_core_lib::Result<()> {
|
||||||
|
if settings.network() != store_network {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("start.store_network_mismatch"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/runtime.rs"]
|
||||||
|
mod tests;
|
||||||
File diff suppressed because it is too large
Load Diff
131
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
Normal file
131
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
Normal file
@@ -0,0 +1,131 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
/// Default bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 256;
|
||||||
|
/// Default number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 8;
|
||||||
|
/// Default cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
|
||||||
|
/// Maximum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 65_536;
|
||||||
|
/// Maximum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 64;
|
||||||
|
/// Maximum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||||
|
/// Minimum bounded admission queue capacity for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY: usize = 1;
|
||||||
|
/// Minimum number of concurrent Store persistence operations for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY: usize = 1;
|
||||||
|
/// Minimum cooperative shutdown drain deadline for one RAW transaction ingest Worker.
|
||||||
|
pub const MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(100);
|
||||||
|
|
||||||
|
/// Validated source-neutral runtime settings for one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawTransactionIngestSettings {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
worker_id: ksp_worker_api::WorkerId,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSettings {
|
||||||
|
/// Creates settings from validated network/Worker identities and explicit bounded runtime limits.
|
||||||
|
pub fn new(
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
worker_id: ksp_worker_api::WorkerId,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
) -> ksp_core_lib::Result<Self> {
|
||||||
|
if let std::result::Result::Err(error) = validate_inclusive_usize(
|
||||||
|
admission_queue_capacity,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
"admission_queue_capacity",
|
||||||
|
) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = validate_inclusive_usize(
|
||||||
|
persistence_concurrency,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
"persistence_concurrency",
|
||||||
|
) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if !(crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT..=crate::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT)
|
||||||
|
.contains(&shutdown_drain_timeout)
|
||||||
|
{
|
||||||
|
return std::result::Result::Err(crate::settings_error("shutdown_drain_timeout"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(Self { network, worker_id, admission_queue_capacity, persistence_concurrency, shutdown_drain_timeout });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates settings using the stable V1 runtime defaults for one validated network and Worker identity.
|
||||||
|
#[must_use]
|
||||||
|
pub fn with_defaults(network: ksp_store_lib::RawNetworkId, worker_id: ksp_worker_api::WorkerId) -> Self {
|
||||||
|
return Self {
|
||||||
|
network,
|
||||||
|
worker_id,
|
||||||
|
admission_queue_capacity: crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
persistence_concurrency: crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
shutdown_drain_timeout: crate::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the Store-compatible logical network binding.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
||||||
|
return &self.network;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated logical identity of this Worker instance.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn worker_id(&self) -> &ksp_worker_api::WorkerId {
|
||||||
|
return &self.worker_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded central admission queue capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_capacity(&self) -> usize {
|
||||||
|
return self.admission_queue_capacity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded maximum number of concurrent Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persistence_concurrency(&self) -> usize {
|
||||||
|
return self.persistence_concurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded cooperative shutdown drain deadline.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn shutdown_drain_timeout(&self) -> std::time::Duration {
|
||||||
|
return self.shutdown_drain_timeout;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSettings {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSettings")
|
||||||
|
.field("network", &self.network)
|
||||||
|
.field("worker_id", &self.worker_id)
|
||||||
|
.field("admission_queue_capacity", &self.admission_queue_capacity)
|
||||||
|
.field("persistence_concurrency", &self.persistence_concurrency)
|
||||||
|
.field("shutdown_drain_timeout", &self.shutdown_drain_timeout)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_inclusive_usize(value: usize, minimum: usize, maximum: usize, field: &'static str) -> ksp_core_lib::Result<()> {
|
||||||
|
if !(minimum..=maximum).contains(&value) {
|
||||||
|
return std::result::Result::Err(crate::settings_error(field));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/settings.rs"]
|
||||||
|
mod tests;
|
||||||
820
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
Normal file
820
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
Normal file
@@ -0,0 +1,820 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
/// Runtime-neutral boxed future resolving to one newer concrete RAW transaction ingest Worker snapshot.
|
||||||
|
pub type RawTransactionIngestSnapshotFuture<'a> =
|
||||||
|
std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = crate::RawTransactionIngestSnapshot> + std::marker::Send + 'a>>;
|
||||||
|
|
||||||
|
/// Source-neutral lifecycle state of the productive RAW transaction ingest source.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub enum RawTransactionIngestSourceState {
|
||||||
|
/// The productive source is active.
|
||||||
|
Active,
|
||||||
|
/// Transport is performing one bounded reconnect/replay attempt.
|
||||||
|
Reconnecting,
|
||||||
|
/// Cooperative source shutdown has started.
|
||||||
|
Closing,
|
||||||
|
/// The productive source closed cleanly.
|
||||||
|
Closed,
|
||||||
|
/// The productive source reached a terminal failure.
|
||||||
|
Failed,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private latest-value source-processing projection emitted by the productive source task.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) struct RawTransactionIngestProcessingFrontierProjection {
|
||||||
|
hydration_pending: usize,
|
||||||
|
processing_frontier_slot: std::option::Option<u64>,
|
||||||
|
oldest_pending_slot: std::option::Option<u64>,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
source_total: usize,
|
||||||
|
source_active: usize,
|
||||||
|
source_reconnecting: usize,
|
||||||
|
source_failed: usize,
|
||||||
|
source_reconnect_total: u64,
|
||||||
|
source_replay_attempt_total: u64,
|
||||||
|
source_continuity_gap_total: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestProcessingFrontierProjection {
|
||||||
|
/// Returns the empty run-local processing projection used before the source observes work.
|
||||||
|
pub(crate) const fn empty() -> Self {
|
||||||
|
return Self {
|
||||||
|
hydration_pending: 0,
|
||||||
|
processing_frontier_slot: std::option::Option::None,
|
||||||
|
oldest_pending_slot: std::option::Option::None,
|
||||||
|
source_state: std::option::Option::None,
|
||||||
|
source_total: 0,
|
||||||
|
source_active: 0,
|
||||||
|
source_reconnecting: 0,
|
||||||
|
source_failed: 0,
|
||||||
|
source_reconnect_total: 0,
|
||||||
|
source_replay_attempt_total: 0,
|
||||||
|
source_continuity_gap_total: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one run-local processing projection from bounded source-owned state.
|
||||||
|
pub(crate) const fn new(
|
||||||
|
hydration_pending: usize,
|
||||||
|
processing_frontier_slot: std::option::Option<u64>,
|
||||||
|
oldest_pending_slot: std::option::Option<u64>,
|
||||||
|
) -> Self {
|
||||||
|
return Self {
|
||||||
|
hydration_pending,
|
||||||
|
processing_frontier_slot,
|
||||||
|
oldest_pending_slot,
|
||||||
|
source_state: std::option::Option::None,
|
||||||
|
source_total: 0,
|
||||||
|
source_active: 0,
|
||||||
|
source_reconnecting: 0,
|
||||||
|
source_failed: 0,
|
||||||
|
source_reconnect_total: 0,
|
||||||
|
source_replay_attempt_total: 0,
|
||||||
|
source_continuity_gap_total: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of source signals still pending hydration/admission processing.
|
||||||
|
pub(crate) const fn hydration_pending(&self) -> usize {
|
||||||
|
return self.hydration_pending;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the highest actually observed slot not blocked by older pending source work.
|
||||||
|
pub(crate) const fn processing_frontier_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.processing_frontier_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the oldest actually observed slot that still owns pending source work.
|
||||||
|
pub(crate) const fn oldest_pending_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.oldest_pending_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns a copy carrying the latest source reconnect/replay projection.
|
||||||
|
pub(crate) const fn with_source_continuity(
|
||||||
|
mut self,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
source_reconnect_total: u64,
|
||||||
|
source_replay_attempt_total: u64,
|
||||||
|
source_continuity_gap_total: u64,
|
||||||
|
) -> Self {
|
||||||
|
self.source_state = source_state;
|
||||||
|
self.source_reconnect_total = source_reconnect_total;
|
||||||
|
self.source_replay_attempt_total = source_replay_attempt_total;
|
||||||
|
self.source_continuity_gap_total = source_continuity_gap_total;
|
||||||
|
return self;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns a copy carrying source-neutral multi-source lifecycle counts.
|
||||||
|
pub(crate) const fn with_source_counts(mut self, source_total: usize, source_active: usize, source_reconnecting: usize, source_failed: usize) -> Self {
|
||||||
|
self.source_total = source_total;
|
||||||
|
self.source_active = source_active;
|
||||||
|
self.source_reconnecting = source_reconnecting;
|
||||||
|
self.source_failed = source_failed;
|
||||||
|
return self;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured logical source count carried by this private projection.
|
||||||
|
pub(crate) const fn source_total(&self) -> usize {
|
||||||
|
return self.source_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of sources currently projected Active.
|
||||||
|
pub(crate) const fn source_active(&self) -> usize {
|
||||||
|
return self.source_active;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of sources currently projected Reconnecting.
|
||||||
|
pub(crate) const fn source_reconnecting(&self) -> usize {
|
||||||
|
return self.source_reconnecting;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of sources currently projected Failed.
|
||||||
|
pub(crate) const fn source_failed(&self) -> usize {
|
||||||
|
return self.source_failed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest source-neutral lifecycle state carried by this private projection.
|
||||||
|
pub(crate) const fn source_state(&self) -> std::option::Option<crate::RawTransactionIngestSourceState> {
|
||||||
|
return self.source_state;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns successful reconnects carried by this private projection.
|
||||||
|
pub(crate) const fn source_reconnect_total(&self) -> u64 {
|
||||||
|
return self.source_reconnect_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns replay-bearing reconnect attempts carried by this private projection.
|
||||||
|
pub(crate) const fn source_replay_attempt_total(&self) -> u64 {
|
||||||
|
return self.source_replay_attempt_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns proven replay-retention gaps carried by this private projection.
|
||||||
|
pub(crate) const fn source_continuity_gap_total(&self) -> u64 {
|
||||||
|
return self.source_continuity_gap_total;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete safe latest-value snapshot of one continuous RAW transaction ingest Worker.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct RawTransactionIngestSnapshot {
|
||||||
|
worker: ksp_worker_api::WorkerSnapshot,
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
admitted_total: u64,
|
||||||
|
canonicalized_total: u64,
|
||||||
|
persisted_total: u64,
|
||||||
|
entity_inserted_total: u64,
|
||||||
|
entity_already_present_total: u64,
|
||||||
|
entity_skipped_purged_total: u64,
|
||||||
|
observation_inserted_total: u64,
|
||||||
|
observation_already_present_total: u64,
|
||||||
|
content_conflict_total: u64,
|
||||||
|
store_failure_total: u64,
|
||||||
|
source_failure_total: u64,
|
||||||
|
backpressure_wait_total: u64,
|
||||||
|
hydration_pending: usize,
|
||||||
|
processing_frontier_slot: std::option::Option<u64>,
|
||||||
|
oldest_pending_slot: std::option::Option<u64>,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
source_total: usize,
|
||||||
|
source_active: usize,
|
||||||
|
source_reconnecting: usize,
|
||||||
|
source_failed: usize,
|
||||||
|
source_reconnect_total: u64,
|
||||||
|
source_replay_attempt_total: u64,
|
||||||
|
source_continuity_gap_total: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshot {
|
||||||
|
/// Returns the common Worker projection carried by this concrete snapshot.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn worker_snapshot(&self) -> &ksp_worker_api::WorkerSnapshot {
|
||||||
|
return &self.worker;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured bounded admission queue capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_capacity(&self) -> usize {
|
||||||
|
return self.admission_queue_capacity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest observed number of queued ingress entries awaiting supervisor admission.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admission_queue_depth(&self) -> usize {
|
||||||
|
return self.admission_queue_depth;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured maximum number of concurrent Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persistence_concurrency(&self) -> usize {
|
||||||
|
return self.persistence_concurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest observed number of in-flight Store persistence operations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn in_flight_persistence(&self) -> usize {
|
||||||
|
return self.in_flight_persistence;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of ingress entries removed from the bounded admission queue.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admitted_total(&self) -> u64 {
|
||||||
|
return self.admitted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of admitted ingress entries successfully canonicalized through the Common RAW contract.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn canonicalized_total(&self) -> u64 {
|
||||||
|
return self.canonicalized_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of successful atomic Store persistence outcomes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn persisted_total(&self) -> u64 {
|
||||||
|
return self.persisted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted canonical RAW transaction entities.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_inserted_total(&self) -> u64 {
|
||||||
|
return self.entity_inserted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of identical canonical RAW transaction entities already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_already_present_total(&self) -> u64 {
|
||||||
|
return self.entity_already_present_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable purge tombstones respected by normal persistence.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity_skipped_purged_total(&self) -> u64 {
|
||||||
|
return self.entity_skipped_purged_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted deterministic acquisition observations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_inserted_total(&self) -> u64 {
|
||||||
|
return self.observation_inserted_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of deterministic acquisition observations already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_already_present_total(&self) -> u64 {
|
||||||
|
return self.observation_already_present_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable Store content conflicts observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn content_conflict_total(&self) -> u64 {
|
||||||
|
return self.content_conflict_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of non-conflict Store persistence failures observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn store_failure_total(&self) -> u64 {
|
||||||
|
return self.store_failure_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of source-task failures observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_failure_total(&self) -> u64 {
|
||||||
|
return self.source_failure_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of supervisor dequeues that observed the bounded admission queue at full capacity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn backpressure_wait_total(&self) -> u64 {
|
||||||
|
return self.backpressure_wait_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of source signals pending hydration/admission processing.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn hydration_pending(&self) -> usize {
|
||||||
|
return self.hydration_pending;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the run-local processing frontier over actually observed source work.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn processing_frontier_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.processing_frontier_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the oldest actually observed slot that still owns pending source work.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn oldest_pending_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.oldest_pending_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the configured number of logical live sources for this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_total(&self) -> usize {
|
||||||
|
return self.source_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of sources projected Active.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_active(&self) -> usize {
|
||||||
|
return self.source_active;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of sources projected Reconnecting.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_reconnecting(&self) -> usize {
|
||||||
|
return self.source_reconnecting;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest number of sources projected Failed.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_failed(&self) -> usize {
|
||||||
|
return self.source_failed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest source-neutral lifecycle state when the productive source has started.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_state(&self) -> std::option::Option<crate::RawTransactionIngestSourceState> {
|
||||||
|
return self.source_state;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns successful automatic Yellowstone reconnects observed by this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_reconnect_total(&self) -> u64 {
|
||||||
|
return self.source_reconnect_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns Yellowstone replay-bearing reconnect attempts observed by this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_replay_attempt_total(&self) -> u64 {
|
||||||
|
return self.source_replay_attempt_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns replay retention gaps conservatively proven by Transport during this Worker run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn source_continuity_gap_total(&self) -> u64 {
|
||||||
|
return self.source_continuity_gap_total;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSnapshot {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSnapshot")
|
||||||
|
.field("worker", &self.worker)
|
||||||
|
.field("admission_queue_capacity", &self.admission_queue_capacity)
|
||||||
|
.field("admission_queue_depth", &self.admission_queue_depth)
|
||||||
|
.field("persistence_concurrency", &self.persistence_concurrency)
|
||||||
|
.field("in_flight_persistence", &self.in_flight_persistence)
|
||||||
|
.field("admitted_total", &self.admitted_total)
|
||||||
|
.field("canonicalized_total", &self.canonicalized_total)
|
||||||
|
.field("persisted_total", &self.persisted_total)
|
||||||
|
.field("entity_inserted_total", &self.entity_inserted_total)
|
||||||
|
.field("entity_already_present_total", &self.entity_already_present_total)
|
||||||
|
.field("entity_skipped_purged_total", &self.entity_skipped_purged_total)
|
||||||
|
.field("observation_inserted_total", &self.observation_inserted_total)
|
||||||
|
.field("observation_already_present_total", &self.observation_already_present_total)
|
||||||
|
.field("content_conflict_total", &self.content_conflict_total)
|
||||||
|
.field("store_failure_total", &self.store_failure_total)
|
||||||
|
.field("source_failure_total", &self.source_failure_total)
|
||||||
|
.field("backpressure_wait_total", &self.backpressure_wait_total)
|
||||||
|
.field("hydration_pending", &self.hydration_pending)
|
||||||
|
.field("processing_frontier_slot", &self.processing_frontier_slot)
|
||||||
|
.field("oldest_pending_slot", &self.oldest_pending_slot)
|
||||||
|
.field("source_state", &self.source_state)
|
||||||
|
.field("source_total", &self.source_total)
|
||||||
|
.field("source_active", &self.source_active)
|
||||||
|
.field("source_reconnecting", &self.source_reconnecting)
|
||||||
|
.field("source_failed", &self.source_failed)
|
||||||
|
.field("source_reconnect_total", &self.source_reconnect_total)
|
||||||
|
.field("source_replay_attempt_total", &self.source_replay_attempt_total)
|
||||||
|
.field("source_continuity_gap_total", &self.source_continuity_gap_total)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cloneable latest-value source exposing concrete and common Worker snapshots from one shared watch state.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct RawTransactionIngestSnapshotSource {
|
||||||
|
receiver: tokio::sync::watch::Receiver<crate::RawTransactionIngestSnapshot>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshotSource {
|
||||||
|
/// Returns the complete current concrete snapshot without replaying intermediate updates.
|
||||||
|
#[must_use]
|
||||||
|
pub fn current(&self) -> crate::RawTransactionIngestSnapshot {
|
||||||
|
return self.receiver.borrow().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reports whether the private publisher has closed after the runtime task returned.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) fn is_closed(&self) -> bool {
|
||||||
|
return self.receiver.has_changed().is_err();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Waits for one concrete snapshot newer than `observed`, coalescing intermediate updates to the latest value.
|
||||||
|
#[must_use]
|
||||||
|
pub fn wait_for_change(&self, observed: ksp_worker_api::WorkerSnapshotSequence) -> crate::RawTransactionIngestSnapshotFuture<'_> {
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = receiver.borrow_and_update().clone();
|
||||||
|
if current.worker_snapshot().sequence().is_after(observed) {
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return receiver.borrow_and_update().clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ksp_worker_api::WorkerSnapshotSource for crate::RawTransactionIngestSnapshotSource {
|
||||||
|
fn current(&self) -> ksp_worker_api::WorkerSnapshot {
|
||||||
|
return self.receiver.borrow().worker_snapshot().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wait_for_change(&self, observed: ksp_worker_api::WorkerSnapshotSequence) -> ksp_worker_api::WorkerSnapshotFuture<'_> {
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = receiver.borrow_and_update().worker_snapshot().clone();
|
||||||
|
if current.sequence().is_after(observed) {
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return receiver.borrow_and_update().worker_snapshot().clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::RawTransactionIngestSnapshotSource {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let current = self.receiver.borrow();
|
||||||
|
return formatter
|
||||||
|
.debug_struct("RawTransactionIngestSnapshotSource")
|
||||||
|
.field("sequence", ¤t.worker_snapshot().sequence())
|
||||||
|
.field("state", ¤t.worker_snapshot().state())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private latest-value publisher and checked counter owner shared by the Worker supervisor.
|
||||||
|
pub(crate) struct RawTransactionIngestSnapshotPublisher {
|
||||||
|
sender: tokio::sync::watch::Sender<crate::RawTransactionIngestSnapshot>,
|
||||||
|
snapshot: crate::RawTransactionIngestSnapshot,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestSnapshotPublisher {
|
||||||
|
/// Creates the initial `Starting` snapshot stream for one validated Worker run.
|
||||||
|
pub(crate) fn new(
|
||||||
|
settings: &crate::RawTransactionIngestSettings,
|
||||||
|
lifecycle: &ksp_worker_api::WorkerLifecycle,
|
||||||
|
source_total: usize,
|
||||||
|
) -> (crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource) {
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
lifecycle.id().clone(),
|
||||||
|
lifecycle.kind().clone(),
|
||||||
|
ksp_worker_api::WorkerSnapshotSequence::initial(),
|
||||||
|
lifecycle.state(),
|
||||||
|
ksp_worker_api::WorkerHealth::Unknown,
|
||||||
|
ksp_worker_api::WorkerActivity::Unknown,
|
||||||
|
);
|
||||||
|
let snapshot = crate::RawTransactionIngestSnapshot {
|
||||||
|
worker,
|
||||||
|
admission_queue_capacity: settings.admission_queue_capacity(),
|
||||||
|
admission_queue_depth: 0,
|
||||||
|
persistence_concurrency: settings.persistence_concurrency(),
|
||||||
|
in_flight_persistence: 0,
|
||||||
|
admitted_total: 0,
|
||||||
|
canonicalized_total: 0,
|
||||||
|
persisted_total: 0,
|
||||||
|
entity_inserted_total: 0,
|
||||||
|
entity_already_present_total: 0,
|
||||||
|
entity_skipped_purged_total: 0,
|
||||||
|
observation_inserted_total: 0,
|
||||||
|
observation_already_present_total: 0,
|
||||||
|
content_conflict_total: 0,
|
||||||
|
store_failure_total: 0,
|
||||||
|
source_failure_total: 0,
|
||||||
|
backpressure_wait_total: 0,
|
||||||
|
hydration_pending: 0,
|
||||||
|
processing_frontier_slot: std::option::Option::None,
|
||||||
|
oldest_pending_slot: std::option::Option::None,
|
||||||
|
source_state: std::option::Option::None,
|
||||||
|
source_total,
|
||||||
|
source_active: 0,
|
||||||
|
source_reconnecting: 0,
|
||||||
|
source_failed: 0,
|
||||||
|
source_reconnect_total: 0,
|
||||||
|
source_replay_attempt_total: 0,
|
||||||
|
source_continuity_gap_total: 0,
|
||||||
|
};
|
||||||
|
let (sender, receiver) = tokio::sync::watch::channel(snapshot.clone());
|
||||||
|
return (Self { sender, snapshot }, crate::RawTransactionIngestSnapshotSource { receiver });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Forces one terminal latest value without advancing the sequence, used only when sequence publication itself is exhausted or invalid.
|
||||||
|
pub(crate) fn force_terminal(&mut self, state: ksp_worker_api::WorkerState) {
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
self.snapshot.worker.id().clone(),
|
||||||
|
self.snapshot.worker.kind().clone(),
|
||||||
|
self.snapshot.worker.sequence(),
|
||||||
|
state,
|
||||||
|
health_for_state(
|
||||||
|
state,
|
||||||
|
self.snapshot.worker.health(),
|
||||||
|
self.snapshot.source_total,
|
||||||
|
self.snapshot.source_active,
|
||||||
|
self.snapshot.source_reconnecting,
|
||||||
|
self.snapshot.source_failed,
|
||||||
|
),
|
||||||
|
ksp_worker_api::WorkerActivity::Idle,
|
||||||
|
);
|
||||||
|
self.snapshot.worker = worker;
|
||||||
|
self.snapshot.admission_queue_depth = 0;
|
||||||
|
self.snapshot.in_flight_persistence = 0;
|
||||||
|
self.sender.send_replace(self.snapshot.clone());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes one lifecycle/depth refresh with a strictly newer common sequence.
|
||||||
|
pub(crate) fn publish_state(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one dequeued ingress that failed canonicalization and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_admission_failure(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let admitted_total = match checked_counter(self.snapshot.admitted_total, "admitted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let backpressure_wait_total =
|
||||||
|
match checked_optional_counter(self.snapshot.backpressure_wait_total, backpressure_wait_observed, "backpressure_wait_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.admitted_total = admitted_total;
|
||||||
|
self.snapshot.backpressure_wait_total = backpressure_wait_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one successfully canonicalized admitted ingress and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_admission_success(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
backpressure_wait_observed: bool,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let admitted_total = match checked_counter(self.snapshot.admitted_total, "admitted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let backpressure_wait_total =
|
||||||
|
match checked_optional_counter(self.snapshot.backpressure_wait_total, backpressure_wait_observed, "backpressure_wait_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let canonicalized_total = match checked_counter(self.snapshot.canonicalized_total, "canonicalized_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.admitted_total = admitted_total;
|
||||||
|
self.snapshot.backpressure_wait_total = backpressure_wait_total;
|
||||||
|
self.snapshot.canonicalized_total = canonicalized_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes one latest run-local processing-frontier projection emitted by the source task.
|
||||||
|
pub(crate) fn record_processing_frontier(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
projection: crate::RawTransactionIngestProcessingFrontierProjection,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
self.snapshot.hydration_pending = projection.hydration_pending();
|
||||||
|
self.snapshot.processing_frontier_slot = projection.processing_frontier_slot();
|
||||||
|
self.snapshot.oldest_pending_slot = projection.oldest_pending_slot();
|
||||||
|
self.snapshot.source_state = projection.source_state();
|
||||||
|
if projection.source_total() > 0 {
|
||||||
|
self.snapshot.source_total = projection.source_total();
|
||||||
|
self.snapshot.source_active = projection.source_active();
|
||||||
|
self.snapshot.source_reconnecting = projection.source_reconnecting();
|
||||||
|
self.snapshot.source_failed = projection.source_failed();
|
||||||
|
}
|
||||||
|
self.snapshot.source_reconnect_total = projection.source_reconnect_total();
|
||||||
|
self.snapshot.source_replay_attempt_total = projection.source_replay_attempt_total();
|
||||||
|
self.snapshot.source_continuity_gap_total = projection.source_continuity_gap_total();
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one failed private source task without retaining provider-specific error material.
|
||||||
|
pub(crate) fn record_source_failure(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let source_failure_total = match checked_counter(self.snapshot.source_failure_total, "source_failure_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.source_failure_total = source_failure_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one successful Store persistence outcome and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_persistence_success(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
outcome: crate::RawTransactionIngestPersistenceOutcome,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
let persisted_total = match checked_counter(self.snapshot.persisted_total, "persisted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut entity_inserted_total = self.snapshot.entity_inserted_total;
|
||||||
|
let mut entity_already_present_total = self.snapshot.entity_already_present_total;
|
||||||
|
let mut entity_skipped_purged_total = self.snapshot.entity_skipped_purged_total;
|
||||||
|
let mut observation_inserted_total = self.snapshot.observation_inserted_total;
|
||||||
|
let mut observation_already_present_total = self.snapshot.observation_already_present_total;
|
||||||
|
match outcome.entity() {
|
||||||
|
crate::RawTransactionIngestEntityPersistence::Inserted => {
|
||||||
|
entity_inserted_total = match checked_counter(entity_inserted_total, "entity_inserted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent => {
|
||||||
|
entity_already_present_total = match checked_counter(entity_already_present_total, "entity_already_present_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestEntityPersistence::SkippedPurged => {
|
||||||
|
entity_skipped_purged_total = match checked_counter(entity_skipped_purged_total, "entity_skipped_purged_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
}
|
||||||
|
match outcome.observation() {
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted => {
|
||||||
|
observation_inserted_total = match checked_counter(observation_inserted_total, "observation_inserted_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestObservationPersistence::AlreadyPresent => {
|
||||||
|
observation_already_present_total = match checked_counter(observation_already_present_total, "observation_already_present_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::RawTransactionIngestObservationPersistence::NotRecorded => {},
|
||||||
|
}
|
||||||
|
self.snapshot.persisted_total = persisted_total;
|
||||||
|
self.snapshot.entity_inserted_total = entity_inserted_total;
|
||||||
|
self.snapshot.entity_already_present_total = entity_already_present_total;
|
||||||
|
self.snapshot.entity_skipped_purged_total = entity_skipped_purged_total;
|
||||||
|
self.snapshot.observation_inserted_total = observation_inserted_total;
|
||||||
|
self.snapshot.observation_already_present_total = observation_already_present_total;
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records one classified persistence fault counter and publishes the resulting latest value.
|
||||||
|
pub(crate) fn record_persistence_fault(
|
||||||
|
&mut self,
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
code: ksp_core_lib::ErrorCode,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT {
|
||||||
|
let next = match checked_counter(self.snapshot.content_conflict_total, "content_conflict_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.content_conflict_total = next;
|
||||||
|
} else if code == crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED {
|
||||||
|
let next = match checked_counter(self.snapshot.store_failure_total, "store_failure_total") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
self.snapshot.store_failure_total = next;
|
||||||
|
}
|
||||||
|
return self.publish(state, admission_queue_depth, in_flight_persistence);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish(&mut self, state: ksp_worker_api::WorkerState, admission_queue_depth: usize, in_flight_persistence: usize) -> ksp_core_lib::Result<()> {
|
||||||
|
let sequence = match self.snapshot.worker.sequence().next() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(crate::counter_exhausted_error("snapshot_sequence")),
|
||||||
|
};
|
||||||
|
let worker = ksp_worker_api::WorkerSnapshot::new(
|
||||||
|
self.snapshot.worker.id().clone(),
|
||||||
|
self.snapshot.worker.kind().clone(),
|
||||||
|
sequence,
|
||||||
|
state,
|
||||||
|
health_for_state(
|
||||||
|
state,
|
||||||
|
self.snapshot.worker.health(),
|
||||||
|
self.snapshot.source_total,
|
||||||
|
self.snapshot.source_active,
|
||||||
|
self.snapshot.source_reconnecting,
|
||||||
|
self.snapshot.source_failed,
|
||||||
|
),
|
||||||
|
activity_for_state(state, admission_queue_depth, in_flight_persistence, self.snapshot.hydration_pending, self.snapshot.source_state),
|
||||||
|
);
|
||||||
|
self.snapshot.worker = worker;
|
||||||
|
self.snapshot.admission_queue_depth = admission_queue_depth;
|
||||||
|
self.snapshot.in_flight_persistence = in_flight_persistence;
|
||||||
|
self.sender.send_replace(self.snapshot.clone());
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn activity_for_state(
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
admission_queue_depth: usize,
|
||||||
|
in_flight_persistence: usize,
|
||||||
|
hydration_pending: usize,
|
||||||
|
source_state: std::option::Option<crate::RawTransactionIngestSourceState>,
|
||||||
|
) -> ksp_worker_api::WorkerActivity {
|
||||||
|
if admission_queue_depth > 0
|
||||||
|
|| in_flight_persistence > 0
|
||||||
|
|| hydration_pending > 0
|
||||||
|
|| matches!(
|
||||||
|
source_state,
|
||||||
|
std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting)
|
||||||
|
| std::option::Option::Some(crate::RawTransactionIngestSourceState::Closing)
|
||||||
|
)
|
||||||
|
{
|
||||||
|
return ksp_worker_api::WorkerActivity::Active;
|
||||||
|
}
|
||||||
|
return match state {
|
||||||
|
ksp_worker_api::WorkerState::Running
|
||||||
|
| ksp_worker_api::WorkerState::Stopping
|
||||||
|
| ksp_worker_api::WorkerState::Stopped
|
||||||
|
| ksp_worker_api::WorkerState::Faulted(_) => ksp_worker_api::WorkerActivity::Idle,
|
||||||
|
_ => ksp_worker_api::WorkerActivity::Unknown,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_counter(current: u64, field: &'static str) -> ksp_core_lib::Result<u64> {
|
||||||
|
return match current.checked_add(1) {
|
||||||
|
std::option::Option::Some(value) => std::result::Result::Ok(value),
|
||||||
|
std::option::Option::None => std::result::Result::Err(crate::counter_exhausted_error(field)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_optional_counter(current: u64, increment: bool, field: &'static str) -> ksp_core_lib::Result<u64> {
|
||||||
|
if !increment {
|
||||||
|
return std::result::Result::Ok(current);
|
||||||
|
}
|
||||||
|
return checked_counter(current, field);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn health_for_state(
|
||||||
|
state: ksp_worker_api::WorkerState,
|
||||||
|
previous: ksp_worker_api::WorkerHealth,
|
||||||
|
source_total: usize,
|
||||||
|
source_active: usize,
|
||||||
|
source_reconnecting: usize,
|
||||||
|
source_failed: usize,
|
||||||
|
) -> ksp_worker_api::WorkerHealth {
|
||||||
|
return match state {
|
||||||
|
ksp_worker_api::WorkerState::Running if source_failed > 0 => ksp_worker_api::WorkerHealth::Unhealthy,
|
||||||
|
ksp_worker_api::WorkerState::Running if source_reconnecting > 0 => ksp_worker_api::WorkerHealth::Degraded,
|
||||||
|
ksp_worker_api::WorkerState::Running if source_total > 0 && source_active < source_total => ksp_worker_api::WorkerHealth::Degraded,
|
||||||
|
ksp_worker_api::WorkerState::Running => ksp_worker_api::WorkerHealth::Healthy,
|
||||||
|
ksp_worker_api::WorkerState::Stopping | ksp_worker_api::WorkerState::Stopped => previous,
|
||||||
|
ksp_worker_api::WorkerState::Faulted(_) => ksp_worker_api::WorkerHealth::Unhealthy,
|
||||||
|
_ => ksp_worker_api::WorkerHealth::Unknown,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/snapshot.rs"]
|
||||||
|
mod tests;
|
||||||
@@ -0,0 +1,365 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/cross_layer_completeness.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! Cross-layer completeness and security canaries through `0.3.13-pre.012`.
|
||||||
|
|
||||||
|
fn dependency_names(manifest: &str) -> std::vec::Vec<&str> {
|
||||||
|
let mut section = "";
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for line in manifest.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.starts_with('[') && trimmed.ends_with(']') {
|
||||||
|
section = trimmed;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if section != "[dependencies]" || trimmed.is_empty() || trimmed.starts_with('#') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match trimmed.split_once('=') {
|
||||||
|
std::option::Option::Some((name, _)) => name.trim().trim_end_matches(".workspace"),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
return names;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_dependency_firewall_is_exact_across_transport_worker_common_raw_store() {
|
||||||
|
let transport = dependency_names(include_str!("../../ksp-onchain-transport-lib/Cargo.toml"));
|
||||||
|
let worker = dependency_names(include_str!("../Cargo.toml"));
|
||||||
|
let common_raw = dependency_names(include_str!("../../ksp-raw-transaction-lib/Cargo.toml"));
|
||||||
|
let store_api = dependency_names(include_str!("../../ksp-store-api/Cargo.toml"));
|
||||||
|
let store = dependency_names(include_str!("../../ksp-store-lib/Cargo.toml"));
|
||||||
|
assert_eq!(
|
||||||
|
transport,
|
||||||
|
std::vec![
|
||||||
|
"futures-util",
|
||||||
|
"http",
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"reqwest",
|
||||||
|
"serde",
|
||||||
|
"serde_json",
|
||||||
|
"tokio",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"tonic-prost",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
worker,
|
||||||
|
std::vec![
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert_eq!(common_raw, std::vec!["base64", "ksp-core-lib", "ksp-store-api", "serde_json", "sha2"]);
|
||||||
|
assert_eq!(store_api, std::vec!["ksp-core-lib"]);
|
||||||
|
assert_eq!(store, std::vec!["ksp-logging-lib", "ksp-store-api", "ksp-store-postgres-lib"]);
|
||||||
|
for forbidden in ["ksp-config-lib", "ksp-job-api", "ksp-job-backfill-lib", "ksp-store-lib", "ksp-worker-api", "ksp-worker-raw-transaction-ingest-lib"] {
|
||||||
|
assert!(!transport.contains(&forbidden), "Transport crossed upward dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"reqwest",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!worker.contains(&forbidden), "Worker crossed direct lower/backend dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
] {
|
||||||
|
assert!(!common_raw.contains(&forbidden), "Common RAW crossed dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp-config-lib", "ksp-onchain-transport-lib", "ksp-raw-transaction-lib", "ksp-store-lib", "ksp-store-postgres-lib", "ksp-worker-api"] {
|
||||||
|
assert!(!store_api.contains(&forbidden), "Store API crossed dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp-config-lib", "ksp-job-api", "ksp-job-backfill-lib", "ksp-onchain-transport-lib", "ksp-raw-transaction-lib", "ksp-worker-api"] {
|
||||||
|
assert!(!store.contains(&forbidden), "Store facade crossed dependency boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_legacy_v0_fixture_matrix_is_exact_across_transport_worker_and_common_raw() {
|
||||||
|
let worker = include_str!("../unit_tests/runtime_resources.rs");
|
||||||
|
let common_raw_wire = include_str!("../../ksp-raw-transaction-lib/unit_tests/wire.rs");
|
||||||
|
let transport = include_str!("../../ksp-onchain-transport-lib/unit_tests/rpc_transactions.rs");
|
||||||
|
for required in [
|
||||||
|
"pre_004_observed_get_transaction_closes_signal_to_common_raw_ingress_with_composite_provenance",
|
||||||
|
"pre_004_finalized_hydration_preserves_request_and_provenance_commitment",
|
||||||
|
"pre_004_v0_null_and_omitted_wire_fields_preserve_common_raw_semantics",
|
||||||
|
r#"\"version\":\"legacy\""#,
|
||||||
|
r#"\"version\":0"#,
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "Worker Legacy/V0 fixture missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"LEGACY_GOLDEN_BASE64",
|
||||||
|
"V0_GOLDEN_BASE64",
|
||||||
|
"pre_006_legacy_v0_and_v1_wire_goldens_are_exact",
|
||||||
|
"RawSolanaMessageVersion::Legacy",
|
||||||
|
"RawSolanaMessageVersion::V0",
|
||||||
|
] {
|
||||||
|
assert!(common_raw_wire.contains(required), "Common RAW Legacy/V0 golden missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"confirmed_transaction_fixture_preserves_meta_version_and_transaction_index_states",
|
||||||
|
"typed_get_transaction_modern_covers_all_agave_v4_2_1_encoding_labels_and_response_shapes",
|
||||||
|
"SolanaTransactionVersion::Legacy",
|
||||||
|
] {
|
||||||
|
assert!(transport.contains(required), "Transport getTransaction fixture missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_redaction_scanner_matrix_is_present_across_all_layers() {
|
||||||
|
let transport_subscribe = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_unary = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_unary.rs");
|
||||||
|
let worker = include_str!("hardening.rs");
|
||||||
|
let common_raw = include_str!("../../ksp-raw-transaction-lib/unit_tests/canonical.rs");
|
||||||
|
let store_api = include_str!("../../ksp-store-api/tests/security_hardening.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/tests/hardening_completeness.rs");
|
||||||
|
assert!(transport_subscribe.contains("yellowstone_subscribe_debug_omits_filter_names_and_future_payloads"));
|
||||||
|
assert!(transport_unary.contains("yellowstone_remote_status_does_not_copy_message_details_or_metadata"));
|
||||||
|
for required in [
|
||||||
|
"pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values",
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted",
|
||||||
|
"v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material",
|
||||||
|
"v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "Worker security scanner missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(common_raw.contains("pre_002_material_and_wire_debug_do_not_render_transaction_or_meta_material"));
|
||||||
|
assert!(store_api.contains("pre_007_raw_debug_surfaces_do_not_render_payload_hash_signature_or_account_bytes"));
|
||||||
|
assert!(store.contains("pre_009_secret_canary_never_crosses_settings_or_pre_io_error_debug"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_public_roots_keep_implementation_modules_private() {
|
||||||
|
let roots = [
|
||||||
|
("transport", include_str!("../../ksp-onchain-transport-lib/src/lib.rs")),
|
||||||
|
("worker", include_str!("../src/lib.rs")),
|
||||||
|
("common_raw", include_str!("../../ksp-raw-transaction-lib/src/lib.rs")),
|
||||||
|
("store_api", include_str!("../../ksp-store-api/src/lib.rs")),
|
||||||
|
("store", include_str!("../../ksp-store-lib/src/lib.rs")),
|
||||||
|
];
|
||||||
|
for (layer, root) in roots {
|
||||||
|
assert!(!root.contains("pub mod "), "public implementation module leaked from {layer}");
|
||||||
|
}
|
||||||
|
let worker = include_str!("../src/lib.rs");
|
||||||
|
for line in worker.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if !trimmed.starts_with("pub use ") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"RawTransactionAdmission",
|
||||||
|
"RawTransactionIngress",
|
||||||
|
"RawTransactionIngestPersistencePort",
|
||||||
|
"JoinHandle",
|
||||||
|
"JoinSet",
|
||||||
|
"HttpTransportPool",
|
||||||
|
"SolanaYellowstoneGrpcSubscribeSession",
|
||||||
|
] {
|
||||||
|
assert!(!trimmed.contains(forbidden), "Worker implementation type leaked publicly: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline() {
|
||||||
|
let transport_root = include_str!("../../ksp-onchain-transport-lib/src/lib.rs");
|
||||||
|
let transport_http_transactions = include_str!("../../ksp-onchain-transport-lib/src/rpc_transactions.rs");
|
||||||
|
let transport_http_blocks = include_str!("../../ksp-onchain-transport-lib/src/rpc_blocks.rs");
|
||||||
|
let transport_ws_protocol = include_str!("../../ksp-onchain-transport-lib/src/ws_protocol_session.rs");
|
||||||
|
let transport_grpc_stream = include_str!("../../ksp-onchain-transport-lib/src/grpc_stream.rs");
|
||||||
|
let worker_resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let worker_persistence = include_str!("../src/persistence.rs");
|
||||||
|
let common_raw_root = include_str!("../../ksp-raw-transaction-lib/src/lib.rs");
|
||||||
|
let store_root = include_str!("../../ksp-store-lib/src/lib.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/src/store.rs");
|
||||||
|
for required in [
|
||||||
|
"HeliusTransaction(crate::RawTransactionIngestHeliusTransactionSource)",
|
||||||
|
"HttpBlockPolling(crate::RawTransactionIngestHttpBlockPollingSource)",
|
||||||
|
"StandardBlock(crate::RawTransactionIngestStandardBlockSource)",
|
||||||
|
"StandardLogs(crate::RawTransactionIngestStandardLogsSource)",
|
||||||
|
"Yellowstone(crate::RawTransactionIngestYellowstoneSource)",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"run_live_sources",
|
||||||
|
"RawTransactionIngress",
|
||||||
|
] {
|
||||||
|
assert!(worker_resources.contains(required), "pre.012 Worker source/pipeline contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["persist_raw_transaction_ingest_converged_acquisition", "RawTransactionIngestPersistenceConvergence", "record_observation"] {
|
||||||
|
assert!(worker_persistence.contains(required), "pre.012 Worker persistence contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["SolanaStandardWsSession", "HeliusLaserStreamWsSession"] {
|
||||||
|
assert!(transport_ws_protocol.contains(required), "pre.012 WS Transport facade missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(transport_grpc_stream.contains("pub struct SolanaYellowstoneGrpcSubscribeSession"));
|
||||||
|
assert!(transport_http_transactions.contains("pub async fn get_transaction_observed("));
|
||||||
|
assert!(transport_http_blocks.contains("pub async fn get_block_observed("));
|
||||||
|
assert!(transport_http_blocks.contains("pub async fn get_blocks_with_limit("));
|
||||||
|
for required in ["RawTransactionAcquisition", "RawSolanaMessageVersion", "RawSolanaTransactionWire"] {
|
||||||
|
assert!(common_raw_root.contains(required), "pre.012 Common RAW contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["RawTransactionWrite", "RawTransactionObservationWrite", "RawPayload"] {
|
||||||
|
assert!(store_root.contains(required), "pre.012 Store facade contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["persist_raw_transaction_acquisition", "record_raw_transaction_observation"] {
|
||||||
|
assert!(store.contains(required), "pre.012 Store dispatch contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(!transport_root.contains("ksp_raw_transaction_lib"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store() {
|
||||||
|
let transport_grpc = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_blocks = include_str!("../../ksp-onchain-transport-lib/unit_tests/rpc_blocks.rs");
|
||||||
|
let worker = include_str!("../unit_tests/runtime_resources.rs");
|
||||||
|
let worker_resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let worker_persistence = include_str!("../src/persistence.rs");
|
||||||
|
let common_raw_wire = include_str!("../../ksp-raw-transaction-lib/unit_tests/wire.rs");
|
||||||
|
let common_raw_canonical = include_str!("../../ksp-raw-transaction-lib/unit_tests/canonical.rs");
|
||||||
|
let store_api = include_str!("../../ksp-store-api/tests/public_api.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/src/store.rs");
|
||||||
|
assert!(transport_grpc.contains("yellowstone_transaction_update_decodes_current_storage_wire_including_v1_config_and_meta"));
|
||||||
|
assert!(transport_blocks.contains("typed_get_block_modern_full_config_preserves_rich_wire_and_simd_fields"));
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_004_standard_block_qualifies_legacy_v0_v1_and_rejects_ambiguous_or_future_versions",
|
||||||
|
"v0_3_13_pre_004_legacy_v0_v1_block_materials_are_direct_common_raw_with_exact_version_and_index",
|
||||||
|
"v0_3_13_pre_006_http_block_polling_qualifies_legacy_v0_v1_and_rejects_ambiguous_or_future_versions",
|
||||||
|
"RawSolanaMessageVersion::Legacy",
|
||||||
|
"RawSolanaMessageVersion::V0",
|
||||||
|
"RawSolanaMessageVersion::V1",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "pre.012 Worker Legacy/V0/V1 proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(worker_resources.contains("SolanaGetTransactionConfig::new"));
|
||||||
|
assert!(worker_resources.contains("std::option::Option::Some(1),"));
|
||||||
|
for required in [
|
||||||
|
"pre_006_legacy_v0_and_v1_wire_goldens_are_exact",
|
||||||
|
"pre_006_v1_structural_guards_reject_alt_duplicates_and_missing_config",
|
||||||
|
"RawSolanaMessageVersion::V1",
|
||||||
|
] {
|
||||||
|
assert!(common_raw_wire.contains(required), "pre.012 Common RAW V1 proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(common_raw_canonical.contains("pre_002_canonical_raw_v1_golden_bytes_and_hash_are_exact"));
|
||||||
|
for required in ["block_time", "content_hash", "format_id", "format_version", "slot"] {
|
||||||
|
assert!(worker_persistence.contains(required), "pre.012 canonical Store identity field missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(store_api.contains("public_pre_003_raw_transaction_and_observation_are_constructible_from_crate_root"));
|
||||||
|
assert!(store_api.contains("RawPayload::try_new(format, 1"));
|
||||||
|
assert!(store.contains("persist_raw_transaction_acquisition"));
|
||||||
|
assert!(store.contains("record_raw_transaction_observation"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration() {
|
||||||
|
let transport_subscribe = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_stream = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_stream.rs");
|
||||||
|
let transport_public = include_str!("../../ksp-onchain-transport-lib/tests/public_api.rs");
|
||||||
|
let worker = include_str!("../unit_tests/runtime_resources.rs");
|
||||||
|
let worker_hardening = include_str!("hardening.rs");
|
||||||
|
for required in [
|
||||||
|
"yellowstone_subscribe_request_identity_is_order_stable_exact_and_debug_redacted",
|
||||||
|
"yellowstone_transaction_update_decodes_current_storage_wire_including_v1_config_and_meta",
|
||||||
|
"yellowstone_transaction_status_update_preserves_error_and_rejects_malformed_signature",
|
||||||
|
"yellowstone_block_update_reuses_transaction_account_entry_dtos_and_preserves_server_counts",
|
||||||
|
] {
|
||||||
|
assert!(transport_subscribe.contains(required), "pre.012 Yellowstone Transport subscribe proof missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"yellowstone_reconnect_replays_from_last_observed_slot_and_counts_duplicate_identity",
|
||||||
|
"yellowstone_replay_info_proves_and_clamps_retention_gap_without_lossless_claim",
|
||||||
|
"yellowstone_shutdown_interrupts_reconnect_backoff_and_mutation_is_rejected_during_reconnect",
|
||||||
|
] {
|
||||||
|
assert!(transport_stream.contains(required), "pre.012 Yellowstone Transport lifecycle proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(transport_public.contains("public_v0_3_13_pre_002_yellowstone_subscribe_identity_is_opaque_and_available_from_crate_root"));
|
||||||
|
for required in [
|
||||||
|
"pre_002_source_accepts_matching_confirmed_transaction_and_get_transaction_route_without_io",
|
||||||
|
"v0_3_13_pre_002_live_source_key_is_stable_and_request_sensitive",
|
||||||
|
"pre_003_transaction_signal_fixture_preserves_exact_source_neutral_identity",
|
||||||
|
"pre_005_block_projects_transaction_signals_in_exact_source_order",
|
||||||
|
"pre_005_block_meta_and_slot_project_continuity_only_without_remote_dead_error",
|
||||||
|
"pre_004_observed_get_transaction_closes_signal_to_common_raw_ingress_with_composite_provenance",
|
||||||
|
"pre_008_reconnect_replay_and_proven_gap_are_distinct_monotone_and_frontier_preserving",
|
||||||
|
"v0_3_13_pre_008_global_hydration_registry_coalesces_cross_source_key_to_one_leader",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "pre.012 Yellowstone Worker non-regression proof missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material",
|
||||||
|
] {
|
||||||
|
assert!(worker_hardening.contains(required), "pre.012 Yellowstone Worker hardening proof missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers() {
|
||||||
|
let transport_subscribe = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_subscribe.rs");
|
||||||
|
let transport_unary = include_str!("../../ksp-onchain-transport-lib/unit_tests/grpc_unary.rs");
|
||||||
|
let transport_ws = include_str!("../../ksp-onchain-transport-lib/unit_tests/ws_helius_transactions.rs");
|
||||||
|
let worker = include_str!("hardening.rs");
|
||||||
|
let common_raw = include_str!("../../ksp-raw-transaction-lib/unit_tests/canonical.rs");
|
||||||
|
let store_api = include_str!("../../ksp-store-api/tests/security_hardening.rs");
|
||||||
|
let store = include_str!("../../ksp-store-lib/tests/hardening_completeness.rs");
|
||||||
|
let postgres = include_str!("../../ksp-store-postgres-lib/tests/hardening_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"yellowstone_subscribe_debug_omits_filter_names_and_future_payloads",
|
||||||
|
"yellowstone_transaction_filters_encode_complete_current_wire_and_redact_selectors",
|
||||||
|
] {
|
||||||
|
assert!(transport_subscribe.contains(required), "pre.012 Transport gRPC redaction proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(transport_unary.contains("yellowstone_remote_status_does_not_copy_message_details_or_metadata"));
|
||||||
|
assert!(transport_ws.contains("helius_transaction_notification_debug_omits_raw_provider_payloads"));
|
||||||
|
for required in [
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit",
|
||||||
|
"v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit",
|
||||||
|
"v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted",
|
||||||
|
"v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity",
|
||||||
|
"v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private",
|
||||||
|
"v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted",
|
||||||
|
"v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe",
|
||||||
|
] {
|
||||||
|
assert!(worker.contains(required), "pre.012 Worker security proof missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(common_raw.contains("pre_002_material_and_wire_debug_do_not_render_transaction_or_meta_material"));
|
||||||
|
assert!(store_api.contains("pre_007_raw_debug_surfaces_do_not_render_payload_hash_signature_or_account_bytes"));
|
||||||
|
assert!(store.contains("pre_009_secret_canary_never_crosses_settings_or_pre_io_error_debug"));
|
||||||
|
assert!(postgres.contains("pre_009_backend_error_bridge_cannot_retain_external_error_or_secret_text"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,734 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
// version: 29
|
||||||
|
|
||||||
|
//! Dependency firewall canaries for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_manifest_dependency_surface_is_exact() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
let names = manifest_dependency_names(dependencies);
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
vec![
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
for required in [
|
||||||
|
"ksp-core-lib = { path = \"../ksp-core-lib\" }",
|
||||||
|
"ksp-logging-lib = { path = \"../ksp-logging-lib\" }",
|
||||||
|
"ksp-onchain-transport-lib = { path = \"../ksp-onchain-transport-lib\" }",
|
||||||
|
"ksp-raw-transaction-lib = { path = \"../ksp-raw-transaction-lib\" }",
|
||||||
|
"ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }",
|
||||||
|
"ksp-worker-api = { path = \"../ksp-worker-api\" }",
|
||||||
|
"sha2 = { workspace = true }",
|
||||||
|
"tokio = { workspace = true, features = [\"macros\", \"rt\", \"sync\", \"time\"] }",
|
||||||
|
] {
|
||||||
|
assert!(dependencies.contains(required), "required Worker dependency missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_manifest_opens_only_the_onchain_transport_live_source_edge() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
for forbidden in [
|
||||||
|
"futures-util",
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-interface-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-program-api",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"ksp-wallet-lib",
|
||||||
|
"reqwest",
|
||||||
|
"solana-",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!dependencies.contains(forbidden), "forbidden Worker dependency present: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!manifest.contains("[dev-dependencies]"), "foundation requires no test-only dependency");
|
||||||
|
assert!(!manifest.contains("[build-dependencies]"), "foundation requires no build dependency");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_003_transaction_and_status_adapters_are_private_and_transport_facade_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestYellowstoneSignalView for ksp_onchain_transport_lib::YellowstoneTransactionUpdate",
|
||||||
|
"RawTransactionIngestYellowstoneSignalView for ksp_onchain_transport_lib::YellowstoneTransactionStatusUpdate",
|
||||||
|
"YellowstoneTransactionUpdate::transaction(self).signature()",
|
||||||
|
"YellowstoneTransactionUpdate::transaction(self).index()",
|
||||||
|
"YellowstoneTransactionStatusUpdate::signature(self)",
|
||||||
|
"YellowstoneTransactionStatusUpdate::index(self)",
|
||||||
|
"RawTransactionSignature::new",
|
||||||
|
"matched_filter_fingerprint",
|
||||||
|
"yellowstone_provider_unrepresentable",
|
||||||
|
"yellowstone_endpoint_unrepresentable",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 signal-adapter contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestSourceSignal",
|
||||||
|
"pub(crate) struct RawTransactionIngestSourceSignal",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceSignal",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(!resources.contains(forbidden) && !root.contains(forbidden), "pre.003 crossed a private/offline boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
let status_impl =
|
||||||
|
match resources.split_once("impl RawTransactionIngestYellowstoneSignalView for ksp_onchain_transport_lib::YellowstoneTransactionStatusUpdate") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::convert::From<") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => tail,
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(!status_impl.contains("::error(self)"), "TransactionStatus remote error material must not enter the private signal");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_004_hydration_contract_uses_only_transport_facade_common_raw_and_existing_ingress() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"get_transaction_observed",
|
||||||
|
"SolanaGetTransactionConfig::new",
|
||||||
|
"SolanaTransactionEncoding::Base64",
|
||||||
|
"format_raw_transaction_signature",
|
||||||
|
"extract_raw_transaction_signature_from_binary_base64",
|
||||||
|
"RawTransactionMaterial::binary_base64",
|
||||||
|
"RawTransactionIngress",
|
||||||
|
"yellowstone_http",
|
||||||
|
"transaction_get_transaction",
|
||||||
|
"status_get_transaction",
|
||||||
|
"RawAcquisitionOrigin::Live",
|
||||||
|
"composite_provenance_codes",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 hydration contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(resources.contains("fn finalize_hydration"));
|
||||||
|
assert!(resources.contains("fn fetch_hydration"));
|
||||||
|
for forbidden in ["ksp_config_lib::", "ksp_job_backfill_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(!resources.contains(forbidden) && !root.contains(forbidden), "pre.004 crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_005_block_and_continuity_adapters_remain_private_and_transport_facade_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestYellowstoneBlockView",
|
||||||
|
"YellowstoneBlockUpdate",
|
||||||
|
"project_yellowstone_block_signals",
|
||||||
|
"RawTransactionIngestYellowstoneContinuityView",
|
||||||
|
"YellowstoneBlockMetaUpdate",
|
||||||
|
"YellowstoneSlotUpdate",
|
||||||
|
"RawTransactionIngestContinuitySignal",
|
||||||
|
"block_get_transaction",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 private adapter contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp_config_lib::", "ksp_job_backfill_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(!resources.contains(forbidden) && !root.contains(forbidden), "pre.005 crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in ["children.spawn", ".run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"] {
|
||||||
|
assert!(runtime.contains(required), "required pre.006 supervisor wiring missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"open_standard_subscribe",
|
||||||
|
"next_update",
|
||||||
|
"RawTransactionIngestHydrationCoordinator",
|
||||||
|
"std::collections::BTreeMap",
|
||||||
|
"tokio::task::JoinSet",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"admission_sender.send(ingress)",
|
||||||
|
"tasks.abort_all()",
|
||||||
|
"session.close().await",
|
||||||
|
"commitment: hydration.commitment.as_str()",
|
||||||
|
"network: signal.network.clone()",
|
||||||
|
"signature: signal.signature",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 productive-source contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(!runtime.contains(forbidden) && !resources.contains(forbidden), "pre.006 crossed a forbidden productive-source boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_006_http_block_polling_reuses_transport_facades_without_becoming_backfill() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHttpBlockPollingSource",
|
||||||
|
"get_slot(&self.polling_role",
|
||||||
|
"get_blocks_with_limit(",
|
||||||
|
"get_block_observed(&self.polling_role",
|
||||||
|
"let mut next_scan_slot = start_slot",
|
||||||
|
"next_scan_slot = slot",
|
||||||
|
"tokio::time::sleep(self.poll_interval)",
|
||||||
|
"RawAcquisitionOrigin::Live",
|
||||||
|
"block_polling_get_block",
|
||||||
|
"SolanaTransactionEncoding::Base64",
|
||||||
|
"SolanaTransactionDetails::Full",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 HTTP polling contract missing: {required}");
|
||||||
|
}
|
||||||
|
let source_impl = match resources.split_once("impl crate::RawTransactionIngestHttpBlockPollingSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for crate::RawTransactionIngestHttpBlockPollingSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert_eq!(source_impl.matches("get_block_observed(").count(), 1);
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(!source_impl.contains(forbidden) && !root.contains(forbidden), "pre.006 HTTP polling crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
"SolanaStandardWsSession::connect",
|
||||||
|
".logs_subscribe(",
|
||||||
|
"SolanaLogsSubscribeFilter",
|
||||||
|
"SolanaCommitmentConfig::new",
|
||||||
|
"project_standard_logs_signal",
|
||||||
|
"RawTransactionIngestSourceFamily::Logs",
|
||||||
|
"logs_get_transaction",
|
||||||
|
"solana_ws_http",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 standard logs contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("run_live_sources"));
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!runtime.contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
|
||||||
|
"pre.003 crossed a forbidden boundary: {forbidden}",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
"SolanaStandardWsSession::connect",
|
||||||
|
".block_subscribe(",
|
||||||
|
"SolanaBlockSubscribeConfig::new",
|
||||||
|
"SolanaTransactionEncoding::Base64",
|
||||||
|
"SolanaTransactionDetails::Full",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
"std::option::Option::Some(false)",
|
||||||
|
"build_standard_block_material",
|
||||||
|
"RawTransactionMaterial::binary_base64_with_embedded_signature",
|
||||||
|
"RawTransactionVersion::Number(1)",
|
||||||
|
"block_subscribe",
|
||||||
|
"source.standard_block_missing",
|
||||||
|
"source.standard_block_remote_error",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 standard block contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!runtime_contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
|
||||||
|
"pre.004 crossed a forbidden standard-block boundary: {forbidden}",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHeliusTransactionSource",
|
||||||
|
"HeliusLaserStreamWsSession::connect",
|
||||||
|
".transaction_subscribe(",
|
||||||
|
"HeliusTransactionSubscribeOptions::new",
|
||||||
|
"HeliusTransactionSubscribeEncoding::Base64",
|
||||||
|
"SolanaTransactionDetails::Full",
|
||||||
|
"std::option::Option::Some(1)",
|
||||||
|
"project_helius_transaction_signal",
|
||||||
|
"helius_ws_http",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 Helius transaction contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert_eq!(resources.matches("get_transaction_observed(").count(), 1);
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"helius_sdk",
|
||||||
|
"unbounded_channel",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!runtime.contains(forbidden) && !resources.contains(forbidden) && !root.contains(forbidden),
|
||||||
|
"pre.005 crossed a forbidden Helius transaction boundary: {forbidden}",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn runtime_contains(value: &str) -> bool {
|
||||||
|
return include_str!("../src/runtime.rs").contains(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_source_surface_hardens_shutdown_and_faults_without_backend_edges() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let admission = include_str!("../src/admission.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
let runtime_resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"tokio::sync::mpsc::channel",
|
||||||
|
"canonicalize_raw_transaction",
|
||||||
|
"assemble_raw_transaction_acquisition",
|
||||||
|
"RawTransactionIngestPersistencePort",
|
||||||
|
"RawTransactionAcquisitionMode::Normal",
|
||||||
|
"RawTransactionIngestSnapshot",
|
||||||
|
"RawTransactionIngestSnapshotSource",
|
||||||
|
"WorkerSnapshotSource",
|
||||||
|
"tokio::sync::watch::channel",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED",
|
||||||
|
"record_source_failure",
|
||||||
|
"tokio::time::timeout",
|
||||||
|
"abort_all",
|
||||||
|
"backpressure_wait_total",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
root.contains(required)
|
||||||
|
|| runtime.contains(required)
|
||||||
|
|| admission.contains(required)
|
||||||
|
|| persistence.contains(required)
|
||||||
|
|| snapshot.contains(required)
|
||||||
|
|| runtime_resources.contains(required),
|
||||||
|
"required pre.009 hardening contract missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::admission::RawTransactionAdmission",
|
||||||
|
"pub use self::admission::RawTransactionIngress",
|
||||||
|
"pub use self::persistence::RawTransactionIngestPersistencePort",
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ForceRehydrate",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!root.contains(forbidden)
|
||||||
|
&& !runtime.contains(forbidden)
|
||||||
|
&& !admission.contains(forbidden)
|
||||||
|
&& !persistence.contains(forbidden)
|
||||||
|
&& !snapshot.contains(forbidden)
|
||||||
|
&& !runtime_resources.contains(forbidden),
|
||||||
|
"pre.009 crossed a forbidden runtime boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
runtime.matches("tokio::sync::watch::channel(false)").count(),
|
||||||
|
2,
|
||||||
|
"runtime retains exactly the external-control and private-source stop watches"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
runtime.matches("tokio::sync::watch::channel(crate::RawTransactionIngestProcessingFrontierProjection::empty())").count(),
|
||||||
|
1,
|
||||||
|
"runtime retains exactly one run-local processing-frontier latest-value watch",
|
||||||
|
);
|
||||||
|
assert_eq!(runtime.matches("tokio::sync::watch::channel").count(), 3, "runtime watch inventory is two stop watches plus one processing-frontier watch");
|
||||||
|
assert_eq!(snapshot.matches("tokio::sync::watch::channel").count(), 1, "exactly one shared concrete snapshot watch is allowed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn dependency_section(manifest: &str) -> &str {
|
||||||
|
let after_header = match manifest.split_once("[dependencies]") {
|
||||||
|
std::option::Option::Some((_, section)) => section,
|
||||||
|
std::option::Option::None => {
|
||||||
|
return "";
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let end = match after_header.find("\n[") {
|
||||||
|
std::option::Option::Some(index) => index,
|
||||||
|
std::option::Option::None => after_header.len(),
|
||||||
|
};
|
||||||
|
return &after_header[..end];
|
||||||
|
}
|
||||||
|
|
||||||
|
fn manifest_dependency_names(section: &str) -> std::vec::Vec<&str> {
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for line in section.lines() {
|
||||||
|
let content = match line.split('#').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if content.is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match content.split('=').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim().trim_end_matches(".workspace"),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if !name.is_empty() {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
return names;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_007_multi_source_supervisor_and_inventory_remain_private_bounded_and_source_neutral() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestSourceInventory",
|
||||||
|
"RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"for (entry_index, source) in self.sources.into_iter().enumerate()",
|
||||||
|
"tokio::task::JoinSet::new()",
|
||||||
|
"supervise_live_source_tasks",
|
||||||
|
"drain_live_source_tasks",
|
||||||
|
"source_stop_sender.send_replace(true)",
|
||||||
|
"processing_frontier_sender.clone()",
|
||||||
|
"source.configured_source_closed",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.007 multi-source supervisor contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("run_live_sources"));
|
||||||
|
assert!(!runtime.contains("validate_single_source_activation"));
|
||||||
|
assert!(!resources.contains("multi_source_activation_pending"));
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestSourceInventory",
|
||||||
|
"pub struct RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!root.contains(forbidden) && !runtime.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.007 crossed a private/bounded boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_007_processing_frontier_remains_run_local_and_backend_neutral_after_continuity_extension() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestProcessingFrontierProjection",
|
||||||
|
"processing_frontier_sender",
|
||||||
|
"wait_processing_frontier",
|
||||||
|
"record_processing_frontier",
|
||||||
|
"hydration_pending",
|
||||||
|
"processing_frontier_slot",
|
||||||
|
"oldest_pending_slot",
|
||||||
|
"RawTransactionIngestProcessingFrontier",
|
||||||
|
"settle_pending",
|
||||||
|
"observe_settled",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
runtime.contains(required) || resources.contains(required) || snapshot.contains(required),
|
||||||
|
"required pre.007 frontier contract missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let frontier = match resources.split_once("struct RawTransactionIngestProcessingFrontier {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("struct RawTransactionIngestProcessingFrontierReporter") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["SubscribeReplayInfo", "from_slot", "repair", "ksp_job_backfill_lib::", "ksp_store_postgres_lib::"] {
|
||||||
|
assert!(!frontier.contains(forbidden), "pre.007 processing frontier absorbed replay/backend responsibility: {forbidden}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp_job_backfill_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(
|
||||||
|
!runtime.contains(forbidden) && !resources.contains(forbidden) && !snapshot.contains(forbidden),
|
||||||
|
"Worker crossed backend boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let transport = include_str!("../../ksp-onchain-transport-lib/src/grpc_stream.rs");
|
||||||
|
for required in [
|
||||||
|
"snapshot_source()",
|
||||||
|
"wait_yellowstone_session_snapshot",
|
||||||
|
"source_reconnect_total",
|
||||||
|
"source_replay_attempt_total",
|
||||||
|
"source_continuity_gap_total",
|
||||||
|
"source.continuity_gap_proven",
|
||||||
|
"RawTransactionIngestSourceState::Reconnecting",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.008 Worker continuity token missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["subscribe_replay_info(", "set_from_slot(", "YellowstoneReplayInfo", "last_requested_from_slot()", "ksp_job_backfill_lib::"] {
|
||||||
|
assert!(!resources.contains(forbidden), "Worker took ownership of replay/repair responsibility: {forbidden}");
|
||||||
|
}
|
||||||
|
for required in ["SubscribeReplayInfo", "set_from_slot(effective_from_slot)", "first_available > requested", "replay_attempt_count"] {
|
||||||
|
assert!(transport.contains(required), "Transport replay ownership token missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_only() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestPersistenceConvergence",
|
||||||
|
"persist_raw_transaction_ingest_converged_acquisition",
|
||||||
|
"RawTransactionIngestPersistencePort",
|
||||||
|
"record_raw_transaction_observation",
|
||||||
|
"RawTransactionObservationWrite",
|
||||||
|
"payload().content_hash()",
|
||||||
|
"content_conflict_error()",
|
||||||
|
] {
|
||||||
|
assert!(persistence.contains(required) || root.contains(required), "required pre.008 persistence convergence contract missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"fetch_hydration_shared",
|
||||||
|
"subscribe_or_lead",
|
||||||
|
"get_transaction_observed",
|
||||||
|
"settings.admission_queue_capacity()",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.008 hydration convergence contract missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("settings.admission_queue_capacity().max(settings.persistence_concurrency())"));
|
||||||
|
for forbidden in ["ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::", "ksp_job_backfill_lib::", "unbounded_channel"] {
|
||||||
|
assert!(
|
||||||
|
!persistence.contains(forbidden) && !runtime.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.008 convergence crossed a dependency/boundedness boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades() {
|
||||||
|
let admission = include_str!("../src/admission.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"validate_hydration_fairness_capacity",
|
||||||
|
"let hydration_pending_budget = settings.admission_queue_capacity();",
|
||||||
|
"let hydration_in_flight_budget = settings.persistence_concurrency();",
|
||||||
|
"hydration_pending_limit",
|
||||||
|
"hydration_in_flight_limit",
|
||||||
|
"tokio::sync::Semaphore",
|
||||||
|
"acquire_owned",
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"RawTransactionIngestCanonicalState",
|
||||||
|
"content_conflict_error()",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
admission.contains(required) || persistence.contains(required) || resources.contains(required),
|
||||||
|
"required pre.009 global-bound contract missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!admission.contains(forbidden) && !persistence.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.009 crossed a Worker facade boundary: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backend_free() {
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in ["source_total", "source_active", "source_reconnecting", "source_failed", "WorkerHealth::Degraded", "with_source_counts"] {
|
||||||
|
assert!(snapshot.contains(required) || resources.contains(required), "required pre.010 source-neutral snapshot contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::", "SourceSnapshotByKey", "provider_health"] {
|
||||||
|
assert!(!snapshot.contains(forbidden), "pre.010 snapshot crossed source-neutral/backend boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
|
||||||
|
"persistence.abort_all()",
|
||||||
|
"children.abort_all()",
|
||||||
|
"while persistence.join_next().await.is_some() {}",
|
||||||
|
"while children.join_next().await.is_some() {}",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
] {
|
||||||
|
assert!(runtime.contains(required), "required pre.011 runtime shutdown guard missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"publish_and_remove",
|
||||||
|
"pending.remove(key)",
|
||||||
|
"source.inventory_key_mismatch",
|
||||||
|
"checked_add(projection.hydration_pending())",
|
||||||
|
"checked_add(projection.source_reconnect_total())",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.011 source-race guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"unbounded_channel",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_job_backfill_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
] {
|
||||||
|
assert!(!runtime.contains(forbidden) && !resources.contains(forbidden), "pre.011 crossed a Worker/facade boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_cross_layer_closure_does_not_expand_worker_dependency_graph() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let dependencies = dependency_section(manifest);
|
||||||
|
let names = manifest_dependency_names(dependencies);
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
vec![
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline",
|
||||||
|
"v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store",
|
||||||
|
"v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration",
|
||||||
|
"v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "pre.012 cross-layer closure proof missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
959
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
Normal file
959
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
Normal file
@@ -0,0 +1,959 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
// version: 24
|
||||||
|
|
||||||
|
//! External public, security, redaction and release-boundary hardening canaries through `pre.012`.
|
||||||
|
|
||||||
|
fn network(value: &'static str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
let result = ksp_store_lib::RawNetworkId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_store_lib::ERROR_CODE_RAW_MODEL_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn worker_id(value: &'static str) -> std::option::Option<ksp_worker_api::WorkerId> {
|
||||||
|
let result = ksp_worker_api::WorkerId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_worker_api::ERROR_CODE_WORKER_ID_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_error_codes_are_stable_unique_and_domain_scoped() {
|
||||||
|
let codes = [
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT, "content_conflict"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED, "counter_exhausted"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT, "drain_timeout"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID, "runtime_invalid"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID, "settings_invalid"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED, "source_failed"),
|
||||||
|
(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED, "store_failed"),
|
||||||
|
];
|
||||||
|
for (index, (code, expected)) in codes.iter().enumerate() {
|
||||||
|
assert_eq!(code.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(code.code(), *expected);
|
||||||
|
for (other_index, (other, _)) in codes.iter().enumerate() {
|
||||||
|
if index != other_index {
|
||||||
|
assert_ne!(code, other);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values() {
|
||||||
|
let raw_worker_id = "raw-ingest-secret-marker-001";
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let parsed_worker_id = match worker_id(raw_worker_id) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(std::format!("{parsed_worker_id:?}"), "WorkerId(..)");
|
||||||
|
let settings = match ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(
|
||||||
|
network.clone(),
|
||||||
|
parsed_worker_id,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
std::time::Duration::from_millis(100),
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let settings_debug = std::format!("{settings:?}");
|
||||||
|
assert!(!settings_debug.contains(raw_worker_id));
|
||||||
|
assert!(settings_debug.contains("WorkerId(..)"));
|
||||||
|
let invalid_worker_id = match worker_id("raw-ingest-secret-marker-002") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let invalid =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(network, invalid_worker_id, 65_537, 1, std::time::Duration::from_millis(100));
|
||||||
|
let error = match invalid {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(error) => error,
|
||||||
|
};
|
||||||
|
let error_debug = std::format!("{error:?}");
|
||||||
|
assert_eq!(error.code(), ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert!(!error_debug.contains("raw-ingest-secret-marker-002"));
|
||||||
|
assert!(!error_debug.contains("65537"));
|
||||||
|
assert!(error.context().iter().any(|context| return context.key() == "field" && context.value() == "admission_queue_capacity"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_manifest_dependency_surface_opens_only_transport_and_remains_backend_neutral() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let mut section = "";
|
||||||
|
let mut normal = std::collections::BTreeSet::new();
|
||||||
|
let mut dev = std::collections::BTreeSet::new();
|
||||||
|
let mut build = std::collections::BTreeSet::new();
|
||||||
|
for line in manifest.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.starts_with('[') && trimmed.ends_with(']') {
|
||||||
|
section = trimmed;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if trimmed.is_empty() || trimmed.starts_with('#') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match trimmed.split_once('=') {
|
||||||
|
std::option::Option::Some((name, _)) => name.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if section == "[dependencies]" {
|
||||||
|
normal.insert(name);
|
||||||
|
} else if section == "[dev-dependencies]" {
|
||||||
|
dev.insert(name);
|
||||||
|
} else if section == "[build-dependencies]" {
|
||||||
|
build.insert(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
normal,
|
||||||
|
std::collections::BTreeSet::from([
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-raw-transaction-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"ksp-worker-api",
|
||||||
|
"sha2",
|
||||||
|
"tokio",
|
||||||
|
])
|
||||||
|
);
|
||||||
|
assert!(dev.is_empty());
|
||||||
|
assert!(build.is_empty());
|
||||||
|
assert!(manifest.contains("ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }"));
|
||||||
|
assert!(manifest.contains("tokio = { workspace = true, features = [\"macros\", \"rt\", \"sync\", \"time\"] }"));
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-job-backfill-lib",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"reqwest",
|
||||||
|
"tokio-tungstenite",
|
||||||
|
"tonic",
|
||||||
|
"yellowstone-grpc-proto",
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains(forbidden), "forbidden Worker manifest dependency present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_source_visibility_contract_uses_crate_root_for_shared_items() {
|
||||||
|
let source_contracts: [(&str, &[&str]); 5] = [
|
||||||
|
(include_str!("../src/settings.rs"), &["RawTransactionIngestSettings"]),
|
||||||
|
(include_str!("../src/runtime.rs"), &["RawTransactionIngestHandle", "RawTransactionIngestWorker"]),
|
||||||
|
(include_str!("../src/snapshot.rs"), &["RawTransactionIngestSnapshot", "RawTransactionIngestSnapshotSource"]),
|
||||||
|
(include_str!("../src/persistence.rs"), &["RawTransactionIngestPersistenceOutcome", "RawTransactionIngestPersistencePort"]),
|
||||||
|
(
|
||||||
|
include_str!("../src/runtime_resources.rs"),
|
||||||
|
&[
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
"RawTransactionIngestYellowstoneSource",
|
||||||
|
"RawTransactionIngestRuntimeResources",
|
||||||
|
],
|
||||||
|
),
|
||||||
|
];
|
||||||
|
for (source, symbols) in source_contracts {
|
||||||
|
for symbol in symbols {
|
||||||
|
let required = std::format!("impl crate::{symbol}");
|
||||||
|
assert!(source.contains(required.as_str()), "shared item must use crate-root impl path: {symbol}");
|
||||||
|
let forbidden = std::format!("impl {symbol}");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "shared item must not use bare impl path: {symbol}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (module, source) in [
|
||||||
|
("admission", include_str!("../src/admission.rs")),
|
||||||
|
("error", include_str!("../src/error.rs")),
|
||||||
|
("identity", include_str!("../src/identity.rs")),
|
||||||
|
("persistence", include_str!("../src/persistence.rs")),
|
||||||
|
("runtime", include_str!("../src/runtime.rs")),
|
||||||
|
("runtime_resources", include_str!("../src/runtime_resources.rs")),
|
||||||
|
("settings", include_str!("../src/settings.rs")),
|
||||||
|
("snapshot", include_str!("../src/snapshot.rs")),
|
||||||
|
] {
|
||||||
|
let forbidden = std::format!("crate::{module}::");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "internal module path bypasses crate-root façade: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_surface_has_no_historical_backfill_or_retriever_contract() {
|
||||||
|
let sources = [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/runtime_resources.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
];
|
||||||
|
for source in sources {
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-worker-raw-retriever",
|
||||||
|
"raw_transaction_retriever",
|
||||||
|
"RawTransactionRetriever",
|
||||||
|
"Backfill",
|
||||||
|
"backfill",
|
||||||
|
"Checkpoint",
|
||||||
|
"checkpoint",
|
||||||
|
"historical",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "historical/retriever surface leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_production_sources_keep_transport_confined_to_runtime_resources() {
|
||||||
|
for source in [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/runtime_resources.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
] {
|
||||||
|
let lower = source.to_ascii_lowercase();
|
||||||
|
for forbidden in ["api_key", "api-key", "authorization", "bearer ", "password", "credential", "secret"] {
|
||||||
|
assert!(!lower.contains(forbidden), "secret-like material leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let transport_source = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(transport_source.contains("ksp_onchain_transport_lib::"));
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_offchain_transport_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"postgresql://",
|
||||||
|
"postgres://",
|
||||||
|
] {
|
||||||
|
assert!(!transport_source.contains(forbidden), "forbidden implementation detail leaked into runtime resources: {forbidden}");
|
||||||
|
}
|
||||||
|
for source in [
|
||||||
|
include_str!("../src/admission.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/persistence.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
include_str!("../src/settings.rs"),
|
||||||
|
include_str!("../src/snapshot.rs"),
|
||||||
|
] {
|
||||||
|
assert!(!source.contains("ksp_onchain_transport_lib::"), "Transport dependency escaped runtime_resources.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_offchain_transport_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tokio_tungstenite::",
|
||||||
|
"tonic::",
|
||||||
|
"yellowstone_grpc_proto::",
|
||||||
|
"postgresql://",
|
||||||
|
"postgres://",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "forbidden implementation detail leaked into Worker production source: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(resources.contains("struct RawTransactionIngestSourceSignal"));
|
||||||
|
assert!(!root.contains("RawTransactionIngestSourceSignal"));
|
||||||
|
assert!(!resources.contains(".field(\"matched_filter_fingerprint\", &self.matched_filter_fingerprint)"));
|
||||||
|
assert!(!resources.contains(".field(\"signature\", &self.signature)"));
|
||||||
|
let signal_struct = match resources.split_once("struct RawTransactionIngestSourceSignal {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for RawTransactionIngestSourceSignal") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["transaction:", "meta:", "error:", "payload:", "body:", "is_vote:"] {
|
||||||
|
assert!(!signal_struct.contains(forbidden), "payload/provider-specific field leaked into private source signal: {forbidden}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"created_at:",
|
||||||
|
"family:",
|
||||||
|
"matched_filter_count:",
|
||||||
|
"matched_filter_fingerprint:",
|
||||||
|
"matched_filter_id:",
|
||||||
|
"network:",
|
||||||
|
"route:",
|
||||||
|
"signature:",
|
||||||
|
"slot:",
|
||||||
|
"transaction_index:",
|
||||||
|
] {
|
||||||
|
assert!(signal_struct.contains(required), "required private signal field missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"route_prefix",
|
||||||
|
"{}.{}:http.{}",
|
||||||
|
"composite_provider_unrepresentable",
|
||||||
|
"composite_endpoint_unrepresentable",
|
||||||
|
"RawAcquisitionOrigin::Live",
|
||||||
|
"with_capture_session_id",
|
||||||
|
"with_commitment",
|
||||||
|
"with_endpoint_id",
|
||||||
|
"with_filter_id",
|
||||||
|
"try_with_observed_at",
|
||||||
|
"hydration.signature_mismatch",
|
||||||
|
"hydration.slot_mismatch",
|
||||||
|
"hydration.transaction_index_mismatch",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 bounded provenance/mismatch guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["source_payload_hash", "source_payload_size_bytes", "HTTP-SECRET-CANARY", "GRPC-SECRET-CANARY", "TransactionStatus.error", ".error()"] {
|
||||||
|
assert!(!resources.contains(forbidden), "pre.004 retained forbidden remote/source material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_005_block_and_continuity_shapes_are_bounded_redacted_and_raw_separated() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"impl RawTransactionIngestYellowstoneBlockView for ksp_onchain_transport_lib::YellowstoneBlockUpdate",
|
||||||
|
"impl RawTransactionIngestYellowstoneContinuityView for ksp_onchain_transport_lib::YellowstoneBlockMetaUpdate",
|
||||||
|
"impl RawTransactionIngestYellowstoneContinuityView for ksp_onchain_transport_lib::YellowstoneSlotUpdate",
|
||||||
|
"RawTransactionIngestSourceFamily::Block",
|
||||||
|
"project_yellowstone_block_signals",
|
||||||
|
"project_yellowstone_continuity_signal",
|
||||||
|
"RawTransactionIngestContinuityStatus::Dead",
|
||||||
|
"block_get_transaction",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 bounded adapter guard missing: {required}");
|
||||||
|
}
|
||||||
|
let continuity_struct = match resources.split_once("struct RawTransactionIngestContinuitySignal {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for RawTransactionIngestContinuitySignal") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for required in [
|
||||||
|
"created_at:",
|
||||||
|
"family:",
|
||||||
|
"matched_filter_count:",
|
||||||
|
"matched_filter_fingerprint:",
|
||||||
|
"matched_filter_id:",
|
||||||
|
"network:",
|
||||||
|
"parent_slot:",
|
||||||
|
"route:",
|
||||||
|
"slot:",
|
||||||
|
"status:",
|
||||||
|
] {
|
||||||
|
assert!(continuity_struct.contains(required), "required continuity-only field missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["signature:", "transaction:", "meta:", "payload:", "body:", "error:", "dead_error:"] {
|
||||||
|
assert!(!continuity_struct.contains(forbidden), "RAW/remote material leaked into continuity-only signal: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!resources.contains("YellowstoneSlotUpdate::dead_error"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_runtime_resource_contract_opens_one_supervised_transport_source() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
for required in ["open_standard_subscribe", "next_update", "get_transaction_observed", "RawTransactionIngestHydrationCoordinator", "session.close().await"]
|
||||||
|
{
|
||||||
|
assert!(resources.contains(required), "productive runtime-resource source behavior missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(runtime.contains("start_with_runtime_resources"));
|
||||||
|
assert!(runtime.contains("run_live_sources(source_settings, stop_receiver, admission_sender, processing_frontier_sender)"));
|
||||||
|
for forbidden in ["ksp_config_lib::", "ksp_store_postgres_lib::", "reqwest::", "tonic::", "yellowstone_grpc_proto::"] {
|
||||||
|
assert!(!resources.contains(forbidden) && !runtime.contains(forbidden), "pre.006 runtime source crossed a forbidden boundary: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_006_source_coalescence_is_bounded_stop_preemptible_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"max_in_flight: usize",
|
||||||
|
"max_pending_signals: usize",
|
||||||
|
"RawTransactionIngestHydrationCoordinator::with_global_registry(",
|
||||||
|
"hydration_pending_limit,",
|
||||||
|
"hydration_in_flight_limit,",
|
||||||
|
"pending_signal_count",
|
||||||
|
"stop_receiver.changed()",
|
||||||
|
"tasks.abort_all()",
|
||||||
|
"source.hydration_pending_saturated",
|
||||||
|
"source.hydration_task_join_failed",
|
||||||
|
"hydration_transport_error(error.code())",
|
||||||
|
"source_transport_error(error.code())",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 bounded/stop-safe source guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["remote_message", "response_body", "dead_error", "TransactionStatus.error", ".error()", "Authorization", "Bearer "] {
|
||||||
|
assert!(!resources.contains(forbidden), "remote/secret material leaked into productive source: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_lower_layers_have_no_dependency_return_to_concrete_worker() {
|
||||||
|
for manifest in [
|
||||||
|
include_str!("../../ksp-core-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-logging-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-raw-transaction-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-api/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-postgres-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-worker-api/Cargo.toml"),
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains("ksp-worker-raw-transaction-ingest-lib"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub mod ",
|
||||||
|
"tokio::",
|
||||||
|
"JoinSet",
|
||||||
|
"watch::Receiver",
|
||||||
|
"watch::Sender",
|
||||||
|
"mpsc::Sender",
|
||||||
|
"ksp_store_postgres_lib::",
|
||||||
|
"ksp_onchain_transport_lib::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"tonic::",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "implementation/backend/live-source detail leaked into public root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_007_processing_frontier_is_bounded_processing_only_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
for required in [
|
||||||
|
"pending_total",
|
||||||
|
"std::collections::BTreeMap<u64, RawTransactionIngestProcessingSlotState>",
|
||||||
|
"settled_interval_highs",
|
||||||
|
"oldest_pending_slot",
|
||||||
|
"processing_frontier_slot",
|
||||||
|
"hydration_pending",
|
||||||
|
"processing_frontier.observe_pending",
|
||||||
|
"processing_frontier.settle_pending",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required) || snapshot.contains(required), "required pre.007 bounded frontier guard missing: {required}");
|
||||||
|
}
|
||||||
|
let frontier = match resources.split_once("struct RawTransactionIngestProcessingFrontier {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("struct RawTransactionIngestProcessingFrontierReporter") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in [
|
||||||
|
"signature: ksp_store_lib::RawTransactionSignature",
|
||||||
|
"filter_id",
|
||||||
|
"provider",
|
||||||
|
"endpoint_id",
|
||||||
|
"transaction:",
|
||||||
|
"meta:",
|
||||||
|
"ReplayInfo",
|
||||||
|
"from_slot",
|
||||||
|
"continuity_gap",
|
||||||
|
"repair",
|
||||||
|
] {
|
||||||
|
assert!(!frontier.contains(forbidden), "pre.007 frontier absorbed forbidden material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestSourceState",
|
||||||
|
"source_reconnect_total",
|
||||||
|
"source_replay_attempt_total",
|
||||||
|
"source_continuity_gap_total",
|
||||||
|
"source.continuity_counter_regression",
|
||||||
|
"source.continuity_gap_proven",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required) || snapshot.contains(required), "required pre.008 bounded continuity guard missing: {required}");
|
||||||
|
}
|
||||||
|
let projection = match snapshot.split_once("struct RawTransactionIngestProcessingFrontierProjection {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestProcessingFrontierProjection") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["signature", "provider", "endpoint", "filter", "transaction", "meta", "from_slot", "first_available", "ReplayInfo"] {
|
||||||
|
assert!(!projection.contains(forbidden), "pre.008 source projection leaked replay/provider/RAW material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
".field(\"filter_kind\"",
|
||||||
|
".field(\"filter_fingerprint_bytes\"",
|
||||||
|
".field(\"source_key_bytes\"",
|
||||||
|
"project_standard_logs_signal",
|
||||||
|
"matched_filter_count: 1",
|
||||||
|
"transaction_index: std::option::Option::None",
|
||||||
|
"created_at: std::option::Option::None",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.003 redaction/reference guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestStandardLogsSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestStandardLogsSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(source_struct.contains("filter: ksp_onchain_transport_lib::SolanaLogsSubscribeFilter"));
|
||||||
|
for forbidden in ["logs:", "err:", "payload:", "body:", "url:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "remote/sensitive material stored in standard logs source: {forbidden}");
|
||||||
|
}
|
||||||
|
let projection = match resources.split_once("fn project_standard_logs_signal") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("fn route_yellowstone_update") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in [".logs()", ".err()"] {
|
||||||
|
assert!(!projection.contains(forbidden), "logs/error material copied into standard logs projection: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
".field(\"filter_kind\"",
|
||||||
|
".field(\"filter_fingerprint_bytes\"",
|
||||||
|
".field(\"source_key_bytes\"",
|
||||||
|
"source.standard_block_context_slot_mismatch",
|
||||||
|
"source.standard_block_remote_error",
|
||||||
|
"source.standard_block_missing",
|
||||||
|
"source.standard_block_transactions_missing",
|
||||||
|
"source.standard_block_transaction_version_unsupported",
|
||||||
|
"source.standard_block_transaction_version_unqualified",
|
||||||
|
"SolanaTransactionVersion::Number(0)",
|
||||||
|
"SolanaTransactionVersion::Number(1)",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.004 standard block hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestStandardBlockSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestStandardBlockSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["err:", "block:", "transaction_data:", "url:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "remote/raw material stored in standard block source: {forbidden}");
|
||||||
|
}
|
||||||
|
let source_impl = match resources.split_once("impl crate::RawTransactionIngestStandardBlockSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for crate::RawTransactionIngestStandardBlockSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(!source_impl.contains("observe_pending("), "direct Standard Block source must not inflate hydration_pending");
|
||||||
|
assert!(source_impl.contains("processing_frontier.observe_settled(slot)"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"RawTransactionIngestHeliusTransactionSource",
|
||||||
|
".field(\"filter\"",
|
||||||
|
".field(\"filter_fingerprint_bytes\"",
|
||||||
|
".field(\"source_key_bytes\"",
|
||||||
|
"HeliusTransactionNotification::Full(value)",
|
||||||
|
"source.helius_transaction_notification_unqualified",
|
||||||
|
"project_helius_transaction_signal",
|
||||||
|
"transaction_index: std::option::Option::Some(response.transaction_index())",
|
||||||
|
"RAW_TRANSACTION_INGEST_HELIUS_TRANSACTION_HTTP_PROTOCOL",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.005 Helius hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestHeliusTransactionSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestHeliusTransactionSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["url:", "api_key", "credential", "transaction:", "payload:", "tier:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "sensitive/provider material stored in Helius transaction source: {forbidden}");
|
||||||
|
}
|
||||||
|
let projection = match resources.split_once("fn project_helius_transaction_signal") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("trait RawTransactionIngestStandardLogsView") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(!projection.contains(".transaction()"), "Helius nested full payload must not enter Worker projection");
|
||||||
|
for forbidden in ["Developer", "Business", "Professional", "paid_tier", "provider_tier"] {
|
||||||
|
assert!(!resources.contains(forbidden), "provider tier must not be coded in Worker: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"let mut next_scan_slot = start_slot",
|
||||||
|
"next_scan_slot = slot",
|
||||||
|
"tokio::time::sleep(self.poll_interval)",
|
||||||
|
"processing_frontier.observe_settled(slot)",
|
||||||
|
"source.http_block_polling_transaction_version_unqualified",
|
||||||
|
"source.http_block_polling_transaction_version_unsupported",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.006 HTTP polling hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
let source_impl = match resources.split_once("impl crate::RawTransactionIngestHttpBlockPollingSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl std::fmt::Debug for crate::RawTransactionIngestHttpBlockPollingSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert_eq!(source_impl.matches("get_block_observed(").count(), 1);
|
||||||
|
assert!(!source_impl.contains("tokio::spawn"), "HTTP polling must not spawn one task per tick");
|
||||||
|
assert!(!source_impl.contains("observe_pending("), "HTTP block polling is direct RAW and must not inflate hydration_pending");
|
||||||
|
let source_struct = match resources.split_once("pub struct RawTransactionIngestHttpBlockPollingSource {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl crate::RawTransactionIngestHttpBlockPollingSource") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["url:", "api_key", "credential", "secret", "transaction:", "payload:", "tier:"] {
|
||||||
|
assert!(!source_struct.contains(forbidden), "sensitive/raw material stored in HTTP polling source: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit() {
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert_eq!(resources.matches("get_transaction_observed(").count(), 1, "Worker must delegate one hydration attempt to Transport");
|
||||||
|
for required in [
|
||||||
|
"coordinator.abort_all(&mut processing_frontier).await",
|
||||||
|
"fn discard_all_pending(&mut self)",
|
||||||
|
"processing_frontier.discard_all_pending()",
|
||||||
|
"source.hydration_pending_saturated",
|
||||||
|
"tasks.abort_all()",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.009 no-orphan/backpressure guard missing: {required}");
|
||||||
|
}
|
||||||
|
let hydration = match resources.split_once("struct RawTransactionIngestHydrationCoordinator {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("fn hydration_method_code(") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for forbidden in ["tokio::time::sleep", "tokio::time::interval", "get_block_observed"] {
|
||||||
|
assert!(!hydration.contains(forbidden), "hydration coordinator introduced forbidden retry behavior: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!resources.contains("unbounded_channel"), "Worker introduced an unbounded channel");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES",
|
||||||
|
"RawTransactionIngestSourceInventory",
|
||||||
|
"std::sync::Mutex",
|
||||||
|
"tokio::task::JoinSet",
|
||||||
|
"source.configured_source_closed",
|
||||||
|
"source.task_join_failed",
|
||||||
|
"source_stop_sender.send_replace(true)",
|
||||||
|
"drain_live_source_tasks",
|
||||||
|
"processing_frontier_slot = std::option::Option::None",
|
||||||
|
"saturating_add",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.007 supervisor hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestSourceInventory",
|
||||||
|
"pub struct RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"pub fn source_key(",
|
||||||
|
"source_keys:",
|
||||||
|
"unbounded_channel",
|
||||||
|
"primary_source",
|
||||||
|
"standby_source",
|
||||||
|
"first_provider_wins",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "private source identity/scheduling policy leaked into public root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"max_entries: usize",
|
||||||
|
"entries.len() >= self.max_entries",
|
||||||
|
"std::sync::Arc::strong_count(entry) == 1",
|
||||||
|
"known_state_value != &canonical_state",
|
||||||
|
"record_observation(observation)",
|
||||||
|
"persistence.additional_observation_not_recorded",
|
||||||
|
] {
|
||||||
|
assert!(persistence.contains(required), "required pre.008 persistence hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
for required in [
|
||||||
|
"max_pending: usize",
|
||||||
|
"pending.len() >= self.max_pending",
|
||||||
|
"source.global_hydration_pending_saturated",
|
||||||
|
"tokio::sync::watch::channel",
|
||||||
|
"publish_and_remove",
|
||||||
|
] {
|
||||||
|
assert!(resources.contains(required), "required pre.008 hydration hardening guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"pub struct RawTransactionIngestPersistenceConvergence",
|
||||||
|
"pub struct RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"pub fn persist_raw_transaction_ingest_converged_acquisition",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.008 private convergence implementation leaked publicly: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!resources.contains("unbounded_channel"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit() {
|
||||||
|
let admission = include_str!("../src/admission.rs");
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"runtime_resources.hydration_pending_capacity_insufficient",
|
||||||
|
"runtime_resources.hydration_concurrency_insufficient",
|
||||||
|
"base + usize::from(hydration_entry_index < remainder)",
|
||||||
|
"hydration_in_flight_limit",
|
||||||
|
"tokio::sync::Semaphore::new(max_in_flight)",
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"source.global_hydration_pending_saturated",
|
||||||
|
"block_time: transaction.block_time()",
|
||||||
|
"slot: transaction.slot()",
|
||||||
|
"content_hash: transaction.payload().content_hash()",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
admission.contains(required) || persistence.contains(required) || resources.contains(required),
|
||||||
|
"required pre.009 adversarial guard missing: {required}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"VecDeque::new()",
|
||||||
|
"unbounded_channel",
|
||||||
|
"first_provider_wins",
|
||||||
|
"provider_priority",
|
||||||
|
"majority_vote",
|
||||||
|
"preferred_provider",
|
||||||
|
"overwrite_conflict",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!admission.contains(forbidden) && !persistence.contains(forbidden) && !resources.contains(forbidden),
|
||||||
|
"pre.009 introduced forbidden unbounded/provider-preference behavior: {forbidden}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted() {
|
||||||
|
let snapshot = include_str!("../src/snapshot.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for required in [
|
||||||
|
"source_total: usize",
|
||||||
|
"source_active: usize",
|
||||||
|
"source_reconnecting: usize",
|
||||||
|
"source_failed: usize",
|
||||||
|
"source_reconnecting > 0",
|
||||||
|
"source_failed > 0",
|
||||||
|
"source_active < source_total",
|
||||||
|
"WorkerHealth::Degraded",
|
||||||
|
"WorkerHealth::Unhealthy",
|
||||||
|
] {
|
||||||
|
assert!(snapshot.contains(required), "required pre.010 health guard missing: {required}");
|
||||||
|
}
|
||||||
|
for required in ["source_total = self.source_projections.len()", "with_source_counts(source_total, source_active, source_reconnecting, source_failed)"] {
|
||||||
|
assert!(resources.contains(required), "required pre.010 inventory count guard missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["source_key: [u8; 32]", "provider_url", "endpoint_url", "filter_fingerprint", "credential", "api_key", "token_header"] {
|
||||||
|
assert!(!snapshot.contains(forbidden), "pre.010 snapshot leaks source/provider material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe() {
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
let supervisor = match runtime.split_once("async fn run_supervisor<Spawner>(") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("fn spawn_persistence(") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
let active_shutdown = match supervisor.split_once("let mut fault = supervise_until_stop(") {
|
||||||
|
std::option::Option::Some((_, value)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
let stop_position = active_shutdown.find("source_stop_sender.send_replace(true);");
|
||||||
|
let stopping_position = active_shutdown.find("begin_stopping(");
|
||||||
|
let drain_position = active_shutdown.find("drain_owned_work(");
|
||||||
|
let terminal_position = active_shutdown.find("match fault {");
|
||||||
|
assert!(matches!((stop_position, stopping_position), (std::option::Option::Some(stop), std::option::Option::Some(stopping)) if stop < stopping));
|
||||||
|
assert!(matches!((stopping_position, drain_position), (std::option::Option::Some(stopping), std::option::Option::Some(drain)) if stopping < drain));
|
||||||
|
assert!(matches!((drain_position, terminal_position), (std::option::Option::Some(drain), std::option::Option::Some(terminal)) if drain < terminal));
|
||||||
|
for required in [
|
||||||
|
"tokio::time::timeout(settings.shutdown_drain_timeout(), drain).await",
|
||||||
|
"persistence.abort_all()",
|
||||||
|
"children.abort_all()",
|
||||||
|
"while persistence.join_next().await.is_some() {}",
|
||||||
|
"while children.join_next().await.is_some() {}",
|
||||||
|
] {
|
||||||
|
assert!(runtime.contains(required), "required pre.011 bounded-drain guard missing: {required}");
|
||||||
|
}
|
||||||
|
let registry_publish = match resources.split_once("fn publish_and_remove(&self, key: &RawTransactionIngestHydrationKey") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("type RawTransactionIngestHydrationTasks") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
let notify_position = registry_publish.find("sender.send_replace(std::option::Option::Some(result));");
|
||||||
|
let remove_position = registry_publish.find("pending.remove(key);");
|
||||||
|
assert!(matches!((notify_position, remove_position), (std::option::Option::Some(notify), std::option::Option::Some(remove)) if notify < remove));
|
||||||
|
let inventory = match resources.split_once("impl RawTransactionIngestSourceInventory {") {
|
||||||
|
std::option::Option::Some((_, tail)) => match tail.split_once("impl RawTransactionIngestSourceInventoryPublisher") {
|
||||||
|
std::option::Option::Some((value, _)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
},
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
for required in [
|
||||||
|
"checked_add(projection.hydration_pending())",
|
||||||
|
"checked_add(projection.source_continuity_gap_total())",
|
||||||
|
"checked_add(projection.source_reconnect_total())",
|
||||||
|
"checked_add(projection.source_replay_attempt_total())",
|
||||||
|
"source.inventory_entry_missing",
|
||||||
|
"source.inventory_key_mismatch",
|
||||||
|
"source.inventory_projection_missing",
|
||||||
|
] {
|
||||||
|
assert!(inventory.contains(required), "required pre.011 inventory guard missing: {required}");
|
||||||
|
}
|
||||||
|
assert!(!inventory.contains("saturating_add"), "pre.011 source inventory must not silently saturate aggregate counters");
|
||||||
|
for source_marker in [
|
||||||
|
"impl crate::RawTransactionIngestYellowstoneSource",
|
||||||
|
"impl crate::RawTransactionIngestHeliusTransactionSource",
|
||||||
|
"impl crate::RawTransactionIngestHttpBlockPollingSource",
|
||||||
|
"impl crate::RawTransactionIngestStandardBlockSource",
|
||||||
|
"impl crate::RawTransactionIngestStandardLogsSource",
|
||||||
|
] {
|
||||||
|
let tail = match resources.split_once(source_marker) {
|
||||||
|
std::option::Option::Some((_, value)) => value,
|
||||||
|
std::option::Option::None => "",
|
||||||
|
};
|
||||||
|
assert!(tail.contains("stop_receiver.changed()"), "pre.011 source lacks stop-preemptible wait: {source_marker}");
|
||||||
|
}
|
||||||
|
assert!(!runtime.contains("unbounded_channel"));
|
||||||
|
assert!(!resources.contains("unbounded_channel"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_cross_layer_security_closure_is_complete_and_source_neutral() {
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store",
|
||||||
|
"v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration",
|
||||||
|
"v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "pre.012 security closure proof missing: {required}");
|
||||||
|
}
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"SourceSnapshotByKey",
|
||||||
|
"SourceHealthByKey",
|
||||||
|
"provider_health",
|
||||||
|
"endpoint_health",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"RawTransactionIngestCanonicalState",
|
||||||
|
"source_key",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.012 public root leaked private/source-specific material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
430
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
Normal file
430
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
Normal file
@@ -0,0 +1,430 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
// version: 22
|
||||||
|
|
||||||
|
//! External public-surface proofs for the RAW transaction ingest Worker foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_kind_code_and_settings_are_consumable_from_crate_root() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE, "raw_transaction_ingest");
|
||||||
|
let kind_result = ksp_worker_api::WorkerKindCode::new(ksp_worker_raw_transaction_ingest_lib::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE);
|
||||||
|
assert!(kind_result.is_ok());
|
||||||
|
let kind = match kind_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(kind.as_str(), "raw_transaction_ingest");
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let settings = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::with_defaults(network, worker_id);
|
||||||
|
assert_eq!(settings.network().as_str(), "mainnet");
|
||||||
|
assert_eq!(settings.worker_id().as_str(), "raw-ingest-mainnet-001");
|
||||||
|
assert_eq!(settings.admission_queue_capacity(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY);
|
||||||
|
assert_eq!(settings.persistence_concurrency(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY);
|
||||||
|
assert_eq!(settings.shutdown_drain_timeout(), ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_invalid_network_and_worker_identity_remain_owned_by_lower_contracts() {
|
||||||
|
let network = ksp_store_lib::RawNetworkId::new("");
|
||||||
|
let worker_id = ksp_worker_api::WorkerId::new("invalid worker id");
|
||||||
|
assert!(network.is_err());
|
||||||
|
assert!(worker_id.is_err());
|
||||||
|
let network_error = match network {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(network_error.code(), ksp_store_lib::ERROR_CODE_RAW_MODEL_INVALID);
|
||||||
|
let worker_error = match worker_id {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(worker_error.code(), ksp_worker_api::ERROR_CODE_WORKER_ID_INVALID);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_runtime_bound_errors_are_stable_and_do_not_echo_values() {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let result = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings::new(network, worker_id, 0, 8, std::time::Duration::from_secs(5));
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert_eq!(error.message(), "invalid RAW transaction ingest Worker settings");
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "admission_queue_capacity");
|
||||||
|
assert!(!std::format!("{error:?}").contains("raw-ingest-mainnet-001"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle() {
|
||||||
|
let _start: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings,
|
||||||
|
std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start;
|
||||||
|
let _request_stop: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle) -> bool =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::request_stop;
|
||||||
|
let _wait_terminal: for<'a> fn(
|
||||||
|
&'a ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestTerminalFuture<'a> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::wait_terminal;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("JoinHandle"));
|
||||||
|
assert!(!root.contains("JoinSet"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_007_store_and_content_conflict_error_codes_are_public_and_stable() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT.code(), "content_conflict");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED.code(), "store_failed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_snapshot_surface_and_common_projection_are_public_and_stable() {
|
||||||
|
let _snapshot_source: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::snapshot_source;
|
||||||
|
let _worker_snapshot_source: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle::worker_snapshot_source;
|
||||||
|
let _current: fn(
|
||||||
|
&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource::current;
|
||||||
|
let _wait: for<'a> fn(
|
||||||
|
&'a ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource,
|
||||||
|
ksp_worker_api::WorkerSnapshotSequence,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotFuture<'a> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource::wait_for_change;
|
||||||
|
fn require_worker_source<T: ksp_worker_api::WorkerSnapshotSource + std::marker::Send + std::marker::Sync>() {}
|
||||||
|
require_worker_source::<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshotSource>();
|
||||||
|
let _state_type = std::any::type_name::<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState>();
|
||||||
|
let _states = [
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Active,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Reconnecting,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Closing,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Closed,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSourceState::Failed,
|
||||||
|
];
|
||||||
|
let _source_state = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_state;
|
||||||
|
let _reconnect = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_reconnect_total;
|
||||||
|
let _replay = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_replay_attempt_total;
|
||||||
|
let _gap = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_continuity_gap_total;
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED.code(), "counter_exhausted");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_source_and_drain_timeout_error_codes_are_public_and_stable() {
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT.code(), "drain_timeout");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED.domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED.code(), "source_failed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_002_runtime_resource_types_are_consumable_without_client_escape_hatch() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::YellowstoneGrpcChannel,
|
||||||
|
ksp_onchain_transport_lib::YellowstoneSubscribeRequest,
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource::new;
|
||||||
|
let _resources_new: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::new;
|
||||||
|
let _start_with_resources: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSettings,
|
||||||
|
std::sync::Arc<ksp_store_lib::Store>,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHandle> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestWorker::start_with_runtime_resources;
|
||||||
|
let source = include_str!("../src/runtime_resources.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub fn channel(",
|
||||||
|
"pub fn http_pool(",
|
||||||
|
"pub fn subscribe_request(",
|
||||||
|
"pub fn hydration_role(",
|
||||||
|
"pub fn enqueue(",
|
||||||
|
"pub fn send(",
|
||||||
|
"pub fn inner(",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "runtime-resource implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_002_multi_source_runtime_resource_surface_is_bounded_and_source_neutral() {
|
||||||
|
let _source_count: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::source_count;
|
||||||
|
let _push_yellowstone: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestYellowstoneSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_yellowstone_source;
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES, 32);
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("RawTransactionIngestLiveSource"));
|
||||||
|
assert!(!root.contains("source_key"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
ksp_onchain_transport_lib::SolanaLogsSubscribeFilter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource::new;
|
||||||
|
let _resources_from_logs: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_logs_source;
|
||||||
|
let _push_logs: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardLogsSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_standard_logs_source;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(root.contains("RawTransactionIngestStandardLogsSource"));
|
||||||
|
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn http_pool(", "pub fn source_key("] {
|
||||||
|
assert!(!resources.contains(forbidden), "standard logs source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_005_helius_transaction_runtime_resource_surface_is_typed_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
ksp_onchain_transport_lib::HeliusTransactionSubscribeFilter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource::new;
|
||||||
|
let _resources_from_helius: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_helius_transaction_source;
|
||||||
|
let _push_helius: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHeliusTransactionSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_helius_transaction_source;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(root.contains("RawTransactionIngestHeliusTransactionSource"));
|
||||||
|
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn http_pool(", "pub fn source_key("] {
|
||||||
|
assert!(!resources.contains(forbidden), "Helius transaction source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_004_standard_block_runtime_resource_surface_is_typed_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::WsEndpointSettings,
|
||||||
|
ksp_onchain_transport_lib::SolanaBlockSubscribeFilter,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource::new;
|
||||||
|
let _resources_from_block: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_standard_block_source;
|
||||||
|
let _push_block: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestStandardBlockSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_standard_block_source;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
assert!(root.contains("RawTransactionIngestStandardBlockSource"));
|
||||||
|
for forbidden in ["pub fn ws_endpoint(", "pub fn filter(", "pub fn source_key("] {
|
||||||
|
assert!(!resources.contains(forbidden), "standard block source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_006_http_block_polling_runtime_resource_surface_is_bounded_and_transport_owned() {
|
||||||
|
let _source_new: fn(
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource::new;
|
||||||
|
let _source_new_with_limits: fn(
|
||||||
|
ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
ksp_onchain_transport_lib::SolanaCommitment,
|
||||||
|
std::time::Duration,
|
||||||
|
u16,
|
||||||
|
) -> ksp_core_lib::Result<ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource::new_with_limits;
|
||||||
|
let _resources_from_polling: fn(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource,
|
||||||
|
) -> ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::from_http_block_polling_source;
|
||||||
|
let _push_polling: fn(
|
||||||
|
&mut ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources,
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestHttpBlockPollingSource,
|
||||||
|
) -> ksp_core_lib::Result<()> = ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestRuntimeResources::try_push_http_block_polling_source;
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL, std::time::Duration::from_secs(1));
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE, 128);
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL, std::time::Duration::from_millis(100));
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL, std::time::Duration::from_secs(30));
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE, 1);
|
||||||
|
assert_eq!(ksp_worker_raw_transaction_ingest_lib::MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE, 1024);
|
||||||
|
let resources = include_str!("../src/runtime_resources.rs");
|
||||||
|
for forbidden in ["pub fn http_pool(", "pub fn polling_role(", "pub fn source_key(", "pub fn profile_fingerprint("] {
|
||||||
|
assert!(!resources.contains(forbidden), "HTTP polling source implementation escape hatch present: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only() {
|
||||||
|
let _hydration_pending: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::hydration_pending;
|
||||||
|
let _processing_frontier_slot: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> std::option::Option<u64> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::processing_frontier_slot;
|
||||||
|
let _oldest_pending_slot: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> std::option::Option<u64> =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::oldest_pending_slot;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in ["ReplayInfo", "from_slot", "repair", "backfill"] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.007 public root overclaims continuity/replay: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_007_source_inventory_and_logical_keys_remain_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in ["RawTransactionIngestSourceInventory", "RawTransactionIngestSourceInventoryPublisher", "RawTransactionIngestLiveSource", "source_key"] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.007 private multi-source implementation leaked through crate root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::persistence::RawTransactionIngestPersistenceConvergence",
|
||||||
|
"pub use self::persistence::persist_raw_transaction_ingest_converged_acquisition",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"RawTransactionIngestHydrationKey",
|
||||||
|
"source_key",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.008 convergence implementation leaked through public crate root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for private_name in [
|
||||||
|
"RawTransactionIngestCanonicalState",
|
||||||
|
"RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"validate_hydration_fairness_capacity",
|
||||||
|
"hydration_pending_limit",
|
||||||
|
"hydration_in_flight_limit",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(&std::format!("pub use self::runtime_resources::{private_name}")));
|
||||||
|
assert!(!root.contains(&std::format!("pub use self::persistence::{private_name}")));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral() {
|
||||||
|
let _source_total: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_total;
|
||||||
|
let _source_active: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_active;
|
||||||
|
let _source_reconnecting: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_reconnecting;
|
||||||
|
let _source_failed: fn(&ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot) -> usize =
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::RawTransactionIngestSnapshot::source_failed;
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in ["SourceHealthByKey", "SourceSnapshotByKey", "provider_health", "endpoint_health"] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.010 public root leaked source-specific health material: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceInventory",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceInventoryPublisher",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestHydrationLeaderGuard",
|
||||||
|
"publish_and_remove",
|
||||||
|
"source.inventory_key_mismatch",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.011 private race-hardening implementation leaked publicly: {forbidden}");
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED,
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "counter_exhausted")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
ksp_worker_raw_transaction_ingest_lib::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT,
|
||||||
|
ksp_core_lib::ErrorCode::new("worker_raw_transaction_ingest", "drain_timeout")
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_completeness_closure_adds_no_public_implementation_surface() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("pub mod "));
|
||||||
|
for forbidden in [
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestLiveSource",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestSourceInventory",
|
||||||
|
"pub use self::runtime_resources::RawTransactionIngestGlobalHydrationRegistry",
|
||||||
|
"pub use self::persistence::RawTransactionIngestCanonicalState",
|
||||||
|
"pub use self::persistence::RawTransactionIngestPersistenceConvergence",
|
||||||
|
] {
|
||||||
|
assert!(!root.contains(forbidden), "pre.012 private implementation leaked through public root: {forbidden}");
|
||||||
|
}
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
assert!(cross_layer.contains("v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline"));
|
||||||
|
assert!(cross_layer.contains("v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,237 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
// version: 20
|
||||||
|
|
||||||
|
//! Release-completeness canaries through the `pre.012` cross-layer completeness/security tranche.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_module_inventory_is_exact() -> std::io::Result<()> {
|
||||||
|
let source_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
||||||
|
let entries = match std::fs::read_dir(source_root) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries {
|
||||||
|
let entry = match entry {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let file_type = match entry.file_type() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if !file_type.is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match entry.file_name().into_string() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => continue,
|
||||||
|
};
|
||||||
|
if name.ends_with(".rs") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
std::vec!["admission.rs", "error.rs", "identity.rs", "lib.rs", "persistence.rs", "runtime.rs", "runtime_resources.rs", "settings.rs", "snapshot.rs",]
|
||||||
|
);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_public_root_export_inventory_is_exact() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
let mut exports = std::vec::Vec::new();
|
||||||
|
for line in root.lines() {
|
||||||
|
let line = line.trim();
|
||||||
|
if !line.starts_with("pub use ") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let path = line.trim_start_matches("pub use ").trim_end_matches(';');
|
||||||
|
let name = match path.rsplit("::").next() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
exports.push(name);
|
||||||
|
}
|
||||||
|
exports.sort_unstable();
|
||||||
|
assert_eq!(
|
||||||
|
exports,
|
||||||
|
std::vec![
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED",
|
||||||
|
"ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_LIVE_SOURCES",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_INTERVAL",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_HTTP_POLL_MAX_BLOCKS_PER_CYCLE",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY",
|
||||||
|
"MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT",
|
||||||
|
"RAW_TRANSACTION_INGEST_WORKER_KIND_CODE",
|
||||||
|
"RawTransactionIngestHandle",
|
||||||
|
"RawTransactionIngestHeliusTransactionSource",
|
||||||
|
"RawTransactionIngestHttpBlockPollingSource",
|
||||||
|
"RawTransactionIngestRuntimeResources",
|
||||||
|
"RawTransactionIngestSettings",
|
||||||
|
"RawTransactionIngestSnapshot",
|
||||||
|
"RawTransactionIngestSnapshotFuture",
|
||||||
|
"RawTransactionIngestSnapshotSource",
|
||||||
|
"RawTransactionIngestSourceState",
|
||||||
|
"RawTransactionIngestStandardBlockSource",
|
||||||
|
"RawTransactionIngestStandardLogsSource",
|
||||||
|
"RawTransactionIngestTerminalFuture",
|
||||||
|
"RawTransactionIngestWorker",
|
||||||
|
"RawTransactionIngestYellowstoneSource",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert!(!root.contains("pub mod "));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
||||||
|
let hardening = include_str!("hardening.rs");
|
||||||
|
for required in [
|
||||||
|
"pre_010_external_error_codes_are_stable_unique_and_domain_scoped",
|
||||||
|
"pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values",
|
||||||
|
"v0_3_12_pre_002_manifest_dependency_surface_opens_only_transport_and_remains_backend_neutral",
|
||||||
|
"v0_3_12_pre_003_private_signal_debug_and_shape_do_not_expose_signature_filters_or_payload",
|
||||||
|
"v0_3_12_pre_004_hydration_provenance_and_remote_material_are_bounded_and_redacted",
|
||||||
|
"v0_3_12_pre_005_block_and_continuity_shapes_are_bounded_redacted_and_raw_separated",
|
||||||
|
"pre_010_source_visibility_contract_uses_crate_root_for_shared_items",
|
||||||
|
"pre_010_production_surface_has_no_historical_backfill_or_retriever_contract",
|
||||||
|
"v0_3_12_pre_002_production_sources_keep_transport_confined_to_runtime_resources",
|
||||||
|
"v0_3_12_pre_006_runtime_resource_contract_opens_one_supervised_transport_source",
|
||||||
|
"v0_3_12_pre_006_source_coalescence_is_bounded_stop_preemptible_and_redacted",
|
||||||
|
"v0_3_12_pre_007_processing_frontier_is_bounded_processing_only_and_redacted",
|
||||||
|
"v0_3_12_pre_008_reconnect_projection_is_source_neutral_bounded_and_contains_no_replay_material",
|
||||||
|
"v0_3_12_pre_009_hydration_retry_ownership_and_no_orphan_cleanup_are_explicit",
|
||||||
|
"pre_010_lower_layers_have_no_dependency_return_to_concrete_worker",
|
||||||
|
"v0_3_12_pre_002_public_root_exposes_contract_types_without_transport_implementation_paths",
|
||||||
|
"v0_3_13_pre_003_standard_logs_redaction_and_reference_only_contract_are_explicit",
|
||||||
|
"v0_3_13_pre_004_standard_block_redaction_version_and_null_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_005_helius_transaction_redaction_full_reference_and_tier_neutrality_are_explicit",
|
||||||
|
"v0_3_13_pre_006_http_block_polling_is_bounded_run_local_stop_preemptible_and_redacted",
|
||||||
|
"v0_3_13_pre_007_multi_source_supervisor_is_fail_closed_joined_and_does_not_publish_source_identity",
|
||||||
|
"v0_3_13_pre_008_cross_source_convergence_is_bounded_conflict_checked_and_private",
|
||||||
|
"v0_3_13_pre_009_duplicate_storm_disagreement_and_starvation_guards_are_explicit",
|
||||||
|
"v0_3_13_pre_010_multi_source_health_is_conservative_counted_and_redacted",
|
||||||
|
"v0_3_13_pre_011_shutdown_races_are_bounded_joined_atomic_and_counter_safe",
|
||||||
|
] {
|
||||||
|
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
|
||||||
|
}
|
||||||
|
let dependency_boundary = include_str!("dependency_boundary.rs");
|
||||||
|
assert!(dependency_boundary.contains("pre_002_manifest_dependency_surface_is_exact"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_006_source_surface_hardens_shutdown_and_faults_without_backend_edges"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_006_productive_source_uses_transport_session_bounded_coalescence_and_existing_admission"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_002_manifest_opens_only_the_onchain_transport_live_source_edge"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_003_transaction_and_status_adapters_are_private_and_transport_facade_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_004_hydration_contract_uses_only_transport_facade_common_raw_and_existing_ingress"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_005_block_and_continuity_adapters_remain_private_and_transport_facade_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_007_processing_frontier_remains_run_local_and_backend_neutral_after_continuity_extension"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_12_pre_008_worker_observes_transport_reconnect_replay_and_faults_only_on_proven_retention_gap"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_003_standard_logs_source_reuses_transport_facades_and_common_hydration_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_004_standard_block_source_is_direct_raw_and_transport_facade_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_005_helius_transaction_source_reuses_transport_facade_and_common_hydration_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_006_http_block_polling_reuses_transport_facades_without_becoming_backfill"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_007_multi_source_supervisor_and_inventory_remain_private_bounded_and_source_neutral"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_008_cross_source_convergence_reuses_store_and_transport_facades_only"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_009_global_bounds_and_fairness_stay_inside_worker_facades"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_010_multi_source_snapshot_health_remains_source_neutral_and_backend_free"));
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_011_shutdown_race_hardening_stays_inside_worker_and_existing_facades"));
|
||||||
|
let public_api = include_str!("public_api.rs");
|
||||||
|
assert!(public_api.contains("pre_003_kind_code_and_settings_are_consumable_from_crate_root"));
|
||||||
|
assert!(public_api.contains("pre_004_start_handle_and_terminal_future_are_consumable_without_public_join_handle"));
|
||||||
|
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
||||||
|
assert!(public_api.contains("pre_009_source_and_drain_timeout_error_codes_are_public_and_stable"));
|
||||||
|
assert!(public_api.contains("v0_3_12_pre_002_runtime_resource_types_are_consumable_without_client_escape_hatch"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_002_multi_source_runtime_resource_surface_is_bounded_and_source_neutral"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_003_standard_logs_runtime_resource_surface_is_typed_and_transport_owned"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_004_standard_block_runtime_resource_surface_is_typed_and_transport_owned"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_006_http_block_polling_runtime_resource_surface_is_bounded_and_transport_owned"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_007_source_inventory_and_logical_keys_remain_private"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_008_convergence_cache_registry_and_source_keys_remain_private"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_009_fairness_budgets_semaphore_and_canonical_state_remain_private"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_010_multi_source_snapshot_counts_are_public_and_source_neutral"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_011_race_hardening_adds_no_public_runtime_or_source_identity_surface"));
|
||||||
|
assert!(public_api.contains("v0_3_12_pre_007_processing_frontier_snapshot_getters_are_public_and_processing_only"));
|
||||||
|
assert!(public_api.contains("pre_008_snapshot_surface_and_common_projection_are_public_and_stable"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_12_pre_010_cross_layer_completeness_suite_is_present_and_exact() -> std::io::Result<()> {
|
||||||
|
let test_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests");
|
||||||
|
let entries = match std::fs::read_dir(test_root) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries {
|
||||||
|
let entry = match entry {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let file_type = match entry.file_type() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if !file_type.is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match entry.file_name().into_string() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => continue,
|
||||||
|
};
|
||||||
|
if name.ends_with(".rs") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(names, std::vec!["cross_layer_completeness.rs", "dependency_boundary.rs", "hardening.rs", "public_api.rs", "release_completeness.rs"]);
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_12_pre_010_dependency_firewall_is_exact_across_transport_worker_common_raw_store",
|
||||||
|
"v0_3_12_pre_010_legacy_v0_fixture_matrix_is_exact_across_transport_worker_and_common_raw",
|
||||||
|
"v0_3_12_pre_010_redaction_scanner_matrix_is_present_across_all_layers",
|
||||||
|
"v0_3_12_pre_010_public_roots_keep_implementation_modules_private",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "required pre.010 cross-layer canary missing: {required}");
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_012_cross_layer_completeness_security_inventory_is_exact() {
|
||||||
|
let cross_layer = include_str!("cross_layer_completeness.rs");
|
||||||
|
for required in [
|
||||||
|
"v0_3_13_pre_012_all_five_live_source_families_share_one_worker_common_raw_store_pipeline",
|
||||||
|
"v0_3_13_pre_012_legacy_v0_v1_is_proven_from_transport_through_worker_common_raw_to_store",
|
||||||
|
"v0_3_13_pre_012_yellowstone_non_regression_covers_identity_v1_reconnect_gap_and_worker_hydration",
|
||||||
|
"v0_3_13_pre_012_security_redaction_matrix_covers_all_live_sources_and_lower_layers",
|
||||||
|
] {
|
||||||
|
assert!(cross_layer.contains(required), "required pre.012 cross-layer canary missing: {required}");
|
||||||
|
}
|
||||||
|
let dependency_boundary = include_str!("dependency_boundary.rs");
|
||||||
|
let hardening = include_str!("hardening.rs");
|
||||||
|
let public_api = include_str!("public_api.rs");
|
||||||
|
assert!(dependency_boundary.contains("v0_3_13_pre_012_cross_layer_closure_does_not_expand_worker_dependency_graph"));
|
||||||
|
assert!(hardening.contains("v0_3_13_pre_012_cross_layer_security_closure_is_complete_and_source_neutral"));
|
||||||
|
assert!(public_api.contains("v0_3_13_pre_012_completeness_closure_adds_no_public_implementation_surface"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,414 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
fn material(
|
||||||
|
network: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
transaction_data: &str,
|
||||||
|
) -> std::option::Option<(ksp_store_lib::RawNetworkId, ksp_raw_transaction_lib::RawTransactionMaterial)> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network.clone(),
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
transaction_data,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
return std::option::Option::Some((network, material));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn provenance() -> std::option::Option<ksp_store_lib::RawAcquisitionProvenance> {
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("admission") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_store_lib::RawAcquisitionProvenance::new(
|
||||||
|
provider,
|
||||||
|
protocol,
|
||||||
|
method,
|
||||||
|
ksp_store_lib::RawAcquisitionOrigin::Live,
|
||||||
|
received_at,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_006_bounded_admission_applies_async_backpressure_until_one_slot_is_consumed() {
|
||||||
|
let (network, first_material) = match material("mainnet", 1, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 2, "BAUG") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [1; 32],
|
||||||
|
};
|
||||||
|
assert!(sender.send(first_ingress).await.is_ok());
|
||||||
|
let second_sender = sender.clone();
|
||||||
|
let second_network = network.clone();
|
||||||
|
let completed = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
|
||||||
|
let completed_task = std::sync::Arc::clone(&completed);
|
||||||
|
let second_task = tokio::spawn(async move {
|
||||||
|
let second_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: second_network,
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [2; 32],
|
||||||
|
};
|
||||||
|
let admitted = second_sender.send(second_ingress).await.is_ok();
|
||||||
|
completed_task.store(admitted, std::sync::atomic::Ordering::Release);
|
||||||
|
return admitted;
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert!(!completed.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
for _ in 0..64 {
|
||||||
|
if completed.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert!(completed.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
let joined = second_task.await;
|
||||||
|
assert!(matches!(joined, std::result::Result::Ok(true)));
|
||||||
|
let second = admission.receive(&network).await;
|
||||||
|
assert!(matches!(second, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_006_stop_preempts_a_blocked_sender_without_silent_post_stop_admission() {
|
||||||
|
let (network, first_material) = match material("mainnet", 3, "BwgJ") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 4, "CgsM") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (stop_sender, stop_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [3; 32],
|
||||||
|
};
|
||||||
|
assert!(sender.send(first_ingress).await.is_ok());
|
||||||
|
let blocked_sender = sender.clone();
|
||||||
|
let blocked_network = network.clone();
|
||||||
|
let blocked_task = tokio::spawn(async move {
|
||||||
|
let mut blocked_stop = stop_receiver;
|
||||||
|
let blocked_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: blocked_network,
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [4; 32],
|
||||||
|
};
|
||||||
|
return tokio::select! {
|
||||||
|
biased;
|
||||||
|
_changed = blocked_stop.changed() => false,
|
||||||
|
result = blocked_sender.send(blocked_ingress) => result.is_ok(),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
stop_sender.send_replace(true);
|
||||||
|
let blocked = blocked_task.await;
|
||||||
|
assert!(matches!(blocked, std::result::Result::Ok(false)));
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
admission.close();
|
||||||
|
let end = admission.receive(&network).await;
|
||||||
|
assert!(matches!(end, std::result::Result::Ok(std::option::Option::None)));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_observation_key_domain_and_common_raw_assembly_golden_are_exact() {
|
||||||
|
let (network, material) = match material("mainnet", 7, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let ingress = crate::RawTransactionIngress { material, network: network.clone(), provenance: provenance.clone(), source_key: [9; 32] };
|
||||||
|
let acquisition = super::canonicalize_ingress(&network, ingress);
|
||||||
|
assert!(acquisition.is_ok());
|
||||||
|
let acquisition = match acquisition {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(acquisition.transaction().reference().network(), &network);
|
||||||
|
assert_eq!(acquisition.transaction().reference().signature().as_bytes(), &[7; 64]);
|
||||||
|
assert_eq!(acquisition.transaction().slot(), 42);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_id().as_str(), ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_version(), ksp_raw_transaction_lib::RAW_TRANSACTION_FORMAT_VERSION);
|
||||||
|
assert_eq!(acquisition.transaction().payload().bytes(), br#"{"transaction":["AQID","base64"]}"#);
|
||||||
|
assert_eq!(acquisition.observation().provenance(), &provenance);
|
||||||
|
assert_eq!(acquisition.observation().transaction(), acquisition.transaction().reference());
|
||||||
|
assert_eq!(
|
||||||
|
acquisition.observation().observation_key().as_bytes(),
|
||||||
|
&[
|
||||||
|
248, 209, 97, 220, 168, 168, 167, 42, 13, 161, 16, 252, 135, 136, 46, 231, 19, 104, 72, 70, 28, 106, 163, 43, 9, 227, 10, 97, 136, 185, 93, 204,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_network_guards_reject_ingress_or_material_mismatch_without_echoing_values() {
|
||||||
|
let (mainnet, mainnet_material) = match material("mainnet", 8, "DQ4P") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (devnet, devnet_material) = match material("devnet", 9, "EBES") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let wrong_ingress = crate::RawTransactionIngress { material: mainnet_material, network: devnet, provenance: first_provenance, source_key: [10; 32] };
|
||||||
|
let wrong_ingress = super::canonicalize_ingress(&mainnet, wrong_ingress);
|
||||||
|
assert!(wrong_ingress.is_err());
|
||||||
|
let wrong_material = crate::RawTransactionIngress {
|
||||||
|
material: devnet_material,
|
||||||
|
network: mainnet.clone(),
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [11; 32],
|
||||||
|
};
|
||||||
|
let wrong_material = super::canonicalize_ingress(&mainnet, wrong_material);
|
||||||
|
assert!(wrong_material.is_err());
|
||||||
|
for result in [wrong_ingress, wrong_material] {
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
let rendered = std::format!("{error:?}");
|
||||||
|
assert!(!rendered.contains("mainnet"));
|
||||||
|
assert!(!rendered.contains("devnet"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_full_queue_dequeue_marks_source_neutral_backpressure_observation() {
|
||||||
|
let (network, material) = match material("mainnet", 13, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
let ingress = crate::RawTransactionIngress { material, network: network.clone(), provenance, source_key: [13; 32] };
|
||||||
|
assert!(sender.send(ingress).await.is_ok());
|
||||||
|
let first = admission.receive(&network).await;
|
||||||
|
assert!(matches!(first, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
assert!(admission.take_backpressure_wait_observed());
|
||||||
|
admission.close();
|
||||||
|
let second = admission.receive(&network).await;
|
||||||
|
assert!(matches!(second, std::result::Result::Ok(std::option::Option::None)));
|
||||||
|
assert!(!admission.take_backpressure_wait_observed());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_008_distinct_source_keys_produce_distinct_idempotent_observation_keys() {
|
||||||
|
let (network, first_material) = match material("mainnet", 51, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, second_material) = match material("mainnet", 51, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(2);
|
||||||
|
let first_ingress = crate::RawTransactionIngress {
|
||||||
|
material: first_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: first_provenance,
|
||||||
|
source_key: [11_u8; 32],
|
||||||
|
};
|
||||||
|
let second_ingress = crate::RawTransactionIngress {
|
||||||
|
material: second_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: second_provenance,
|
||||||
|
source_key: [12_u8; 32],
|
||||||
|
};
|
||||||
|
if sender.send(first_ingress).await.is_err() || sender.send(second_ingress).await.is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
std::mem::drop(sender);
|
||||||
|
let first = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
let second = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.transaction().reference(), second.transaction().reference());
|
||||||
|
assert_eq!(first.transaction().payload().content_hash(), second.transaction().payload().content_hash());
|
||||||
|
assert_ne!(first.observation().observation_key(), second.observation().observation_key());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_009_duplicate_storm_cannot_starve_second_ready_source_on_bounded_admission() {
|
||||||
|
let (network, initial_material) = match material("mainnet", 60, "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, peer_material) = match material("mainnet", 99, "BwgJ") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let initial_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let storm_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let peer_provenance = match provenance() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (mut admission, sender) = crate::RawTransactionAdmission::new(1);
|
||||||
|
assert!(
|
||||||
|
sender
|
||||||
|
.send(crate::RawTransactionIngress {
|
||||||
|
material: initial_material,
|
||||||
|
network: network.clone(),
|
||||||
|
provenance: initial_provenance,
|
||||||
|
source_key: [60_u8; 32],
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.is_ok()
|
||||||
|
);
|
||||||
|
let storm_sender = sender.clone();
|
||||||
|
let storm_network = network.clone();
|
||||||
|
let storm_task = tokio::spawn(async move {
|
||||||
|
for _ in 0..8 {
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
storm_network.clone(),
|
||||||
|
ksp_store_lib::RawTransactionSignature::new([61_u8; 64]),
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
"BAUG",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
if storm_sender
|
||||||
|
.send(crate::RawTransactionIngress {
|
||||||
|
material,
|
||||||
|
network: storm_network.clone(),
|
||||||
|
provenance: storm_provenance.clone(),
|
||||||
|
source_key: [61_u8; 32],
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
let peer_sender = sender.clone();
|
||||||
|
let peer_network = network.clone();
|
||||||
|
let peer_task = tokio::spawn(async move {
|
||||||
|
return peer_sender
|
||||||
|
.send(crate::RawTransactionIngress { material: peer_material, network: peer_network, provenance: peer_provenance, source_key: [99_u8; 32] })
|
||||||
|
.await
|
||||||
|
.is_ok();
|
||||||
|
});
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert_eq!(admission.queue_depth(), 1);
|
||||||
|
let first = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.transaction().reference().signature().as_bytes(), &[60_u8; 64]);
|
||||||
|
let second = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(second.transaction().reference().signature().as_bytes(), &[61_u8; 64]);
|
||||||
|
let third = match admission.receive(&network).await {
|
||||||
|
std::result::Result::Ok(std::option::Option::Some(value)) => value,
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert_eq!(third.transaction().reference().signature().as_bytes(), &[99_u8; 64]);
|
||||||
|
for _ in 0..7 {
|
||||||
|
let next = admission.receive(&network).await;
|
||||||
|
assert!(matches!(next, std::result::Result::Ok(std::option::Option::Some(_))));
|
||||||
|
assert!(admission.queue_depth() <= 1);
|
||||||
|
}
|
||||||
|
let peer_joined = peer_task.await;
|
||||||
|
assert!(matches!(peer_joined, std::result::Result::Ok(true)));
|
||||||
|
let storm_joined = storm_task.await;
|
||||||
|
assert!(matches!(storm_joined, std::result::Result::Ok(true)));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,408 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum PortResponse {
|
||||||
|
Outcome(ksp_store_lib::RawEntityWriteOutcome, ksp_store_lib::RawObservationWriteOutcome),
|
||||||
|
Conflict,
|
||||||
|
StoreFailure,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FakePersistencePort {
|
||||||
|
calls: std::sync::atomic::AtomicUsize,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool,
|
||||||
|
observation_calls: std::sync::atomic::AtomicUsize,
|
||||||
|
observation_response: std::option::Option<ksp_store_lib::RawObservationWriteOutcome>,
|
||||||
|
response: PortResponse,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakePersistencePort {
|
||||||
|
fn new(network: ksp_store_lib::RawNetworkId, response: PortResponse) -> Self {
|
||||||
|
return Self {
|
||||||
|
calls: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
network,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool::new(false),
|
||||||
|
observation_calls: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
observation_response: std::option::Option::None,
|
||||||
|
response,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn with_observation_response(mut self, observation_response: ksp_store_lib::RawObservationWriteOutcome) -> Self {
|
||||||
|
self.observation_response = std::option::Option::Some(observation_response);
|
||||||
|
return self;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for FakePersistencePort {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return &self.network == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
_transaction: ksp_store_lib::RawTransaction,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
self.calls.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
self.normal_mode_seen.store(mode == ksp_store_lib::RawTransactionAcquisitionMode::Normal, std::sync::atomic::Ordering::Release);
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
return match response {
|
||||||
|
PortResponse::Outcome(entity, observation) => std::result::Result::Ok(ksp_store_lib::RawAcquisitionWriteOutcome::new(entity, observation)),
|
||||||
|
PortResponse::Conflict => std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict")),
|
||||||
|
PortResponse::StoreFailure => std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic remote failure text that must not escape",
|
||||||
|
)),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>> {
|
||||||
|
self.observation_calls.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
let observation_response = self.observation_response;
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
if let std::option::Option::Some(outcome) = observation_response {
|
||||||
|
return std::result::Result::Ok(outcome);
|
||||||
|
}
|
||||||
|
return match response {
|
||||||
|
PortResponse::Outcome(_, observation) => std::result::Result::Ok(observation),
|
||||||
|
PortResponse::Conflict => std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict")),
|
||||||
|
PortResponse::StoreFailure => std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic remote failure text that must not escape",
|
||||||
|
)),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn acquisition(network_name: &str, signature_byte: u8) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
return acquisition_with_observation_key(network_name, signature_byte, signature_byte);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn acquisition_with_observation_key(
|
||||||
|
network_name: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
observation_byte: u8,
|
||||||
|
) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
return acquisition_with_shape(network_name, signature_byte, observation_byte, 42, std::option::Option::Some(1_700_000_000), "AQID");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn acquisition_with_shape(
|
||||||
|
network_name: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
observation_byte: u8,
|
||||||
|
slot: u64,
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
transaction_data: &str,
|
||||||
|
) -> std::option::Option<ksp_raw_transaction_lib::RawTransactionAcquisition> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network_name) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network,
|
||||||
|
signature,
|
||||||
|
slot,
|
||||||
|
block_time,
|
||||||
|
transaction_data,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let transaction = match ksp_raw_transaction_lib::canonicalize_raw_transaction(material) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("persistence") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Live, received_at);
|
||||||
|
return std::option::Option::Some(ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(
|
||||||
|
transaction,
|
||||||
|
ksp_store_lib::RawObservationKey::new([observation_byte; 32]),
|
||||||
|
provenance,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn network(name: &str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
return match ksp_store_lib::RawNetworkId::new(name) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_normal_mode_classifies_new_idempotent_and_purged_outcomes() {
|
||||||
|
let cases = [
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::Inserted,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::Inserted,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::AlreadyPresent,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::AlreadyPresent,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::NotRecorded),
|
||||||
|
crate::RawTransactionIngestEntityPersistence::SkippedPurged,
|
||||||
|
crate::RawTransactionIngestObservationPersistence::NotRecorded,
|
||||||
|
),
|
||||||
|
];
|
||||||
|
for (index, (response, expected_entity, expected_observation)) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, response);
|
||||||
|
let acquisition = match acquisition("mainnet", index as u8 + 1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
let outcome = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(outcome.entity(), expected_entity);
|
||||||
|
assert_eq!(outcome.observation(), expected_observation);
|
||||||
|
assert!(port.normal_mode_seen.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_content_conflict_maps_to_terminal_worker_code_without_remote_text() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, PortResponse::Conflict);
|
||||||
|
let acquisition = match acquisition("mainnet", 9) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT);
|
||||||
|
assert!(error.context().is_empty());
|
||||||
|
assert!(!std::format!("{error:?}").contains("synthetic conflict"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_failure_keeps_only_safe_lower_error_code() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, PortResponse::StoreFailure);
|
||||||
|
let acquisition = match acquisition("mainnet", 10) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED);
|
||||||
|
assert_eq!(error.context().len(), 2);
|
||||||
|
assert_eq!(error.context()[0].value(), "synthetic_store");
|
||||||
|
assert_eq!(error.context()[1].value(), "write_failed");
|
||||||
|
assert!(!std::format!("{error:?}").contains("synthetic remote failure text"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_rehydrated_or_impossible_store_outcomes_are_runtime_invalid() {
|
||||||
|
let cases = [
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Rehydrated, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent),
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
];
|
||||||
|
for (index, response) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, response);
|
||||||
|
let acquisition = match acquisition("mainnet", index as u8 + 20) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "persistence.store_outcome_invalid");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_network_mismatch_is_rejected_before_write() {
|
||||||
|
let network = match network("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
);
|
||||||
|
let acquisition = match acquisition("mainnet", 30) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::persist_raw_transaction_ingest_acquisition(&port, acquisition).await;
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context()[0].value(), "persistence.store_network_mismatch");
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_008_same_canonical_identity_uses_atomic_entity_once_then_additional_observation() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
);
|
||||||
|
let convergence = crate::RawTransactionIngestPersistenceConvergence::new(8);
|
||||||
|
let first = match acquisition_with_observation_key("mainnet", 41, 1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second = match acquisition_with_observation_key("mainnet", 41, 2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_outcome = crate::persist_raw_transaction_ingest_converged_acquisition(&port, first, &convergence).await;
|
||||||
|
let second_outcome = crate::persist_raw_transaction_ingest_converged_acquisition(&port, second, &convergence).await;
|
||||||
|
assert!(first_outcome.is_ok());
|
||||||
|
assert!(second_outcome.is_ok());
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(port.observation_calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
let second_outcome = match second_outcome {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(second_outcome.entity(), crate::RawTransactionIngestEntityPersistence::AlreadyPresent);
|
||||||
|
assert_eq!(second_outcome.observation(), crate::RawTransactionIngestObservationPersistence::Inserted);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_008_replayed_same_observation_is_idempotent_without_raw_resubmission() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
)
|
||||||
|
.with_observation_response(ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent);
|
||||||
|
let convergence = crate::RawTransactionIngestPersistenceConvergence::new(8);
|
||||||
|
let first = match acquisition_with_observation_key("mainnet", 42, 3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second = match acquisition_with_observation_key("mainnet", 42, 3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_result = crate::persist_raw_transaction_ingest_converged_acquisition(&port, first, &convergence).await;
|
||||||
|
assert!(first_result.is_ok());
|
||||||
|
let second_result = crate::persist_raw_transaction_ingest_converged_acquisition(&port, second, &convergence).await;
|
||||||
|
let second_outcome = match second_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(port.observation_calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(second_outcome.observation(), crate::RawTransactionIngestObservationPersistence::AlreadyPresent);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_009_source_skew_and_provider_disagreement_fail_before_additional_observation() {
|
||||||
|
let cases = [
|
||||||
|
(43_u64, std::option::Option::Some(1_700_000_000_i64), "AQID"),
|
||||||
|
(42_u64, std::option::Option::Some(1_700_000_001_i64), "AQID"),
|
||||||
|
(42_u64, std::option::Option::Some(1_700_000_000_i64), "BAUG"),
|
||||||
|
];
|
||||||
|
for (index, (slot, block_time, transaction_data)) in cases.into_iter().enumerate() {
|
||||||
|
let network = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(
|
||||||
|
network,
|
||||||
|
PortResponse::Outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
);
|
||||||
|
let convergence = crate::RawTransactionIngestPersistenceConvergence::new(8);
|
||||||
|
let first = match acquisition_with_shape("mainnet", 51, 10, 42, std::option::Option::Some(1_700_000_000), "AQID") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let divergent = match acquisition_with_shape("mainnet", 51, index as u8 + 20, slot, block_time, transaction_data) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(crate::persist_raw_transaction_ingest_converged_acquisition(&port, first, &convergence).await.is_ok());
|
||||||
|
let conflict = crate::persist_raw_transaction_ingest_converged_acquisition(&port, divergent, &convergence).await;
|
||||||
|
let error = match conflict {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT);
|
||||||
|
assert_eq!(port.calls.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert_eq!(port.observation_calls.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,996 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
// version: 11
|
||||||
|
|
||||||
|
struct ActiveTaskGuard {
|
||||||
|
active: std::sync::Arc<std::sync::atomic::AtomicUsize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ActiveTaskGuard {
|
||||||
|
fn new(active: std::sync::Arc<std::sync::atomic::AtomicUsize>) -> Self {
|
||||||
|
active.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return Self { active };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::ops::Drop for ActiveTaskGuard {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings(network: &str) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-runtime-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngestSettings::with_defaults(network, worker_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_active_count(active: &std::sync::Arc<std::sync::atomic::AtomicUsize>, expected: usize) -> bool {
|
||||||
|
for _ in 0..64 {
|
||||||
|
if active.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return active.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_current_runtime_is_required_before_spawn() {
|
||||||
|
let result = super::current_runtime_handle();
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].value(), "start.runtime_unavailable");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_004_store_network_mismatch_is_rejected_without_echoing_network_values() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let other = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let result = super::validate_store_network(&settings, &other);
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID);
|
||||||
|
let rendered = std::format!("{error:?}");
|
||||||
|
assert!(!rendered.contains("mainnet"));
|
||||||
|
assert!(!rendered.contains("devnet"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_immediate_stop_is_idempotent_and_reaches_stopped_terminal() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(handle.snapshot_source().current().worker_snapshot().state(), ksp_worker_api::WorkerState::Starting);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
assert!(terminal.is_ok());
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert!(terminal.is_terminal());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_running_lifecycle_stops_through_cloned_handle() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert_eq!(handle.snapshot_source().current().worker_snapshot().state(), ksp_worker_api::WorkerState::Running);
|
||||||
|
let clone = handle.clone();
|
||||||
|
assert!(clone.request_stop());
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_004_dropping_last_control_handle_causes_private_runtime_exit() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation(settings, tokio::runtime::Handle::current(), std::option::Option::None) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let observer = handle.snapshot_source();
|
||||||
|
std::mem::drop(handle);
|
||||||
|
loop {
|
||||||
|
let current = observer.current();
|
||||||
|
if current.worker_snapshot().state().is_terminal() {
|
||||||
|
assert_eq!(current.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let _changed = observer.wait_for_change(current.worker_snapshot().sequence()).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_supervisor_joins_all_cooperative_source_tasks_before_terminal() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active = std::sync::Arc::clone(&active);
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children, stop_receiver, _admission_sender| {
|
||||||
|
for _ in 0..3 {
|
||||||
|
let active = std::sync::Arc::clone(&source_active);
|
||||||
|
let mut child_stop = stop_receiver.clone();
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let _guard = ActiveTaskGuard::new(active);
|
||||||
|
loop {
|
||||||
|
let changed = child_stop.changed().await;
|
||||||
|
if changed.is_err() || *child_stop.borrow() {
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_active_count(&active, 3).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_005_supervisor_reaps_completed_source_task_and_still_joins_live_child_on_stop() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let completed = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active = std::sync::Arc::clone(&active);
|
||||||
|
let source_completed = std::sync::Arc::clone(&completed);
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children, stop_receiver, _admission_sender| {
|
||||||
|
let completed = std::sync::Arc::clone(&source_completed);
|
||||||
|
let _completed_abort_handle = children.spawn(async move {
|
||||||
|
completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
let active = std::sync::Arc::clone(&source_active);
|
||||||
|
let mut child_stop = stop_receiver.clone();
|
||||||
|
let _active_abort_handle = children.spawn(async move {
|
||||||
|
let _guard = ActiveTaskGuard::new(active);
|
||||||
|
loop {
|
||||||
|
let changed = child_stop.changed().await;
|
||||||
|
if changed.is_err() || *child_stop.borrow() {
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_active_count(&active, 1).await);
|
||||||
|
for _ in 0..64 {
|
||||||
|
if completed.load(std::sync::atomic::Ordering::Acquire) == 1 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum RuntimePortResponse {
|
||||||
|
Conflict,
|
||||||
|
StoreFailure,
|
||||||
|
StoreFailureBlocked,
|
||||||
|
SuccessBlocked,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RuntimePortActiveGuard<'a> {
|
||||||
|
active: &'a std::sync::atomic::AtomicUsize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::ops::Drop for RuntimePortActiveGuard<'_> {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
self.active.fetch_sub(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RuntimePersistencePort {
|
||||||
|
active: std::sync::atomic::AtomicUsize,
|
||||||
|
completed: std::sync::atomic::AtomicUsize,
|
||||||
|
max_active: std::sync::atomic::AtomicUsize,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool,
|
||||||
|
notify: tokio::sync::Notify,
|
||||||
|
released: std::sync::atomic::AtomicBool,
|
||||||
|
response: RuntimePortResponse,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RuntimePersistencePort {
|
||||||
|
fn new(network: ksp_store_lib::RawNetworkId, response: RuntimePortResponse, released: bool) -> Self {
|
||||||
|
return Self {
|
||||||
|
active: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
completed: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
max_active: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
network,
|
||||||
|
normal_mode_seen: std::sync::atomic::AtomicBool::new(false),
|
||||||
|
notify: tokio::sync::Notify::new(),
|
||||||
|
released: std::sync::atomic::AtomicBool::new(released),
|
||||||
|
response,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn release(&self) {
|
||||||
|
self.released.store(true, std::sync::atomic::Ordering::Release);
|
||||||
|
self.notify.notify_waiters();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn update_max_active(&self, current: usize) {
|
||||||
|
let mut observed = self.max_active.load(std::sync::atomic::Ordering::Acquire);
|
||||||
|
while current > observed {
|
||||||
|
match self.max_active.compare_exchange_weak(observed, current, std::sync::atomic::Ordering::AcqRel, std::sync::atomic::Ordering::Acquire) {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => observed = value,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::RawTransactionIngestPersistencePort for RuntimePersistencePort {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return &self.network == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
_transaction: ksp_store_lib::RawTransaction,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
self.normal_mode_seen.store(mode == ksp_store_lib::RawTransactionAcquisitionMode::Normal, std::sync::atomic::Ordering::Release);
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
if matches!(response, RuntimePortResponse::StoreFailureBlocked | RuntimePortResponse::SuccessBlocked) {
|
||||||
|
let current = self.active.fetch_add(1, std::sync::atomic::Ordering::AcqRel) + 1;
|
||||||
|
let _active_guard = RuntimePortActiveGuard { active: &self.active };
|
||||||
|
self.update_max_active(current);
|
||||||
|
while !self.released.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
self.notify.notified().await;
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::SuccessBlocked) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(ksp_store_lib::RawAcquisitionWriteOutcome::new(
|
||||||
|
ksp_store_lib::RawEntityWriteOutcome::Inserted,
|
||||||
|
ksp_store_lib::RawObservationWriteOutcome::Inserted,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::Conflict) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict"));
|
||||||
|
}
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic store failure",
|
||||||
|
));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn record_observation<'a>(
|
||||||
|
&'a self,
|
||||||
|
_observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawObservationWriteOutcome>> {
|
||||||
|
let response = self.response;
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
if matches!(response, RuntimePortResponse::StoreFailureBlocked | RuntimePortResponse::SuccessBlocked) {
|
||||||
|
let current = self.active.fetch_add(1, std::sync::atomic::Ordering::AcqRel) + 1;
|
||||||
|
let _active_guard = RuntimePortActiveGuard { active: &self.active };
|
||||||
|
self.update_max_active(current);
|
||||||
|
while !self.released.load(std::sync::atomic::Ordering::Acquire) {
|
||||||
|
self.notify.notified().await;
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::SuccessBlocked) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Ok(ksp_store_lib::RawObservationWriteOutcome::Inserted);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if matches!(response, RuntimePortResponse::Conflict) {
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "synthetic conflict"));
|
||||||
|
}
|
||||||
|
self.completed.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("synthetic_store", "write_failed"),
|
||||||
|
"synthetic store failure",
|
||||||
|
));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn runtime_ingress(network: &ksp_store_lib::RawNetworkId, signature_byte: u8) -> std::option::Option<crate::RawTransactionIngress> {
|
||||||
|
let signature = ksp_store_lib::RawTransactionSignature::new([signature_byte; 64]);
|
||||||
|
let material = ksp_raw_transaction_lib::RawTransactionMaterial::binary_base64(
|
||||||
|
network.clone(),
|
||||||
|
signature,
|
||||||
|
42,
|
||||||
|
std::option::Option::Some(1_700_000_000),
|
||||||
|
"AQID",
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
ksp_raw_transaction_lib::RawTransactionWireField::Omitted,
|
||||||
|
);
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("deterministic-harness") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("internal") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("runtime") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Live, received_at);
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngress { material, network: network.clone(), provenance, source_key: [signature_byte; 32] });
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings_with_persistence_concurrency(concurrency: usize) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
return settings_with_runtime_limits(8, concurrency, std::time::Duration::from_secs(5));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn settings_with_runtime_limits(
|
||||||
|
admission_queue_capacity: usize,
|
||||||
|
persistence_concurrency: usize,
|
||||||
|
shutdown_drain_timeout: std::time::Duration,
|
||||||
|
) -> std::option::Option<crate::RawTransactionIngestSettings> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-runtime-persistence-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return match crate::RawTransactionIngestSettings::new(network, worker_id, admission_queue_capacity, persistence_concurrency, shutdown_drain_timeout) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_completed(port: &std::sync::Arc<RuntimePersistencePort>, expected: usize) -> bool {
|
||||||
|
for _ in 0..128 {
|
||||||
|
if port.completed.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return port.completed.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_max_active(port: &std::sync::Arc<RuntimePersistencePort>, expected: usize) -> bool {
|
||||||
|
for _ in 0..128 {
|
||||||
|
if port.max_active.load(std::sync::atomic::Ordering::Acquire) == expected {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
return port.max_active.load(std::sync::atomic::Ordering::Acquire) == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_runtime_bounds_in_flight_store_persistence_to_configured_concurrency() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 1..=4 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 2).await);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
assert_eq!(port.max_active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
port.release();
|
||||||
|
assert!(wait_for_completed(&port, 4).await);
|
||||||
|
assert!(port.normal_mode_seen.load(std::sync::atomic::Ordering::Acquire));
|
||||||
|
assert_eq!(port.max_active.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_content_conflict_becomes_terminal_after_private_drain() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::Conflict, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 11) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_CONTENT_CONFLICT));
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 1);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_007_store_failure_becomes_terminal_without_exposing_remote_error_text() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailure, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 12) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 1);
|
||||||
|
assert!(!std::format!("{handle:?}").contains("synthetic store failure"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_runtime_snapshots_count_successful_pipeline_and_retain_terminal() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 21..=23 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let concrete_source = handle.snapshot_source();
|
||||||
|
let common_source = handle.worker_snapshot_source();
|
||||||
|
assert!(wait_for_completed(&port, 3).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
let snapshot = concrete_source.current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
assert_eq!(snapshot.admitted_total(), 3);
|
||||||
|
assert_eq!(snapshot.canonicalized_total(), 3);
|
||||||
|
assert_eq!(snapshot.persisted_total(), 3);
|
||||||
|
assert_eq!(snapshot.entity_inserted_total(), 3);
|
||||||
|
assert_eq!(snapshot.entity_already_present_total(), 0);
|
||||||
|
assert_eq!(snapshot.entity_skipped_purged_total(), 0);
|
||||||
|
assert_eq!(snapshot.observation_inserted_total(), 3);
|
||||||
|
assert_eq!(snapshot.observation_already_present_total(), 0);
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 0);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 0);
|
||||||
|
assert_eq!(snapshot.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&common_source);
|
||||||
|
assert_eq!(&common, snapshot.worker_snapshot());
|
||||||
|
let retained = concrete_source.wait_for_change(snapshot.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(retained, snapshot);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_fault_snapshots_count_classified_store_failures_and_project_unhealthy() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailure, true));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 24) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.source_failed")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = handle.wait_terminal().await;
|
||||||
|
let terminal = match terminal {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
assert_eq!(snapshot.admitted_total(), 1);
|
||||||
|
assert_eq!(snapshot.canonicalized_total(), 1);
|
||||||
|
assert_eq!(snapshot.persisted_total(), 0);
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.content_conflict_total(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>, _stop_receiver, _admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SOURCE_FAILED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let retained = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(retained, terminal);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_stop_does_not_hide_store_failure_observed_during_bounded_drain() {
|
||||||
|
let settings = match settings_with_persistence_concurrency(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::StoreFailureBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 41) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
port.release();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_STORE_FAILED));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.store_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_saturation_is_observable_without_drop_or_unbounded_admission() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, std::time::Duration::from_secs(5)) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_stage = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_stage_for_task = source_stage.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
for signature_byte in 51..=53 {
|
||||||
|
let ingress = match runtime_ingress(&network, signature_byte) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
source_stage_for_task.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
for _ in 0..128 {
|
||||||
|
if source_stage.load(std::sync::atomic::Ordering::Acquire) == 2 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(source_stage.load(std::sync::atomic::Ordering::Acquire), 2);
|
||||||
|
port.release();
|
||||||
|
assert!(wait_for_completed(&port, 3).await);
|
||||||
|
for _ in 0..128 {
|
||||||
|
if source_stage.load(std::sync::atomic::Ordering::Acquire) == 3 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(source_stage.load(std::sync::atomic::Ordering::Acquire), 3);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Stopped);
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.persisted_total(), 3);
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 0);
|
||||||
|
assert!(snapshot.backpressure_wait_total() >= 1);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_009_drain_timeout_aborts_and_joins_all_owned_source_and_persistence_tasks() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active_for_task = source_active.clone();
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 61) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
let _guard = ActiveTaskGuard::new(source_active_for_task);
|
||||||
|
return std::future::pending::<ksp_core_lib::Result<()>>().await;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(wait_for_active_count(&source_active, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
assert_eq!(snapshot.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(source_active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_011_source_counter_exhaustion_stays_terminal_and_is_not_collapsed_to_source_failed() {
|
||||||
|
let settings = match settings("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let handle = match super::start_foundation_with_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::None,
|
||||||
|
move |children: &mut tokio::task::JoinSet<ksp_core_lib::Result<()>>, _stop_receiver, _admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
return std::result::Result::Err(crate::counter_exhausted_error("source_reconnect_total"));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = handle.snapshot_source();
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED));
|
||||||
|
let snapshot = source.current();
|
||||||
|
assert_eq!(snapshot.source_failure_total(), 1);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(snapshot.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
let terminal_sequence = snapshot.worker_snapshot().sequence();
|
||||||
|
assert!(!handle.request_stop());
|
||||||
|
let retained = source.current();
|
||||||
|
assert_eq!(retained.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn v0_3_13_pre_011_drain_timeout_prevents_late_persistence_completion_after_terminal() {
|
||||||
|
let settings = match settings_with_runtime_limits(1, 1, crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = settings.network().clone();
|
||||||
|
let port = std::sync::Arc::new(RuntimePersistencePort::new(network.clone(), RuntimePortResponse::SuccessBlocked, false));
|
||||||
|
let runtime_port: super::PersistencePort = port.clone();
|
||||||
|
let source_active = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||||
|
let source_active_for_task = std::sync::Arc::clone(&source_active);
|
||||||
|
let handle = match super::start_foundation_with_port_and_source_spawner(
|
||||||
|
settings,
|
||||||
|
tokio::runtime::Handle::current(),
|
||||||
|
std::option::Option::Some(runtime_port),
|
||||||
|
move |children, _stop_receiver, admission_sender| {
|
||||||
|
let _abort_handle = children.spawn(async move {
|
||||||
|
let ingress = match runtime_ingress(&network, 81) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(crate::runtime_error("test.ingress_invalid")),
|
||||||
|
};
|
||||||
|
if admission_sender.send(ingress).await.is_err() {
|
||||||
|
return std::result::Result::Err(crate::runtime_error("test.source_failed"));
|
||||||
|
}
|
||||||
|
let _guard = ActiveTaskGuard::new(source_active_for_task);
|
||||||
|
return std::future::pending::<ksp_core_lib::Result<()>>().await;
|
||||||
|
});
|
||||||
|
},
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(wait_for_max_active(&port, 1).await);
|
||||||
|
assert!(wait_for_active_count(&source_active, 1).await);
|
||||||
|
assert!(handle.request_stop());
|
||||||
|
let terminal = match handle.wait_terminal().await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(terminal, ksp_worker_api::WorkerState::Faulted(crate::ERROR_CODE_RAW_TRANSACTION_INGEST_DRAIN_TIMEOUT));
|
||||||
|
let snapshot = handle.snapshot_source().current();
|
||||||
|
let terminal_sequence = snapshot.worker_snapshot().sequence();
|
||||||
|
assert_eq!(snapshot.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(port.active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
assert_eq!(source_active.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
port.release();
|
||||||
|
for _ in 0..64 {
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert_eq!(port.completed.load(std::sync::atomic::Ordering::Acquire), 0);
|
||||||
|
let retained = handle.snapshot_source().current();
|
||||||
|
assert_eq!(retained.worker_snapshot().state(), terminal);
|
||||||
|
assert_eq!(retained.worker_snapshot().sequence(), terminal_sequence);
|
||||||
|
assert_eq!(retained.in_flight_persistence(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,105 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/settings.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn identities() -> std::option::Option<(ksp_store_lib::RawNetworkId, ksp_worker_api::WorkerId)> {
|
||||||
|
let network_result = ksp_store_lib::RawNetworkId::new("mainnet");
|
||||||
|
assert!(network_result.is_ok());
|
||||||
|
let network = match network_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id_result = ksp_worker_api::WorkerId::new("raw-ingest-mainnet-001");
|
||||||
|
assert!(worker_id_result.is_ok());
|
||||||
|
let worker_id = match worker_id_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some((network, worker_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_defaults_are_exact_and_preserve_typed_identity() {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network.clone(), worker_id.clone());
|
||||||
|
assert_eq!(settings.network(), &network);
|
||||||
|
assert_eq!(settings.worker_id(), &worker_id);
|
||||||
|
assert_eq!(settings.admission_queue_capacity(), 256);
|
||||||
|
assert_eq!(settings.persistence_concurrency(), 8);
|
||||||
|
assert_eq!(settings.shutdown_drain_timeout(), std::time::Duration::from_secs(5));
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY, 256);
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY, 8);
|
||||||
|
assert_eq!(crate::DEFAULT_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT, std::time::Duration::from_secs(5));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_inclusive_bounds_accept_exact_minimum_and_maximum() {
|
||||||
|
for (queue, concurrency, drain) in [
|
||||||
|
(
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MIN_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY,
|
||||||
|
crate::MAX_RAW_TRANSACTION_INGEST_SHUTDOWN_DRAIN_TIMEOUT,
|
||||||
|
),
|
||||||
|
] {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::new(network, worker_id, queue, concurrency, drain);
|
||||||
|
assert!(settings.is_ok());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_invalid_runtime_bounds_return_stable_field_scoped_error() {
|
||||||
|
let invalid = [
|
||||||
|
(0, 8, std::time::Duration::from_secs(5), "admission_queue_capacity"),
|
||||||
|
(65_537, 8, std::time::Duration::from_secs(5), "admission_queue_capacity"),
|
||||||
|
(256, 0, std::time::Duration::from_secs(5), "persistence_concurrency"),
|
||||||
|
(256, 65, std::time::Duration::from_secs(5), "persistence_concurrency"),
|
||||||
|
(256, 8, std::time::Duration::from_millis(99), "shutdown_drain_timeout"),
|
||||||
|
(256, 8, std::time::Duration::from_millis(30_001), "shutdown_drain_timeout"),
|
||||||
|
];
|
||||||
|
for (queue, concurrency, drain, field) in invalid {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = crate::RawTransactionIngestSettings::new(network, worker_id, queue, concurrency, drain);
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_SETTINGS_INVALID);
|
||||||
|
assert_eq!(error.code().domain(), "worker_raw_transaction_ingest");
|
||||||
|
assert_eq!(error.code().code(), "settings_invalid");
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "field");
|
||||||
|
assert_eq!(error.context()[0].value(), field);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_debug_keeps_worker_identity_redacted() {
|
||||||
|
let (network, worker_id) = match identities() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network, worker_id);
|
||||||
|
let rendered = std::format!("{settings:?}");
|
||||||
|
assert!(rendered.contains("mainnet"));
|
||||||
|
assert!(rendered.contains("WorkerId(..)"));
|
||||||
|
assert!(!rendered.contains("raw-ingest-mainnet-001"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -0,0 +1,251 @@
|
|||||||
|
// file: crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
fn snapshot_foundation_with_source_total(
|
||||||
|
source_total: usize,
|
||||||
|
) -> std::option::Option<(crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource)> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let worker_id = match ksp_worker_api::WorkerId::new("raw-ingest-snapshot-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let settings = crate::RawTransactionIngestSettings::with_defaults(network, worker_id.clone());
|
||||||
|
let kind = match ksp_worker_api::WorkerKindCode::new(crate::RAW_TRANSACTION_INGEST_WORKER_KIND_CODE) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_worker_api::WorkerLifecycle::new(worker_id, kind);
|
||||||
|
if lifecycle.start().is_err() {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(crate::RawTransactionIngestSnapshotPublisher::new(&settings, &lifecycle, source_total));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn snapshot_foundation() -> std::option::Option<(crate::RawTransactionIngestSnapshotPublisher, crate::RawTransactionIngestSnapshotSource)> {
|
||||||
|
return snapshot_foundation_with_source_total(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_initial_snapshot_and_common_projection_are_exact() {
|
||||||
|
let (publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let _publisher = publisher;
|
||||||
|
let concrete = source.current();
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence().value(), 0);
|
||||||
|
assert_eq!(concrete.worker_snapshot().state(), ksp_worker_api::WorkerState::Starting);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unknown);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Unknown);
|
||||||
|
assert_eq!(concrete.admission_queue_capacity(), crate::DEFAULT_RAW_TRANSACTION_INGEST_ADMISSION_QUEUE_CAPACITY);
|
||||||
|
assert_eq!(concrete.admission_queue_depth(), 0);
|
||||||
|
assert_eq!(concrete.persistence_concurrency(), crate::DEFAULT_RAW_TRANSACTION_INGEST_PERSISTENCE_CONCURRENCY);
|
||||||
|
assert_eq!(concrete.in_flight_persistence(), 0);
|
||||||
|
assert_eq!(concrete.admitted_total(), 0);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 0);
|
||||||
|
assert_eq!(concrete.persisted_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_inserted_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_already_present_total(), 0);
|
||||||
|
assert_eq!(concrete.entity_skipped_purged_total(), 0);
|
||||||
|
assert_eq!(concrete.observation_inserted_total(), 0);
|
||||||
|
assert_eq!(concrete.observation_already_present_total(), 0);
|
||||||
|
assert_eq!(concrete.content_conflict_total(), 0);
|
||||||
|
assert_eq!(concrete.store_failure_total(), 0);
|
||||||
|
assert_eq!(concrete.source_failure_total(), 0);
|
||||||
|
assert_eq!(concrete.backpressure_wait_total(), 0);
|
||||||
|
assert_eq!(concrete.hydration_pending(), 0);
|
||||||
|
assert_eq!(concrete.processing_frontier_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(concrete.oldest_pending_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(concrete.source_state(), std::option::Option::None);
|
||||||
|
assert_eq!(concrete.source_total(), 0);
|
||||||
|
assert_eq!(concrete.source_active(), 0);
|
||||||
|
assert_eq!(concrete.source_reconnecting(), 0);
|
||||||
|
assert_eq!(concrete.source_failed(), 0);
|
||||||
|
assert_eq!(concrete.source_reconnect_total(), 0);
|
||||||
|
assert_eq!(concrete.source_replay_attempt_total(), 0);
|
||||||
|
assert_eq!(concrete.source_continuity_gap_total(), 0);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
assert_eq!(&common, concrete.worker_snapshot());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_checked_counter_exhaustion_is_stable_and_never_wraps() {
|
||||||
|
let result = super::checked_counter(u64::MAX, "admitted_total");
|
||||||
|
assert!(result.is_err());
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_RAW_TRANSACTION_INGEST_COUNTER_EXHAUSTED);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "field");
|
||||||
|
assert_eq!(error.context()[0].value(), "admitted_total");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_slow_concrete_and_common_listeners_coalesce_to_same_latest_sequence() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let observed = source.current().worker_snapshot().sequence();
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.record_admission_success(ksp_worker_api::WorkerState::Running, 1, 1, false).is_ok());
|
||||||
|
let concrete = source.wait_for_change(observed).await;
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, observed).await;
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence(), common.sequence());
|
||||||
|
assert_eq!(concrete.worker_snapshot().sequence().value(), 2);
|
||||||
|
assert_eq!(concrete.worker_snapshot().state(), ksp_worker_api::WorkerState::Running);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
assert_eq!(concrete.admission_queue_depth(), 1);
|
||||||
|
assert_eq!(concrete.in_flight_persistence(), 1);
|
||||||
|
assert_eq!(concrete.admitted_total(), 1);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 1);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn pre_008_terminal_latest_value_is_retained_after_publisher_drop() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Stopping, 0, 0).is_ok());
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Stopped, 0, 0).is_ok());
|
||||||
|
std::mem::drop(publisher);
|
||||||
|
let retained = source.current();
|
||||||
|
assert_eq!(retained.worker_snapshot().state(), ksp_worker_api::WorkerState::Stopped);
|
||||||
|
assert_eq!(retained.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(retained.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
let same_terminal = source.wait_for_change(retained.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(same_terminal, retained);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::wait_for_change(&source, retained.worker_snapshot().sequence()).await;
|
||||||
|
assert_eq!(common, retained.worker_snapshot().clone());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_source_failure_and_backpressure_counters_advance_without_changing_projection_contract() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(publisher.publish_state(ksp_worker_api::WorkerState::Running, 0, 0).is_ok());
|
||||||
|
assert!(publisher.record_admission_success(ksp_worker_api::WorkerState::Running, 0, 1, true).is_ok());
|
||||||
|
assert!(publisher.record_source_failure(ksp_worker_api::WorkerState::Running, 0, 1).is_ok());
|
||||||
|
let concrete = source.current();
|
||||||
|
assert_eq!(concrete.admitted_total(), 1);
|
||||||
|
assert_eq!(concrete.canonicalized_total(), 1);
|
||||||
|
assert_eq!(concrete.backpressure_wait_total(), 1);
|
||||||
|
assert_eq!(concrete.source_failure_total(), 1);
|
||||||
|
assert_eq!(concrete.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(concrete.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let common = ksp_worker_api::WorkerSnapshotSource::current(&source);
|
||||||
|
assert_eq!(&common, concrete.worker_snapshot());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_007_processing_frontier_projection_is_latest_value_and_activity_aware() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let initial = source.current();
|
||||||
|
assert_eq!(initial.hydration_pending(), 0);
|
||||||
|
assert_eq!(initial.processing_frontier_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(initial.oldest_pending_slot(), std::option::Option::None);
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(40), std::option::Option::Some(41));
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let pending = source.current();
|
||||||
|
assert_eq!(pending.hydration_pending(), 2);
|
||||||
|
assert_eq!(pending.processing_frontier_slot(), std::option::Option::Some(40));
|
||||||
|
assert_eq!(pending.oldest_pending_slot(), std::option::Option::Some(41));
|
||||||
|
assert_eq!(pending.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(45), std::option::Option::None);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let settled = source.current();
|
||||||
|
assert_eq!(settled.hydration_pending(), 0);
|
||||||
|
assert_eq!(settled.processing_frontier_slot(), std::option::Option::Some(45));
|
||||||
|
assert_eq!(settled.oldest_pending_slot(), std::option::Option::None);
|
||||||
|
assert_eq!(settled.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_source_continuity_projection_preserves_processing_frontier_and_distinguishes_replay() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(2, std::option::Option::Some(40), std::option::Option::Some(41))
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting), 0, 1, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let reconnecting = source.current();
|
||||||
|
assert_eq!(reconnecting.hydration_pending(), 2);
|
||||||
|
assert_eq!(reconnecting.processing_frontier_slot(), std::option::Option::Some(40));
|
||||||
|
assert_eq!(reconnecting.oldest_pending_slot(), std::option::Option::Some(41));
|
||||||
|
assert_eq!(reconnecting.source_state(), std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting));
|
||||||
|
assert_eq!(reconnecting.source_reconnect_total(), 0);
|
||||||
|
assert_eq!(reconnecting.source_replay_attempt_total(), 1);
|
||||||
|
assert_eq!(reconnecting.source_continuity_gap_total(), 0);
|
||||||
|
assert_eq!(reconnecting.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let projection = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(45), std::option::Option::None)
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 1, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, projection).is_ok());
|
||||||
|
let resumed = source.current();
|
||||||
|
assert_eq!(resumed.hydration_pending(), 0);
|
||||||
|
assert_eq!(resumed.processing_frontier_slot(), std::option::Option::Some(45));
|
||||||
|
assert_eq!(resumed.source_state(), std::option::Option::Some(crate::RawTransactionIngestSourceState::Active));
|
||||||
|
assert_eq!(resumed.source_reconnect_total(), 1);
|
||||||
|
assert_eq!(resumed.source_replay_attempt_total(), 1);
|
||||||
|
assert_eq!(resumed.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_13_pre_010_multi_source_counts_and_health_are_conservative_and_source_neutral() {
|
||||||
|
let (mut publisher, source) = match snapshot_foundation_with_source_total(3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let initial = source.current();
|
||||||
|
assert_eq!(initial.source_total(), 3);
|
||||||
|
assert_eq!(initial.source_active(), 0);
|
||||||
|
assert_eq!(initial.source_reconnecting(), 0);
|
||||||
|
assert_eq!(initial.source_failed(), 0);
|
||||||
|
assert_eq!(initial.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unknown);
|
||||||
|
let reconnecting = crate::RawTransactionIngestProcessingFrontierProjection::new(1, std::option::Option::Some(50), std::option::Option::Some(50))
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Reconnecting), 0, 1, 0)
|
||||||
|
.with_source_counts(3, 2, 1, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, reconnecting).is_ok());
|
||||||
|
let degraded = source.current();
|
||||||
|
assert_eq!(degraded.source_total(), 3);
|
||||||
|
assert_eq!(degraded.source_active(), 2);
|
||||||
|
assert_eq!(degraded.source_reconnecting(), 1);
|
||||||
|
assert_eq!(degraded.source_failed(), 0);
|
||||||
|
assert_eq!(degraded.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Degraded);
|
||||||
|
assert_eq!(degraded.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Active);
|
||||||
|
let active = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(51), std::option::Option::None)
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Active), 1, 1, 0)
|
||||||
|
.with_source_counts(3, 3, 0, 0);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, active).is_ok());
|
||||||
|
let healthy = source.current();
|
||||||
|
assert_eq!(healthy.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Healthy);
|
||||||
|
assert_eq!(healthy.worker_snapshot().activity(), ksp_worker_api::WorkerActivity::Idle);
|
||||||
|
let failed = crate::RawTransactionIngestProcessingFrontierProjection::new(0, std::option::Option::Some(51), std::option::Option::None)
|
||||||
|
.with_source_continuity(std::option::Option::Some(crate::RawTransactionIngestSourceState::Failed), 1, 1, 0)
|
||||||
|
.with_source_counts(3, 2, 0, 1);
|
||||||
|
assert!(publisher.record_processing_frontier(ksp_worker_api::WorkerState::Running, 0, 0, failed).is_ok());
|
||||||
|
let unhealthy = source.current();
|
||||||
|
assert_eq!(unhealthy.source_failed(), 1);
|
||||||
|
assert_eq!(unhealthy.worker_snapshot().health(), ksp_worker_api::WorkerHealth::Unhealthy);
|
||||||
|
return;
|
||||||
|
}
|
||||||
263
deltas/0.3.10/pre.001.md
Normal file
263
deltas/0.3.10/pre.001.md
Normal file
@@ -0,0 +1,263 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta 0.3.10-pre.001 — audit/sizing RAW Transaction + Worker ingest
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
archive : khadhroony-solana-project-v0.3.9.zip
|
||||||
|
SHA-256 : 41ef2589a11410aabfb329d387f0008e116006da60ada0802e5f72bc30d3466f
|
||||||
|
ZIP bytes : 7859118
|
||||||
|
ZIP entries : 1855
|
||||||
|
unzip -t : PASS
|
||||||
|
workspace members : 19
|
||||||
|
workspace.package.version base : 0.3.9
|
||||||
|
```
|
||||||
|
|
||||||
|
Le ZIP a été contrôlé sans path traversal/entrée absolue, symlink, `.git/`, `target/`, `node_modules/`, `Cargo.lock`, `.env` ou fichier de clé privée détecté.
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-general-0.3.10-pre.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le type `general` est obligatoire parce que `Cargo.toml` racine est synchronisé vers `0.3.10-pre.1` par `VER-ID-009`. L’archive d’échange ne doit contenir que les fichiers ajoutés/modifiés et le delta, conformément à `VER-ARCHIVE-004`.
|
||||||
|
|
||||||
|
## Baseline exécutée avant modification
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (332 table(s), 750 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
`cargo` n’est pas installé dans l’environnement d’assemblage. Aucun gate Cargo local n’est déclaré PASS. Le journal opérateur stable `0.3.9` reste une preuve externe distincte.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Exécuter le gate d’ouverture `0.3.10-pre.001` demandé par le prompt `029` : vérifier intégralement la base `v0.3.9`, auditer l’extraction RAW commune et les capabilities live, réévaluer `TR-B..TR-F`, fixer les frontières Common/Worker/Transport/Config/Store, puis dimensionner les tranches suivantes sans commencer l’implémentation lourde.
|
||||||
|
|
||||||
|
## Mission exécutée
|
||||||
|
|
||||||
|
`pre.001` reste une tranche de planification, conformément au prompt `029` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
lecture règles/architecture obligatoires
|
||||||
|
inventaire workspace/manifests/features
|
||||||
|
extraction RAW commune auditée
|
||||||
|
capability matrix Worker implementation-ready
|
||||||
|
TR-B..TR-F réévalués
|
||||||
|
fraîcheur Solana/Yellowstone/providers revalidée
|
||||||
|
runtime/source supervision décidés
|
||||||
|
continuité/gap repair décidés
|
||||||
|
snapshot concret décidé
|
||||||
|
Config/secrets boundary décidée
|
||||||
|
threat model et preuves définis
|
||||||
|
dependency graph cible fixé
|
||||||
|
prereleases redimensionnées
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune implémentation lourde des deux nouvelles crates n’est commencée.
|
||||||
|
|
||||||
|
## Décisions principales
|
||||||
|
|
||||||
|
### Common RAW
|
||||||
|
|
||||||
|
```text
|
||||||
|
crate : ksp-raw-transaction-lib
|
||||||
|
RAW format : ksp.solana.raw_transaction v1 inchangé
|
||||||
|
common deps : ksp-core-lib + ksp-store-lib(default-features=false) + serde_json + sha2
|
||||||
|
aucun Transport/Config/Job/Worker/runtime/backend
|
||||||
|
RawObservationKey reste producer-owned
|
||||||
|
Backfill conserve son observation key/job_id/scope_fingerprint/origin
|
||||||
|
```
|
||||||
|
|
||||||
|
Golden confirmé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
byte_len = 112
|
||||||
|
sha256 = 220792d2b15d262fda242cb220774ee9ddeffebf04dcfadabcf8ef76a9b1a7c3
|
||||||
|
```
|
||||||
|
|
||||||
|
Toute source full doit prouver une parité byte-for-byte avec RAW v1 avant persistence directe ; sinon elle reste discovery + hydration.
|
||||||
|
|
||||||
|
### Worker concret
|
||||||
|
|
||||||
|
```text
|
||||||
|
crate : ksp-worker-raw-transaction-ingest-lib
|
||||||
|
caller-owned Tokio runtime
|
||||||
|
Worker-owned source/supervisor tasks
|
||||||
|
bounded channels uniquement
|
||||||
|
multi-source simultané
|
||||||
|
Store idempotence comme vérité de convergence
|
||||||
|
observations distinctes conservées
|
||||||
|
content conflict terminal
|
||||||
|
repair borné au run frontier
|
||||||
|
reconnect != replay
|
||||||
|
aucun paramètre historique métier au start
|
||||||
|
```
|
||||||
|
|
||||||
|
### Transport gaps
|
||||||
|
|
||||||
|
```text
|
||||||
|
TR-B confirmé : get_block_observed
|
||||||
|
TR-C confirmé/scindé : common material + adapters Worker + Transport change only if DTO lacks fidelity
|
||||||
|
TR-D confirmé : provenance durable full-material + source runtime snapshot
|
||||||
|
TR-E confirmé : reuse from_slot/replay info, aucun second Yellowstone client
|
||||||
|
TR-F réservé : EARLY seulement si protocole/accès/rôle RAW prouvés
|
||||||
|
```
|
||||||
|
|
||||||
|
### Config
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun edge Worker -> Config
|
||||||
|
std.transport V3 actuel réutilisé
|
||||||
|
pas de nouveau document métier Worker imposé en 0.3.10
|
||||||
|
adaptation Config seulement si un manque Transport concret l’exige
|
||||||
|
secrets restent Config-owned et redacted
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fraîcheur externe
|
||||||
|
|
||||||
|
Vérifié le 5 septembre 2026 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
yellowstone-grpc-proto latest = 12.7.0 (2026-08-29)
|
||||||
|
workspace stable = ^12.7 -> aucun bump initial
|
||||||
|
Yellowstone block from_slot replay a reçu un fix upstream en juillet 2026 -> smoke block replay obligatoire
|
||||||
|
Helius transactionSubscribe courant ; Helius blockSubscribe explicitement non supporté
|
||||||
|
Helius LaserStream replay 24 h documenté
|
||||||
|
PublicNode Yellowstone Mainnet/Testnet visible ; replay depth non prouvée
|
||||||
|
OrbitFlare gRPC Devnet disponible Free/Developer
|
||||||
|
QuickNode gRPC Scale+ ; port 443/x-token ; port 10000 sunset 2026-10-01
|
||||||
|
Alchemy replay docs contradictoires 6000 vs 432000 slots -> aucune constante KSP
|
||||||
|
Jito ShredStream shutdown annoncé pour 2026-09-05 -> SUNSET, non retenu TR-F
|
||||||
|
```
|
||||||
|
|
||||||
|
## Plan recalibré
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 common RAW foundation
|
||||||
|
pre.003 Backfill migration
|
||||||
|
pre.004 observed block + HTTP block material
|
||||||
|
pre.005 WS/Helius full parity
|
||||||
|
pre.006 Yellowstone parity
|
||||||
|
pre.007 Worker runtime foundation
|
||||||
|
pre.008 Yellowstone live/replay
|
||||||
|
pre.009 WS/HTTP live
|
||||||
|
pre.010 multi-source persistence/hardening
|
||||||
|
pre.011 gap repair
|
||||||
|
pre.012 Config/profiles + smokes accessibles
|
||||||
|
pre.013 EARLY si prouvé, sinon consolidation
|
||||||
|
pre.014 gate technique/live final
|
||||||
|
pre.015 réconciliation documentaire
|
||||||
|
pre.016 préparation publication
|
||||||
|
rel.001 stable
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
deltas/0.3.10/pre.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
`Cargo.toml` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
file version 489 -> 490
|
||||||
|
workspace.package.version 0.3.9 -> 0.3.10-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce bump est requis par `VER-ID-009` pour toute prerelease non-fix, même lorsque la tranche est documentaire.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation après modification
|
||||||
|
|
||||||
|
Audits statiques réellement exécutés après matérialisation de `pre.001` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (339 table(s), 753 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
Le premier passage Markdown post-modification a détecté 32 erreurs d’alignement uniquement dans les nouvelles tables. Elles ont été corrigées selon la règle KSP, puis le gate complet ci-dessus a été relancé jusqu’à PASS.
|
||||||
|
|
||||||
|
La comparaison byte-à-byte contre le contenu du ZIP stable, après suppression du cache Python d’audit, donne exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ajoutés :
|
||||||
|
deltas/0.3.10/pre.001.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
modifié :
|
||||||
|
Cargo.toml
|
||||||
|
supprimé :
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
Les commandes Cargo/Tauri ne sont pas exécutables dans l’environnement d’assemblage courant (`cargo` absent). Elles restent donc explicitement `NON EXÉCUTÉES LOCALEMENT` pour ce delta ; aucune preuve opérateur antérieure n’est transformée en PASS local.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo tree --duplicates
|
||||||
|
builds Tauri
|
||||||
|
smokes live opt-in
|
||||||
|
```
|
||||||
|
|
||||||
|
Motif : toolchain Cargo absent du sandbox d’assemblage et aucune credential/provider live n’est injectée. Les preuves stables fournies par l’opérateur sont conservées comme contexte externe distinct ; elles ne remplacent pas les gates futurs de `0.3.10`.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question de boundary ne bloque `pre.002`. Restent volontairement à résoudre par preuve pendant les tranches prévues :
|
||||||
|
|
||||||
|
```text
|
||||||
|
parité byte-for-byte RAW v1 des sources WS/Helius/Yellowstone full
|
||||||
|
profondeur replay réellement observable par provider au moment du smoke
|
||||||
|
nécessité éventuelle d’un profil/rôle Transport supplémentaire pour un smoke accessible
|
||||||
|
admission d’une source EARLY seulement si protocole, accès et rôle RAW sont effectivement prouvés
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces points ne doivent pas être anticipés par des constantes, des claims lossless ou une RAW v2.
|
||||||
|
|
||||||
|
## Non inclus
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune nouvelle crate créée
|
||||||
|
aucun Rust modifié
|
||||||
|
aucun Transport/Config/Store code modifié
|
||||||
|
aucun kbot3 ouvert ou copié
|
||||||
|
aucun Worker live
|
||||||
|
aucun RAW v2
|
||||||
|
aucun CHANGELOG/ROADMAP modifié
|
||||||
|
aucun prompt 0.3.11 créé
|
||||||
|
aucun Cargo gate déclaré localement
|
||||||
|
```
|
||||||
138
deltas/0.3.10/pre.002-fix.001.md
Normal file
138
deltas/0.3.10/pre.002-fix.001.md
Normal file
@@ -0,0 +1,138 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.002-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.002-fix.001` — correction des canaris d’intégration RAW et du gate Clippy
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.10-pre.002
|
||||||
|
workspace.package.version = 0.3.10-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le delta historique `deltas/0.3.10/pre.002.md` reste inchangé. Le présent fix trace séparément les défauts révélés par la validation opérateur du 2026-09-06.
|
||||||
|
|
||||||
|
## Résultat opérateur de la base
|
||||||
|
|
||||||
|
La validation communiquée confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all : exécuté
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (339 table(s), 754 file(s))
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo test -p ksp-raw-transaction-lib : PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les tests de `ksp-raw-transaction-lib` passent fonctionnellement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
9 tests unitaires : PASS
|
||||||
|
8 tests d’intégration : PASS
|
||||||
|
0 échec
|
||||||
|
```
|
||||||
|
|
||||||
|
Les graphes communiqués confirment la frontière prévue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-store-api
|
||||||
|
serde_json
|
||||||
|
sha2
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun runtime, Transport, Config, Job, Worker ou backend Store n’apparaît dans le graphe normal de la common crate.
|
||||||
|
|
||||||
|
## Motif du fix
|
||||||
|
|
||||||
|
Le gate strict :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
```
|
||||||
|
|
||||||
|
échoue exclusivement sur les nouveaux tests d’intégration de `ksp-raw-transaction-lib`.
|
||||||
|
|
||||||
|
### Rustdoc de crate manquante
|
||||||
|
|
||||||
|
Quatre fichiers `tests/*.rs` n’avaient pas de documentation de crate :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/security_hardening.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
Avec `missing_docs` promu en erreur par `-D warnings`, chaque binaire de test d’intégration doit documenter sa crate.
|
||||||
|
|
||||||
|
Correction : ajout d’une rustdoc `//! ...` conforme aux autres tests d’intégration KSP. Aucun `#[allow(missing_docs)]` n’est ajouté.
|
||||||
|
|
||||||
|
### `clippy::collapsible_if`
|
||||||
|
|
||||||
|
Le canari d’inventaire des modules production utilisait deux conditions imbriquées pour filtrer les fichiers `.rs` puis extraire leur nom.
|
||||||
|
|
||||||
|
Correction : filtrage anticipé avec `continue`, puis `let ... else`, sans `#[allow(clippy::collapsible_if)]` et sans changement de l’inventaire attendu.
|
||||||
|
|
||||||
|
## Version Cargo
|
||||||
|
|
||||||
|
Ce fix modifie du Rust de test. Le signal technique Cargo est donc synchronisé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.2 -> 0.3.10-pre.2.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le header du `Cargo.toml` racine passe de `491` à `492`.
|
||||||
|
|
||||||
|
## Invariants volontairement inchangés
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_FORMAT_ID = ksp.solana.raw_transaction
|
||||||
|
RAW_TRANSACTION_FORMAT_VERSION = 1
|
||||||
|
golden byte_len = 112
|
||||||
|
golden sha256 = 220792d2b15d262fda242cb220774ee9ddeffebf04dcfadabcf8ef76a9b1a7c3
|
||||||
|
RawObservationKey = producer-owned
|
||||||
|
dépendances common = core + store-api + serde_json + sha2
|
||||||
|
aucune migration Backfill
|
||||||
|
aucun Worker
|
||||||
|
aucune nouvelle feature
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/security_hardening.rs
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.002-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validation à rejouer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-raw-transaction-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.003` reste bloquée tant que ce gate n’est pas intégralement vert.
|
||||||
261
deltas/0.3.10/pre.002.md
Normal file
261
deltas/0.3.10/pre.002.md
Normal file
@@ -0,0 +1,261 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.002` — fondation RAW Transaction commune
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.001
|
||||||
|
workspace.package.version = 0.3.10-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.001` passe `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et Clippy workspace `--all-targets --all-features -- -D warnings`. Le message ne fournit pas de `cargo test` supplémentaire ; aucun test non communiqué n’est déclaré exécuté.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Créer uniquement `ksp-raw-transaction-lib` comme lower-layer commune de canonicalisation RAW Transaction v1 : matériau source-neutral, parser de signature, canonical JSON, SHA-256 et assemblage transaction/observation. Aucun Worker et aucune migration du Backfill dans cette tranche.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `Cargo.toml` racine passe du header `490` au header `491` et ajoute `crates/ksp-raw-transaction-lib` aux membres du workspace.
|
||||||
|
|
||||||
|
## Correction normative du plan `pre.001`
|
||||||
|
|
||||||
|
Le sizing `pre.001` avait retenu `ksp-store-lib(default-features=false)` pour la common crate. La relecture au moment de matérialiser la frontière révèle une règle plus spécifique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
DEP-PIPE-006
|
||||||
|
Le pipeline raw ingestion peut dépendre des modèles homogènes de
|
||||||
|
ksp-onchain-transport-lib et de ksp-store-api, mais pas de ksp-store-lib.
|
||||||
|
```
|
||||||
|
|
||||||
|
La dépendance est donc corrigée vers `ksp-store-api`. Cette correction ne change aucun invariant RAW ; elle retire au contraire la façade/runtime Store de la lower-layer réutilisable.
|
||||||
|
|
||||||
|
## Dépendances exactes
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-store-api
|
||||||
|
-> serde_json
|
||||||
|
-> sha2
|
||||||
|
```
|
||||||
|
|
||||||
|
Absents de la crate :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-store-lib
|
||||||
|
ksp-store-postgres-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-config-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
ksp-worker-api
|
||||||
|
Tokio / futures
|
||||||
|
provider SDK
|
||||||
|
```
|
||||||
|
|
||||||
|
## Surface publique
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_FORMAT_ID = ksp.solana.raw_transaction
|
||||||
|
RAW_TRANSACTION_FORMAT_VERSION = 1
|
||||||
|
MIN_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES = 64
|
||||||
|
MAX_RAW_TRANSACTION_SIGNATURE_TEXT_BYTES = 88
|
||||||
|
|
||||||
|
RawTransactionMaterial
|
||||||
|
RawTransactionWireField<T>
|
||||||
|
RawTransactionVersion
|
||||||
|
RawTransactionAcquisition
|
||||||
|
|
||||||
|
parse_raw_transaction_signature(...)
|
||||||
|
canonicalize_raw_transaction(...)
|
||||||
|
assemble_raw_transaction_acquisition(...)
|
||||||
|
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_CANONICALIZATION_INVALID
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_MATERIAL_INVALID
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_SIGNATURE_INVALID
|
||||||
|
```
|
||||||
|
|
||||||
|
Tous les modules restent privés et les éléments consommables sont réexportés au crate-root.
|
||||||
|
|
||||||
|
## Contrat matérialisé
|
||||||
|
|
||||||
|
### Signature
|
||||||
|
|
||||||
|
Le parser reçoit un texte Base58 borné de `64..=88` octets et n’accepte que les représentations décodant exactement vers `64` octets. Aucun Solana SDK ou codec externe n’est ajouté.
|
||||||
|
|
||||||
|
Les erreurs ne recopient jamais la signature hostile ; seules les longueurs numériques sûres peuvent apparaître en contexte.
|
||||||
|
|
||||||
|
### Matériau source-neutral
|
||||||
|
|
||||||
|
`RawTransactionMaterial::binary_base64(...)` reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
signature canonique 64 bytes
|
||||||
|
slot u64
|
||||||
|
block_time Option<i64>
|
||||||
|
transaction data binaire Base64 déjà obtenu de la source
|
||||||
|
meta omitted/null/value
|
||||||
|
version omitted/null/value (legacy ou u8)
|
||||||
|
transactionIndex omitted/null/value u32
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun type Transport/provider n’entre dans la common crate. La représentation interne peut être étendue ultérieurement pour Yellowstone uniquement lorsqu’une parité RAW v1 sera démontrée.
|
||||||
|
|
||||||
|
### Canonicalisation RAW v1
|
||||||
|
|
||||||
|
La sortie conserve exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ordre top-level : transaction -> meta -> version -> transactionIndex
|
||||||
|
transaction = [data, "base64"]
|
||||||
|
tri récursif des clés d’objets JSON
|
||||||
|
ordre des arrays conservé
|
||||||
|
omitted != null != value
|
||||||
|
block_time hors payload, converti en RawTimestamp
|
||||||
|
SHA-256 sur les bytes canoniques exacts
|
||||||
|
format id/version gelés
|
||||||
|
```
|
||||||
|
|
||||||
|
Golden verrouillé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
bytes = {"transaction":["AQID","base64"],"meta":{"a":{"x":null,"y":true},"z":1},"version":"legacy","transactionIndex":7}
|
||||||
|
byte_len = 112
|
||||||
|
sha256 = 220792d2b15d262fda242cb220774ee9ddeffebf04dcfadabcf8ef76a9b1a7c3
|
||||||
|
```
|
||||||
|
|
||||||
|
### Observation producer-owned
|
||||||
|
|
||||||
|
`assemble_raw_transaction_acquisition(...)` reçoit la `RawObservationKey` déjà choisie par le producteur et construit l’observation sur la référence exacte de la transaction. Aucun domaine de clé Backfill ou Worker n’est introduit dans common.
|
||||||
|
|
||||||
|
## Tests ajoutés
|
||||||
|
|
||||||
|
Tests unitaires :
|
||||||
|
|
||||||
|
```text
|
||||||
|
signature exacte 64 bytes
|
||||||
|
signature bornes / caractère Base58 invalide / overflow / longueur décodée non canonique
|
||||||
|
golden bytes + hash RAW v1
|
||||||
|
omitted/null/value
|
||||||
|
golden JSON récursif / arrays / escaping / nombres
|
||||||
|
block_time négatif / overflow / maximum
|
||||||
|
borne payload
|
||||||
|
Debug material/wire redacted
|
||||||
|
assemblage acquisition + observation key producer-owned
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests d’intégration :
|
||||||
|
|
||||||
|
```text
|
||||||
|
dependency boundary exacte
|
||||||
|
absence runtime/Transport/Config/Job/Worker/backend
|
||||||
|
consommation crate-root externe
|
||||||
|
assemblage avec modèles Store API
|
||||||
|
inventaire modules production
|
||||||
|
absence de contrat Worker/Backfill/Transport dans la façade
|
||||||
|
error codes stables
|
||||||
|
signature hostile non recopiée
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-raw-transaction-lib/Cargo.toml
|
||||||
|
crates/ksp-raw-transaction-lib/src/acquisition.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/canonical.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/error.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/lib.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/signature.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/acquisition.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/canonical.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/signature.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/security_hardening.rs
|
||||||
|
deltas/0.3.10/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Documentation volontairement différée
|
||||||
|
|
||||||
|
`README.md` et `USAGE.md` de la nouvelle crate ne sont pas finalisés dans cette tranche de fondation. La réconciliation version-neutral reste réservée au couloir documentaire `pre.015`, après parité cross-source, migration Backfill et freeze fonctionnelle.
|
||||||
|
|
||||||
|
## Validations exécutées dans l’environnement d’assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (339 table(s), 754 file(s))
|
||||||
|
|
||||||
|
contrôle TOML : 20 membres workspace, nouvelle crate présente
|
||||||
|
contrôle manifest common : dépendances exactes core/store-api/serde_json/sha2, aucune feature/dev/build dependency
|
||||||
|
```
|
||||||
|
|
||||||
|
L’environnement d’assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.002` n’est donc déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-raw-transaction-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-raw-transaction-lib -e features
|
||||||
|
smokes live : non pertinents pour cette tranche purement déterministe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW v1 reste unique et inchangé
|
||||||
|
ksp-store-api remplace ksp-store-lib dans common pour conformité DEP-PIPE-006
|
||||||
|
RawObservationKey reste producer-owned
|
||||||
|
aucun type Transport dans common
|
||||||
|
aucun async/runtime dans common
|
||||||
|
aucune migration Backfill avant pre.003
|
||||||
|
aucun Worker avant pre.007 selon le plan recalibré
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.003` après un gate opérateur vert. Les projections structurées WS/Helius/Yellowstone restent volontairement différées jusqu’aux tranches de parité prévues ; elles ne doivent pas modifier RAW v1 par anticipation.
|
||||||
236
deltas/0.3.10/pre.003.md
Normal file
236
deltas/0.3.10/pre.003.md
Normal file
@@ -0,0 +1,236 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.003` — migration du Backfill vers la common RAW Transaction
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.002-fix.001
|
||||||
|
workspace.package.version = 0.3.10-pre.2.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué le 2026-09-06 ferme intégralement la base : audits statiques propres, `cargo check --workspace` PASS, Clippy workspace `--all-targets --all-features -- -D warnings` PASS, `cargo test -p ksp-raw-transaction-lib` PASS avec 9 tests unitaires et 8 tests d’intégration, arbres normal/features conformes.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Migrer uniquement `ksp-job-backfill-lib` vers `ksp-raw-transaction-lib` sans modifier le comportement de campagne : common devient propriétaire du parser Base58, de RAW v1, du hash payload et de l’assemblage transaction/observation ; Backfill conserve Transport, provenance, observation key, Job scope et Store persistence.
|
||||||
|
|
||||||
|
Aucun Worker, aucune nouvelle source Transport et aucune extension du format RAW ne sont introduits.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le header du `Cargo.toml` racine passe de `492` à `493`. Le nombre de membres workspace reste `20`.
|
||||||
|
|
||||||
|
## Migration de dépendance
|
||||||
|
|
||||||
|
`ksp-job-backfill-lib` ajoute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib = { path = "../ksp-raw-transaction-lib" }
|
||||||
|
```
|
||||||
|
|
||||||
|
Le reste de son graphe runtime est conservé. `serde_json` reste nécessaire au type Transport `SolanaWireField<serde_json::Value>`. `sha2` reste nécessaire aux fingerprints et à la `RawObservationKey` Backfill ; il n’est plus utilisé pour le hash du payload RAW.
|
||||||
|
|
||||||
|
## Chemin RAW après migration
|
||||||
|
|
||||||
|
```text
|
||||||
|
BackfillSignature
|
||||||
|
-> parse_raw_transaction_signature
|
||||||
|
-> RawTransactionSignature
|
||||||
|
|
||||||
|
getTransaction observed
|
||||||
|
-> adapter SolanaEncodedTransaction/SolanaWireField
|
||||||
|
-> RawTransactionMaterial::binary_base64
|
||||||
|
-> canonicalize_raw_transaction
|
||||||
|
-> RawTransaction
|
||||||
|
|
||||||
|
Backfill provenance + Backfill RawObservationKey
|
||||||
|
-> assemble_raw_transaction_acquisition
|
||||||
|
-> BackfillRawAcquisition
|
||||||
|
-> Store persistence existante
|
||||||
|
```
|
||||||
|
|
||||||
|
Les erreurs du parser/canonicalizer common sont remappées au boundary Backfill vers :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Job ne change donc pas de domaine d’erreur externe par effet de la migration.
|
||||||
|
|
||||||
|
## Code privé supprimé du Backfill
|
||||||
|
|
||||||
|
La migration retire de `conversion.rs` les implémentations locales de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
base58_digit(...)
|
||||||
|
algorithme de décodage Base58 64-byte
|
||||||
|
convert_block_time(...)
|
||||||
|
canonical_payload_bytes(...)
|
||||||
|
append_wire_value(...)
|
||||||
|
append_canonical_json(...)
|
||||||
|
append_json_string(...)
|
||||||
|
RawPayload::try_new + SHA-256 du payload
|
||||||
|
construction directe RawTransactionObservation
|
||||||
|
```
|
||||||
|
|
||||||
|
`CanonicalTransactionFields` reste un adapter Transport privé ; il ne définit aucun format RAW.
|
||||||
|
|
||||||
|
## Propriété Backfill conservée
|
||||||
|
|
||||||
|
Le Backfill continue de posséder :
|
||||||
|
|
||||||
|
```text
|
||||||
|
getTransaction observed / rôle HTTP
|
||||||
|
provider gagnant
|
||||||
|
endpoint gagnant
|
||||||
|
commitment
|
||||||
|
job_id
|
||||||
|
scope_fingerprint
|
||||||
|
RawAcquisitionOrigin::Backfill
|
||||||
|
capture_session
|
||||||
|
observation contract version = 1
|
||||||
|
algorithme RawObservationKey Backfill
|
||||||
|
persistence Store
|
||||||
|
sémantique Missing
|
||||||
|
```
|
||||||
|
|
||||||
|
La common crate ne reçoit donc aucun vocabulaire de campagne.
|
||||||
|
|
||||||
|
## Parité déterministe
|
||||||
|
|
||||||
|
Le golden historique reste inchangé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
format_id = ksp.solana.raw_transaction
|
||||||
|
format_version = 1
|
||||||
|
bytes = {"transaction":["AQID","base64"],"meta":{"a":{"x":null,"y":true},"z":1},"version":"legacy","transactionIndex":7}
|
||||||
|
byte_len = 112
|
||||||
|
sha256 = 220792d2b15d262fda242cb220774ee9ddeffebf04dcfadabcf8ef76a9b1a7c3
|
||||||
|
```
|
||||||
|
|
||||||
|
La provenance historique reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
provider = publicnode
|
||||||
|
protocol = solana.http.json_rpc
|
||||||
|
method = getTransaction
|
||||||
|
endpoint = devnet-primary
|
||||||
|
commitment = finalized
|
||||||
|
capture_session = backfill:pre006
|
||||||
|
origin = Backfill
|
||||||
|
```
|
||||||
|
|
||||||
|
Le canari d’observation key est désormais exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
provider = provider
|
||||||
|
endpoint = endpoint-a
|
||||||
|
key = b8550f0f2165f370de918bd4fb0ec782391afdddb88c57f010743e000b136702
|
||||||
|
```
|
||||||
|
|
||||||
|
## Canaris mis à jour
|
||||||
|
|
||||||
|
```text
|
||||||
|
BackfillSignature délègue au parser common
|
||||||
|
bornes Backfill == bornes common
|
||||||
|
format id/version Backfill == common
|
||||||
|
wire omitted/null reste distinct via common
|
||||||
|
non-Base64 reste rejeté avec error code Backfill
|
||||||
|
block_time invalide reste terminal avec error code Backfill
|
||||||
|
golden bytes/hash/provenance inchangés
|
||||||
|
observation key exacte inchangée
|
||||||
|
manifest Backfill exige l’edge common
|
||||||
|
conversion.rs exige parser + canonicalizer + assembler common
|
||||||
|
conversion.rs interdit canonical JSON/Base58/RawPayload privés
|
||||||
|
lower-layer common interdit toute dépendance retour vers Job
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/README.md
|
||||||
|
crates/ksp-job-backfill-lib/src/conversion.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/request.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/hardening.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/conversion.rs
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l’environnement d’assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (339 table(s), 756 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L’environnement d’assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
smokes live : non pertinents pour cette tranche de migration déterministe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
common possède parser + canonical RAW v1 + payload hash + assembly
|
||||||
|
Backfill conserve observation key/provenance/campagne
|
||||||
|
les erreurs common sont remappées vers le code Backfill historique
|
||||||
|
les constantes publiques RAW du Backfill sont des reexports common
|
||||||
|
les bornes publiques de signature Backfill suivent les constantes common
|
||||||
|
aucun changement du Store persistence path
|
||||||
|
aucun changement de source/acquisition réseau
|
||||||
|
aucun Worker avant la tranche planifiée
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque le gate de `pre.003`. `pre.004` reste strictement dédiée à `get_block_observed` et au matériau HTTP block après validation opérateur complète de cette migration.
|
||||||
165
deltas/0.3.10/pre.004-fix.001.md
Normal file
165
deltas/0.3.10/pre.004-fix.001.md
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.004-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.004-fix.001` — correction du canari HTTP block et du gate Clippy
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.10-pre.004
|
||||||
|
workspace.package.version = 0.3.10-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le delta historique `deltas/0.3.10/pre.004.md` reste inchangé. Le présent fix trace séparément le défaut révélé par la validation opérateur du 2026-09-06.
|
||||||
|
|
||||||
|
## Résultat opérateur de la base
|
||||||
|
|
||||||
|
La validation communiquée confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all : exécuté
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (339 table(s), 757 file(s))
|
||||||
|
cargo check --workspace : PASS
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : FAIL
|
||||||
|
cargo test -p ksp-raw-transaction-lib : PASS
|
||||||
|
cargo test -p ksp-onchain-transport-lib : PASS
|
||||||
|
cargo test -p ksp-job-backfill-lib --test http_block_material : PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les tests fonctionnels communiqués confirment notamment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib : 12 unit + 9 intégration, 0 échec
|
||||||
|
ksp-onchain-transport-lib : 387 unit + 51 public API + 43 release completeness + 4 doctests, 0 échec
|
||||||
|
http_block_material : 1 test, PASS
|
||||||
|
smokes live Transport : ignorés comme prévu
|
||||||
|
```
|
||||||
|
|
||||||
|
Les arbres normaux communiqués conservent les frontières prévues : la common RAW dépend de `base64`, `ksp-core-lib`, `ksp-store-api`, `serde_json`, `sha2`, et `ksp-onchain-transport-lib` ne dépend pas de `ksp-raw-transaction-lib`.
|
||||||
|
|
||||||
|
## Motif du fix
|
||||||
|
|
||||||
|
Le gate strict :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
```
|
||||||
|
|
||||||
|
échoue exclusivement sur sept `clippy::expect_used` dans les deux helpers privés du nouveau test d’intégration :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pool_for_url(...)
|
||||||
|
serve_once(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Les usages concernés sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
parsing HttpEndpointUrl
|
||||||
|
construction HttpTransportPool
|
||||||
|
bind du listener local
|
||||||
|
lecture de local_addr
|
||||||
|
accept de la connexion fixture
|
||||||
|
read de la requête fixture
|
||||||
|
write de la réponse fixture
|
||||||
|
```
|
||||||
|
|
||||||
|
La configuration commune autorise `expect` dans les fonctions de test, mais ces helpers ne portent pas eux-mêmes l’attribut `#[test]`/`#[tokio::test]` et sont donc correctement rejetés par Clippy.
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
Aucun lint n’est désactivé.
|
||||||
|
|
||||||
|
`pool_for_url(...)` retourne désormais explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp_core_lib::Result<HttpTransportPool>
|
||||||
|
```
|
||||||
|
|
||||||
|
et propage les deux erreurs possibles par `match` explicite.
|
||||||
|
|
||||||
|
`serve_once(...)` retourne désormais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
std::io::Result<(String, JoinHandle<std::io::Result<()>>)>
|
||||||
|
```
|
||||||
|
|
||||||
|
Le bind, `local_addr`, `accept`, `read` et `write_all` retournent leurs erreurs par contrôle de flux explicite. Les assertions/`expect()` de fixture restent uniquement dans le corps `#[tokio::test]`, conformément à `allow-expect-in-tests = true`.
|
||||||
|
|
||||||
|
## Version Cargo
|
||||||
|
|
||||||
|
Ce fix modifie du Rust de test. Le signal technique Cargo est donc synchronisé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.4 -> 0.3.10-pre.4.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le header du `Cargo.toml` racine passe de `494` à `495`.
|
||||||
|
|
||||||
|
## Invariants volontairement inchangés
|
||||||
|
|
||||||
|
```text
|
||||||
|
get_block_observed : inchangé
|
||||||
|
winner provider/endpoint : inchangé
|
||||||
|
RAW_TRANSACTION_FORMAT_ID = ksp.solana.raw_transaction
|
||||||
|
RAW_TRANSACTION_FORMAT_VERSION = 1
|
||||||
|
extraction Base64/short_vec : inchangée
|
||||||
|
projection 2 transactions / index 0-1 : inchangée
|
||||||
|
TR-C2 : adapter productif toujours Worker-owned
|
||||||
|
aucun changement de code production
|
||||||
|
aucune dépendance ou feature ajoutée
|
||||||
|
aucun nouveau Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/tests/http_block_material.rs
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.004-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l’environnement d’assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (339 table(s), 758 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L’environnement d’assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`; aucun gate Cargo du fix n’est donc déclaré PASS localement.
|
||||||
|
|
||||||
|
## Validation à rejouer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib --test http_block_material
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-onchain-transport-lib --edges normal
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.005` reste bloquée tant que ce gate n’est pas intégralement vert.
|
||||||
279
deltas/0.3.10/pre.004.md
Normal file
279
deltas/0.3.10/pre.004.md
Normal file
@@ -0,0 +1,279 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.004` — HTTP observed block + matériau RAW block
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.003
|
||||||
|
workspace.package.version = 0.3.10-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué le 2026-09-06 ferme intégralement `pre.003` : audits statiques propres, `cargo check --workspace` PASS, Clippy workspace `--all-targets --all-features -- -D warnings` PASS, tests de `ksp-raw-transaction-lib` PASS, tests de `ksp-job-backfill-lib` PASS avec 51 tests unitaires et 20 tests d’intégration, arbres normal/features conformes.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer `TR-B` et la preuve HTTP block de `TR-C` sans ouvrir le Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
getBlock pool HTTP
|
||||||
|
-> valeur observée + provider/endpoint winner
|
||||||
|
-> SolanaConfirmedBlock
|
||||||
|
-> N SolanaBlockTransaction full Base64
|
||||||
|
-> matériau RAW source-neutral
|
||||||
|
```
|
||||||
|
|
||||||
|
La tranche ne crée aucun adapter productif Transport -> common. Conformément à `TR-C2`, cet adapter reste producer-owned et sera matérialisé dans le futur `ksp-worker-raw-transaction-ingest-lib`.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le header du `Cargo.toml` racine passe de `493` à `494`. Le nombre de membres workspace reste `20`.
|
||||||
|
|
||||||
|
## `get_block_observed`
|
||||||
|
|
||||||
|
`ksp-onchain-transport-lib` ajoute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HttpTransportPool::get_block_observed(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le wrapper :
|
||||||
|
|
||||||
|
```text
|
||||||
|
réutilise exactement la validation/config moderne de get_block
|
||||||
|
utilise le routing/admission/timeout/retry standard du pool
|
||||||
|
retourne HttpObservedValue<Option<SolanaConfirmedBlock>>
|
||||||
|
conserve endpoint_name + provider du winner réel après retry/reroute
|
||||||
|
préserve result = null comme None
|
||||||
|
n’expose ni URL, ni headers, ni raw HTTP body
|
||||||
|
réutilise le décodage typé SolanaConfirmedBlock
|
||||||
|
```
|
||||||
|
|
||||||
|
`get_block(...)` et `get_block_observed(...)` partagent désormais le même helper privé de paramètres et la même validation du commitment `processed`.
|
||||||
|
|
||||||
|
## Provenance winner déterministe
|
||||||
|
|
||||||
|
Le canari Transport configure deux endpoints :
|
||||||
|
|
||||||
|
```text
|
||||||
|
endpoint 1 -> HTTP 429
|
||||||
|
endpoint 2 -> getBlock success
|
||||||
|
```
|
||||||
|
|
||||||
|
La valeur observée doit reporter exclusivement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
endpoint_name = winner-endpoint
|
||||||
|
provider = winner-provider
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Debug conserve uniquement la projection sûre et ne rend pas le transaction wire Base64.
|
||||||
|
|
||||||
|
## Matériau Base64 source-neutral
|
||||||
|
|
||||||
|
`ksp-raw-transaction-lib` ajoute `base64` à son graphe common et expose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
extract_raw_transaction_signature_from_binary_base64(...)
|
||||||
|
RawTransactionMaterial::binary_base64_with_embedded_signature(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
L’extracteur lit directement le wire transaction Solana Base64 sans dépendance à une crate transaction runtime :
|
||||||
|
|
||||||
|
```text
|
||||||
|
borne la représentation textuelle avant décodage
|
||||||
|
exige Base64 STANDARD canonique
|
||||||
|
lit un short_vec de signature count borné et canonique
|
||||||
|
refuse count = 0
|
||||||
|
refuse short_vec tronqué/non canonique
|
||||||
|
refuse tableau de signatures tronqué
|
||||||
|
exige au moins un byte de message après les signatures déclarées
|
||||||
|
retourne uniquement la première signature exacte de 64 octets
|
||||||
|
ne copie jamais le transaction wire dans erreur/Debug
|
||||||
|
```
|
||||||
|
|
||||||
|
Le constructeur common réutilise cette signature puis conserve les dimensions RAW v1 existantes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
slot
|
||||||
|
block_time
|
||||||
|
transaction Base64
|
||||||
|
meta omitted/null/value
|
||||||
|
version omitted/null/legacy/number
|
||||||
|
transaction_index
|
||||||
|
```
|
||||||
|
|
||||||
|
Le format canonique reste strictement `ksp.solana.raw_transaction` v1.
|
||||||
|
|
||||||
|
## Preuve cross-layer sans nouveau couplage de production
|
||||||
|
|
||||||
|
Le nouveau canari test-only :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/tests/http_block_material.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
utilise une crate qui dépend déjà de Transport et de common pour prouver la composition future sans créer d’edge architectural supplémentaire :
|
||||||
|
|
||||||
|
```text
|
||||||
|
get_block_observed
|
||||||
|
-> SolanaConfirmedBlock
|
||||||
|
-> 2 SolanaBlockTransaction distinctes
|
||||||
|
-> full Base64 + meta + version + transactionIndex 0/1
|
||||||
|
-> RawTransactionMaterial::binary_base64_with_embedded_signature
|
||||||
|
-> canonicalize_raw_transaction
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deux transaction wires minimales font `66` octets chacune et utilisent des signatures distinctes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
transaction 0 -> signature [0; 64]
|
||||||
|
transaction 1 -> signature [1; 64]
|
||||||
|
```
|
||||||
|
|
||||||
|
Le canari vérifie que ces signatures et les index 0/1 restent associés à la bonne transaction, avec le slot et le block time communs du bloc.
|
||||||
|
|
||||||
|
Cette preuve reste strictement test-only. Aucun symbole `ksp_raw_transaction_lib` n’est ajouté aux sources de production de `ksp-onchain-transport-lib`.
|
||||||
|
|
||||||
|
## Dépendances common
|
||||||
|
|
||||||
|
Le graphe runtime direct de `ksp-raw-transaction-lib` devient exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
base64
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-store-api
|
||||||
|
serde_json
|
||||||
|
sha2
|
||||||
|
```
|
||||||
|
|
||||||
|
Restent interdits :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-store-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-config-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
ksp-worker-api
|
||||||
|
tokio
|
||||||
|
futures
|
||||||
|
```
|
||||||
|
|
||||||
|
La règle `DEP-PIPE-006` reste donc respectée.
|
||||||
|
|
||||||
|
## Documentation durable
|
||||||
|
|
||||||
|
`crates/ksp-onchain-transport-lib/USAGE.md` documente la différence entre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
get_block(...)
|
||||||
|
get_block_observed(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
sans transformer le guide en journal de prerelease.
|
||||||
|
|
||||||
|
`docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md` ferme `TR-B`, conserve `TR-C2` producer-owned et corrige le graphe common vers `ksp-store-api`.
|
||||||
|
|
||||||
|
Le plan `031` et la validation `027` enregistrent également la fermeture opérateur de `pre.003` et le périmètre exact de `pre.004`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/tests/http_block_material.rs
|
||||||
|
crates/ksp-onchain-transport-lib/fixtures/http/get_block.observed_material.success.json
|
||||||
|
deltas/0.3.10/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/USAGE.md
|
||||||
|
crates/ksp-onchain-transport-lib/src/rpc_blocks.rs
|
||||||
|
crates/ksp-onchain-transport-lib/tests/public_api.rs
|
||||||
|
crates/ksp-onchain-transport-lib/unit_tests/rpc_blocks.rs
|
||||||
|
crates/ksp-raw-transaction-lib/Cargo.toml
|
||||||
|
crates/ksp-raw-transaction-lib/src/canonical.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/error.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/lib.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/signature.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/canonical.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/signature.rs
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l’environnement d’assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (339 table(s), 757 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L’environnement d’assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib --test http_block_material
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-onchain-transport-lib --edges normal
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib --test http_block_material
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-onchain-transport-lib --edges normal
|
||||||
|
smokes live : non requis pour cette tranche déterministe
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
TR-B est fermé par get_block_observed
|
||||||
|
get_block et get_block_observed partagent validation + décodage
|
||||||
|
la common extrait la signature d’un transaction wire Base64 complet
|
||||||
|
aucune dépendance Solana transaction runtime n’est ajoutée à la common
|
||||||
|
aucun adapter productif Transport -> common dans Transport
|
||||||
|
TR-C2 reste Worker-owned
|
||||||
|
la preuve block -> common est cross-layer et test-only
|
||||||
|
RAW v1 reste inchangé
|
||||||
|
aucun Worker live n’est ouvert dans pre.004
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque le gate de `pre.004`. `pre.005` reste dédiée à la parité WS `blockSubscribe` + Helius `transactionSubscribe`, avec golden parity ou fallback hydration explicitement qualifié selon la représentation réellement disponible.
|
||||||
92
deltas/0.3.10/pre.005-fix.001.md
Normal file
92
deltas/0.3.10/pre.005-fix.001.md
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.005-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.005-fix.001` — correction de typage du canari WS RAW parity
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.005
|
||||||
|
```
|
||||||
|
|
||||||
|
## Motif du fix
|
||||||
|
|
||||||
|
Le premier rejeu opérateur de `pre.005` valide les audits statiques, `cargo check --workspace`, la common RAW, la suite Transport complète et les arbres Cargo, mais Clippy strict et la suite Backfill s’arrêtent sur une unique erreur de compilation `E0308` dans `tests/ws_raw_parity.rs`.
|
||||||
|
|
||||||
|
Le canari comparait :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransaction::block_time().unix_millis() -> u64
|
||||||
|
FIXTURE_BLOCK_TIME * 1_000 -> i64
|
||||||
|
```
|
||||||
|
|
||||||
|
L’assertion produisait donc `Option<u64> != Option<i64>` au niveau du type.
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
La constante de fixture devient directement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
FIXTURE_BLOCK_TIME_MILLIS: u64 = 1_787_072_400_000
|
||||||
|
```
|
||||||
|
|
||||||
|
L’assertion compare désormais `Option<u64>` à `Option<u64>` sans cast, conversion runtime, `unwrap`, `expect` ou modification du comportement testé.
|
||||||
|
|
||||||
|
## Invariants inchangés
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockSubscribe standard confirmed/full/base64 legacy-v0 -> golden RAW-direct positif
|
||||||
|
Helius transactionSubscribe full/base64 -> signal + hydration HTTP
|
||||||
|
RAW v1 bytes/hash contracts
|
||||||
|
TR-C2 Worker-owned
|
||||||
|
tokio-tungstenite strictement dev-only
|
||||||
|
aucun code de production modifié
|
||||||
|
aucune dépendance ni feature modifiée
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version technique
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.5.fix.1
|
||||||
|
Cargo.toml header = 497
|
||||||
|
workspace members = 20
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.005-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audits d’assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (339 table(s), 760 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur à rejouer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Les arbres Cargo n’ont pas besoin d’être rejoués pour ce fix : ni `Cargo.toml` de crate, ni dépendance, ni feature ne change.
|
||||||
101
deltas/0.3.10/pre.005-fix.002.md
Normal file
101
deltas/0.3.10/pre.005-fix.002.md
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.005-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.005-fix.002` — conformité Clippy du canari WS + inventaire dev exact
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.005-fix.001
|
||||||
|
```
|
||||||
|
|
||||||
|
## Motif du fix
|
||||||
|
|
||||||
|
Le second rejeu opérateur confirme `cargo check --workspace`, la common RAW et la suite Transport complète, mais révèle deux défauts test-only :
|
||||||
|
|
||||||
|
```text
|
||||||
|
25 clippy::question_mark_used dans crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs
|
||||||
|
1 canari hardening encore calibré sur dev-dependencies = {tokio}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le manifest `pre.005` contient légitimement `tokio-tungstenite` en dev-only pour les serveurs WebSocket déterministes ; `tests/dependency_boundary.rs` verrouille déjà qu’il ne migre pas vers les dépendances de production.
|
||||||
|
|
||||||
|
## Corrections
|
||||||
|
|
||||||
|
`ws_raw_parity.rs` remplace tous les `?` par du contrôle de flux explicite :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Option -> match + retour None
|
||||||
|
Result -> match + retour Err/None
|
||||||
|
appel async Result -> if let Err(error) + retour Err(error)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `#[allow(...)]`, aucun changement de golden et aucune modification du scénario positif `blockSubscribe` ou négatif Helius ne sont introduits.
|
||||||
|
|
||||||
|
`hardening.rs` attend désormais exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
dev-dependencies = {tokio, tokio-tungstenite}
|
||||||
|
```
|
||||||
|
|
||||||
|
L’inventaire des dépendances normales reste inchangé.
|
||||||
|
|
||||||
|
## Invariants inchangés
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockSubscribe standard confirmed/full/base64 legacy-v0 -> golden RAW-direct positif
|
||||||
|
Helius transactionSubscribe full/base64 -> signal + hydration HTTP
|
||||||
|
RAW v1 bytes/hash contracts
|
||||||
|
TR-C2 Worker-owned
|
||||||
|
tokio-tungstenite strictement dev-only
|
||||||
|
aucun code de production modifié
|
||||||
|
aucune dépendance ni feature modifiée par ce fix
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version technique
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.5.fix.2
|
||||||
|
Cargo.toml header = 498
|
||||||
|
workspace members = 20
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/tests/hardening.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.005-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audits d’assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (339 table(s), 761 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur à rejouer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Les arbres Cargo n’ont pas besoin d’être rejoués pour ce fix : aucun manifest de crate, aucune dépendance et aucune feature ne change.
|
||||||
232
deltas/0.3.10/pre.005.md
Normal file
232
deltas/0.3.10/pre.005.md
Normal file
@@ -0,0 +1,232 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.005` — qualification RAW WS standard et Helius
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.10-pre.004-fix.001
|
||||||
|
workspace.package.version = 0.3.10-pre.4.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rejeu opérateur communiqué le 2026-09-06 ferme intégralement le fix : audits statiques propres, `cargo check --workspace` PASS, Clippy strict PASS, tests common PASS, 387 tests Transport plus suites publiques/release/doctests PASS, et suite complète Backfill PASS. Les arbres Cargo n'ont pas été rejoués sur le fix car aucune dépendance ni feature n'y avait changé.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Qualifier les deux surfaces WebSocket déjà présentes sans ouvrir le Worker :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana standard blockSubscribe full/base64
|
||||||
|
-> prouver ou refuser la production RAW directe
|
||||||
|
|
||||||
|
Helius transactionSubscribe full/base64
|
||||||
|
-> prouver ou refuser la production RAW directe
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun adapter productif n'est placé dans Transport ou Backfill. `TR-C2` reste Worker-owned.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Le header du `Cargo.toml` racine passe de `495` à `496`. Le workspace reste à `20` membres.
|
||||||
|
|
||||||
|
## Qualification positive — `blockSubscribe`
|
||||||
|
|
||||||
|
Le canari cross-layer `crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs` monte un serveur HTTP local et un serveur WebSocket standard local sur une fixture commune.
|
||||||
|
|
||||||
|
Configuration qualifiée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commitment = confirmed
|
||||||
|
encoding = base64
|
||||||
|
transactionDetails = full
|
||||||
|
maxSupportedTransactionVersion = 0
|
||||||
|
showRewards = false
|
||||||
|
```
|
||||||
|
|
||||||
|
Le test compare :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HTTP getBlock observed
|
||||||
|
vs
|
||||||
|
WS blockSubscribe
|
||||||
|
```
|
||||||
|
|
||||||
|
puis exige :
|
||||||
|
|
||||||
|
```text
|
||||||
|
même SolanaConfirmedBlock
|
||||||
|
même RawTransactionReference
|
||||||
|
même slot
|
||||||
|
même block_time
|
||||||
|
mêmes bytes RAW v1
|
||||||
|
même content_hash
|
||||||
|
```
|
||||||
|
|
||||||
|
La qualification directe est volontairement bornée au sous-ensemble legacy/v0 prouvé. Transaction V1 reste au gate `pre.006`.
|
||||||
|
|
||||||
|
## Qualification négative — Helius `transactionSubscribe`
|
||||||
|
|
||||||
|
Le même canari ouvre une session Helius locale et une hydration HTTP de référence pour la même transaction.
|
||||||
|
|
||||||
|
L'enveloppe full/base64 qualifiée conserve :
|
||||||
|
|
||||||
|
```text
|
||||||
|
transaction wire
|
||||||
|
meta
|
||||||
|
signature
|
||||||
|
slot
|
||||||
|
transactionIndex
|
||||||
|
```
|
||||||
|
|
||||||
|
mais ne porte pas, dans le contrat typé actuel :
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockTime
|
||||||
|
version
|
||||||
|
```
|
||||||
|
|
||||||
|
Le test construit donc la meilleure projection RAW possible sans inventer ces champs et prouve :
|
||||||
|
|
||||||
|
```text
|
||||||
|
même identité que HTTP
|
||||||
|
même slot que HTTP
|
||||||
|
block_time différent
|
||||||
|
bytes RAW v1 différents
|
||||||
|
content_hash différent
|
||||||
|
```
|
||||||
|
|
||||||
|
Décision normative :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Helius transactionSubscribe full/base64
|
||||||
|
-> signal live riche
|
||||||
|
-> HTTP hydration obligatoire
|
||||||
|
-> persistence RAW depuis le matériau HTTP complet
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun RAW v2 automatique et aucune normalisation ad hoc ne sont introduits.
|
||||||
|
|
||||||
|
## Dependency firewall
|
||||||
|
|
||||||
|
Pour les serveurs WebSocket locaux du canari, le manifest Backfill ajoute uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
[dev-dependencies]
|
||||||
|
tokio-tungstenite = { workspace = true, features = ["handshake"] }
|
||||||
|
```
|
||||||
|
|
||||||
|
`tests/dependency_boundary.rs` vérifie explicitement que `tokio-tungstenite` n'apparaît pas dans `[dependencies]` et reste dans `[dev-dependencies]`.
|
||||||
|
|
||||||
|
Il n'existe toujours aucun edge productif :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-onchain-transport-lib -> ksp-raw-transaction-lib
|
||||||
|
ksp-raw-transaction-lib -> ksp-onchain-transport-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
`docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md` distingue désormais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockSubscribe full/base64 legacy-v0 -> direct après parité prouvée
|
||||||
|
Helius transactionSubscribe full/base64 -> signal + hydration HTTP
|
||||||
|
```
|
||||||
|
|
||||||
|
Le plan `031` et la validation `027` enregistrent la même qualification, la fermeture opérateur de `pre.004-fix.001` et le maintien de Transaction V1 dans `pre.006`.
|
||||||
|
|
||||||
|
## Fraîcheur externe revalidée
|
||||||
|
|
||||||
|
Les surfaces changeantes ont été revalidées le 2026-09-06 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana blockSubscribe : méthode unstable, confirmed/finalized, base64, full, maxSupportedTransactionVersion disponible
|
||||||
|
Helius transactionSubscribe : full transaction streaming disponible ; enveloppe KSP qualifiée sans blockTime/version
|
||||||
|
```
|
||||||
|
|
||||||
|
Sources documentaires :
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://solana.com/docs/rpc/websocket/blocksubscribe
|
||||||
|
https://solana.com/docs/rpc/json-structures
|
||||||
|
https://www.helius.dev/docs/rpc/websocket/transaction-subscribe
|
||||||
|
https://www.helius.dev/blog/introducing-next-generation-enhanced-websockets
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces faits d'accès/protocole restent documentaires et ne deviennent pas des constantes provider dans le Worker.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/tests/ws_raw_parity.rs
|
||||||
|
deltas/0.3.10/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (339 table(s), 759 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt` ; aucun gate Cargo de `pre.005` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges dev
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Les arbres sont requis cette fois-ci parce que `pre.005` ajoute une dev-dependency et doit prouver qu'elle ne contamine pas le graphe productif.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
blockSubscribe full/base64 legacy-v0 est qualifié RAW-direct
|
||||||
|
Helius transactionSubscribe full/base64 n'est pas qualifié RAW-direct
|
||||||
|
Helius doit hydrater par HTTP avant persistence RAW
|
||||||
|
aucun champ manquant n'est inventé
|
||||||
|
aucun RAW v2 automatique
|
||||||
|
TR-C2 reste Worker-owned
|
||||||
|
aucun code de production ajouté dans cette tranche
|
||||||
|
Transaction V1 reste à pre.006
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque le gate déterministe de `pre.005`. `pre.006` reste dédiée aux adapters Yellowstone transactions/blocks et au gate Transaction V1 avant toute persistance directe de ces matériaux.
|
||||||
166
deltas/0.3.10/pre.006-fix.001.md
Normal file
166
deltas/0.3.10/pre.006-fix.001.md
Normal file
@@ -0,0 +1,166 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.006-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.006-fix.001` — corrections Rust du gate `pre.006` + recalibrage du sizing
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.006
|
||||||
|
```
|
||||||
|
|
||||||
|
L'archive opérateur `khadhroony-solana-project- v0.3.10-pre.006.zip` est la base autoritaire de ce correctif.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement les défauts Rust révélés par le premier gate opérateur de `pre.006` sans modifier le contrat RAW V1, les goldens, la décision Yellowstone ni les dépendances de production, puis corriger le plan devenu surdimensionné avant toute implémentation lourde du Worker.
|
||||||
|
|
||||||
|
Le gate opérateur a révélé trois familles de défauts dans la responsabilité `pre.006` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
clippy::collapsible_if dans la validation V1 du heap size
|
||||||
|
assert_eq! sur Result<&str, &Error> exigeant à tort PartialEq sur Error
|
||||||
|
canari Yellowstone utilisant Error::new avec un &str et formatant ErrorCode via Display inexistant
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif modifie du Rust et synchronise donc la version technique Cargo conformément à `VER-ID-007` et `VER-ID-010` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Corrections Rust
|
||||||
|
|
||||||
|
`ksp-raw-transaction-lib/src/wire.rs` applique la forme Clippy proposée pour la garde `heap_size` V1 sans modifier les bornes ni le résultat fonctionnel.
|
||||||
|
|
||||||
|
`unit_tests/wire.rs` conserve exactement les quatre golden Base64 mais compare les succès par pattern matching, sans imposer `PartialEq` au type d'erreur KSP.
|
||||||
|
|
||||||
|
`yellowstone_raw_parity.rs` construit désormais l'erreur de projection test-only avec un `ErrorCode` structuré et formate les codes d'erreur explicitement via `domain()` + `code()`. Aucun code d'erreur public de production n'est ajouté.
|
||||||
|
|
||||||
|
## Recalibrage du sizing
|
||||||
|
|
||||||
|
Le plan initial associait dans `0.3.10` deux moitiés devenues nettement séparables :
|
||||||
|
|
||||||
|
```text
|
||||||
|
common RAW + migration/parités = déjà matérialisées en pre.002 à pre.006
|
||||||
|
runtime Worker + sources + repair = pas encore commencé
|
||||||
|
```
|
||||||
|
|
||||||
|
Les anciennes tranches `pre.007` à `pre.011` cumulaient plusieurs responsabilités chacune et ne respectaient plus le budget normatif d'environ 15 à 20 minutes. La clôture de toute la mission initiale dans une seule session devenait également incertaine.
|
||||||
|
|
||||||
|
Le redécoupage durable retenu est donc :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10 = common RAW + observed block + qualification HTTP/WS/Helius/Yellowstone
|
||||||
|
0.3.11 = Worker foundation/runtime + persistence déterministe
|
||||||
|
0.3.12 = Yellowstone + hydration HTTP + continuity
|
||||||
|
0.3.13 = WS/Helius/HTTP live + convergence multi-source
|
||||||
|
0.3.14 = gap repair/hardening + smokes provider accessibles
|
||||||
|
0.3.15 = Desk d'ingestion
|
||||||
|
0.3.16 = Backfill multi-source/multi-stratégie
|
||||||
|
```
|
||||||
|
|
||||||
|
La queue de fermeture de `0.3.10` devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.007 = gate technique final, budget 15-20 min
|
||||||
|
pre.008 = réconciliation documentaire, budget 10-15 min
|
||||||
|
pre.009 = préparation de publication, budget 5-10 min
|
||||||
|
rel.001 = publication stable mécanique
|
||||||
|
```
|
||||||
|
|
||||||
|
Chaque nouvelle release Worker recommencera par sa propre `pre.001` de sizing et devra scinder toute prerelease estimée au-delà du budget.
|
||||||
|
|
||||||
|
## Vérification de l'archive
|
||||||
|
|
||||||
|
Exécuté sur l'archive autoritaire avant modification :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unzip -t : PASS
|
||||||
|
racine ZIP unique : PASS
|
||||||
|
chemins absolus/traversal : 0
|
||||||
|
symlinks : 0
|
||||||
|
workspace.package.version base : 0.3.10-pre.6
|
||||||
|
prompt 029 présent : PASS
|
||||||
|
règles obligatoires présentes : PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées
|
||||||
|
|
||||||
|
Après les corrections Rust et le recalibrage documentaire :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.10
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultats :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit : clean
|
||||||
|
Rust export completeness audit : 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit : clean
|
||||||
|
Markdown table audit : clean
|
||||||
|
```
|
||||||
|
|
||||||
|
Les contrôles structurels ont également vérifié que le delta ne contient aucun chemin d'archive dangereux et que les références durables principales du nouveau découpage sont cohérentes.
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Les commandes suivantes ne sont donc **pas** déclarées PASS ici :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges dev
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Elles constituent le rejeu opérateur requis pour fermer le fix.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.006-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
ROADMAP.md
|
||||||
|
crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/wire.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/wire.rs
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/architecture/010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
- Les goldens Legacy/V0/V1 et le contrat RAW V1 de `pre.006` ne changent pas.
|
||||||
|
- Le canari Yellowstone reste test-only et n'ajoute aucun nouvel edge productif.
|
||||||
|
- `0.3.10` est réduit à la common RAW et aux qualifications cross-source déjà matérialisées avant ouverture du runtime Worker.
|
||||||
|
- Le Worker concret est réparti sur `0.3.11` à `0.3.14` afin de respecter une release concrète par session et des prereleases bornées.
|
||||||
|
- La Desk et le Backfill multi-source sont décalés respectivement à `0.3.15` et `0.3.16`.
|
||||||
|
- Les anciens prompts/plans de releases déjà ouvertes restent des traces historiques et ne sont pas réécrits rétroactivement uniquement pour refléter ce recalibrage.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question fonctionnelle nouvelle pour ce fix. Le prochain gate est le rejeu Rust opérateur du correctif, puis `pre.007` si ce rejeu est vert.
|
||||||
118
deltas/0.3.10/pre.006-fix.002.md
Normal file
118
deltas/0.3.10/pre.006-fix.002.md
Normal file
@@ -0,0 +1,118 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.006-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.006-fix.002` — rustdoc Yellowstone + réconciliation architecture/prompt handoff
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.006-fix.001 appliquée
|
||||||
|
workspace.package.version = 0.3.10-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 7 septembre 2026 confirme que les erreurs Rust corrigées par `fix.001` ont disparu. Le seul blocage restant sous Clippy strict est le lint crate-level `missing_docs` de `crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs`.
|
||||||
|
|
||||||
|
## 2. Version technique
|
||||||
|
|
||||||
|
Le correctif modifie un test Rust compilé. Conformément à `VER-ID-007` / `VER-ID-010` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.6.fix.2
|
||||||
|
commit attendu = v0.3.10-pre.006-fix.002
|
||||||
|
aucun tag prerelease
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune version npm/Tauri n'est modifiée : aucune application Desk n'est touchée.
|
||||||
|
|
||||||
|
## 3. Correction Rust
|
||||||
|
|
||||||
|
`tests/yellowstone_raw_parity.rs` reçoit la rustdoc crate-level manquante :
|
||||||
|
|
||||||
|
```text
|
||||||
|
//! Deterministic local Yellowstone gRPC parity canary for source-neutral RAW transaction v1 projection.
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `#[allow(missing_docs)]` n'est ajouté. Le serveur fixture, la projection Yellowstone, les goldens, les assertions et les dépendances restent inchangés.
|
||||||
|
|
||||||
|
## 4. Inventaire des composants
|
||||||
|
|
||||||
|
`docs/architecture/004-COMPONENT_INVENTORY.md` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
+ ksp-raw-transaction-lib comme common RAW Transaction implémentée en 0.3.10
|
||||||
|
CORE processor -> STRUCTURAL job
|
||||||
|
CORE worker -> STRUCTURAL worker
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Job STRUCTURAL représente le traitement borné/rejouable RAW -> CORE ; le Worker STRUCTURAL représente le service continu backlog RAW -> CORE. Les références durables correspondantes dans `009-ACQUISITION_WORKERS_AND_JOBS.md`, `010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md` et la séquence fonctionnelle active sont synchronisées. Les noms/packagings concrets restent à fixer lors de l'ouverture de la couche CORE.
|
||||||
|
|
||||||
|
## 5. Graphe de dépendances
|
||||||
|
|
||||||
|
`docs/architecture/005-DEPENDENCY_GRAPH.md` matérialise la lower layer réelle :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-api
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> base64 / serde_json / sha2
|
||||||
|
|
||||||
|
-X-> Transport
|
||||||
|
-X-> ksp-store-lib
|
||||||
|
-X-> Job / Worker
|
||||||
|
-X-> Config / runtime async / backend Store
|
||||||
|
```
|
||||||
|
|
||||||
|
Le graphe Backfill ajoute son edge réel vers `ksp-raw-transaction-lib`, et le graphe du futur Worker réserve la même common crate avant la persistence par `ksp-store-lib`.
|
||||||
|
|
||||||
|
## 6. Note de génération des prompts `0.3.12` à `0.3.15`
|
||||||
|
|
||||||
|
Le plan `031` n'introduit pas quatre prompts prématurés. Il fixe un handoff séquentiel :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11 ferme -> génère le prompt 0.3.12
|
||||||
|
0.3.12 ferme -> génère le prompt 0.3.13
|
||||||
|
0.3.13 ferme -> génère le prompt 0.3.14
|
||||||
|
0.3.14 ferme -> génère le prompt 0.3.15
|
||||||
|
```
|
||||||
|
|
||||||
|
Chaque prompt doit partir de la base stable réellement obtenue, refaire son audit/sizing `pre.001`, rester sous le budget 15–20 min par tranche et rescinder sa release si la clôture dans une session devient incertaine. La note transfère les missions Yellowstone, WS/Helius/HTTP, gap repair/hardening puis Desk sans les figer comme vérité future.
|
||||||
|
|
||||||
|
Le plan corrige aussi l'ancienne référence résiduelle de la Desk `0.3.11` vers sa cible recalibrée `0.3.15`.
|
||||||
|
|
||||||
|
## 7. Validation opérateur requise
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges dev
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune commande ne doit être déclarée PASS avant exécution réelle. Si ce gate est vert, `pre.006-fix.002` ferme le défaut technique restant de `pre.006` et `pre.007` peut rester le gate technique final recalibré de `0.3.10`.
|
||||||
|
|
||||||
|
## 8. Périmètre du delta
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/architecture/010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md
|
||||||
|
docs/plans/002-FUNCTIONAL_RELEASE_SEQUENCE.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
deltas/0.3.10/pre.006-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun code de production, manifest de crate, dépendance, feature, golden RAW, Config, Store schema, frontend ou application n'est modifié.
|
||||||
206
deltas/0.3.10/pre.006.md
Normal file
206
deltas/0.3.10/pre.006.md
Normal file
@@ -0,0 +1,206 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.006.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.006` — Transaction V1 + parité Yellowstone transaction/block
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
livraison : 0.3.10-pre.005-fix.002
|
||||||
|
workspace.package.version = 0.3.10-pre.5.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué le 2026-09-06 ferme cette baseline : audits statiques propres, `cargo check --workspace` PASS, Clippy strict PASS, common RAW PASS, 387 tests Transport avec suites publiques/release/doctests PASS, et suite Backfill complète PASS, y compris hardening et les deux canaris WS/Helius.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer avant le Worker concret :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transaction V1 / SIMD-0385 dans la common source-neutral
|
||||||
|
Yellowstone Transaction -> transaction wire exact
|
||||||
|
Yellowstone Block -> même transaction wire + block_time
|
||||||
|
qualification RAW-direct vs hydration
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun adapter productif `Transport DTO -> common` n'est ajouté. `TR-C2` reste Worker-owned et `pre.007` reste le premier point où la crate Worker concrète peut apparaître.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.6
|
||||||
|
Cargo.toml header = 499
|
||||||
|
workspace members = 20
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common wire Legacy/V0/V1
|
||||||
|
|
||||||
|
`ksp-raw-transaction-lib` ajoute un modèle source-neutral :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawSolanaMessageVersion
|
||||||
|
RawSolanaMessageHeader
|
||||||
|
RawSolanaCompiledInstruction
|
||||||
|
RawSolanaAddressTableLookup
|
||||||
|
RawSolanaTransactionConfig
|
||||||
|
RawSolanaTransactionMessage
|
||||||
|
RawSolanaTransactionWire
|
||||||
|
```
|
||||||
|
|
||||||
|
et deux serializers :
|
||||||
|
|
||||||
|
```text
|
||||||
|
serialize_solana_transaction_wire
|
||||||
|
serialize_solana_transaction_wire_base64
|
||||||
|
```
|
||||||
|
|
||||||
|
Legacy et V0 conservent le layout signatures-first existant. V1 suit SIMD-0385 : `0x81`, header, config mask, lifetime, compteurs, adresses, valeurs config, headers/payloads d'instructions et signatures terminales.
|
||||||
|
|
||||||
|
Les gardes V1 verrouillent notamment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
transaction <= 4096 bytes
|
||||||
|
1..=12 signatures
|
||||||
|
<= 64 adresses
|
||||||
|
<= 64 instructions
|
||||||
|
num_readonly_signed < num_required_signatures
|
||||||
|
indexes programme/comptes bornés
|
||||||
|
aucune Address Lookup Table
|
||||||
|
aucune adresse dupliquée
|
||||||
|
config présente, y compris vide
|
||||||
|
aucun trailing byte après les signatures
|
||||||
|
```
|
||||||
|
|
||||||
|
Les goldens couvrent Legacy, V0, V1 configuration complète et V1 configuration vide.
|
||||||
|
|
||||||
|
## Extraction de signature V1
|
||||||
|
|
||||||
|
`extract_raw_transaction_signature_from_binary_base64` distingue désormais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
premier octet == 0x81 -> parser V1 et signatures terminales
|
||||||
|
sinon -> parser short-vector signatures-first Legacy/V0
|
||||||
|
```
|
||||||
|
|
||||||
|
Le parser V1 parcourt structurellement mask/config, adresses, headers, payloads et signatures et exige la fin exacte du wire.
|
||||||
|
|
||||||
|
## Parité Yellowstone
|
||||||
|
|
||||||
|
`crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs` monte un serveur Geyser local déterministe, ouvre le vrai `YellowstoneGrpcChannel::open_standard_subscribe` puis reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SubscribeUpdateTransaction V1
|
||||||
|
SubscribeUpdateBlock V1 portant la même transaction
|
||||||
|
```
|
||||||
|
|
||||||
|
La projection vers `RawSolanaTransactionWire` est strictement test-only. Le canari exige le même wire V1 exact pour les deux updates et vérifie slot, transaction index et `block_time` côté Block.
|
||||||
|
|
||||||
|
## Qualification RAW
|
||||||
|
|
||||||
|
Le gate ferme volontairement une qualification partielle :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Yellowstone transaction wire = direct/fiable
|
||||||
|
Yellowstone Transaction RAW = hydration HTTP obligatoire : block_time absent
|
||||||
|
Yellowstone Block RAW complet= hydration HTTP obligatoire : meta JSON byte-identical non prouvée
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun champ manquant n'est inventé et aucune projection protobuf -> JSON ad hoc n'est introduite pour forcer un hash RAW identique.
|
||||||
|
|
||||||
|
## Dependency firewall
|
||||||
|
|
||||||
|
Le Backfill ajoute uniquement en dev :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tonic = { workspace = true, features = ["codegen", "server"] }
|
||||||
|
yellowstone-grpc-proto = { workspace = true, features = ["tonic"] }
|
||||||
|
```
|
||||||
|
|
||||||
|
`dependency_boundary.rs` verrouille leur absence des dépendances production et `hardening.rs` recalibre le set dev exact. Les dépendances normales restent inchangées.
|
||||||
|
|
||||||
|
Il n'existe toujours aucun edge productif :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-onchain-transport-lib -> ksp-raw-transaction-lib
|
||||||
|
ksp-raw-transaction-lib -> ksp-onchain-transport-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fraîcheur externe revalidée
|
||||||
|
|
||||||
|
Revalidation le 2026-09-07 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SIMD-0385 : Transaction V1, status Review, version byte 129, signatures terminales, sans ALT
|
||||||
|
Solana Transaction V1 examples : capacité Yellowstone avec Message.config ; activation cluster encore feature-gated/pre-release
|
||||||
|
```
|
||||||
|
|
||||||
|
Sources :
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://github.com/solana-foundation/solana-improvement-documents/blob/main/proposals/0385-transaction-v1.md
|
||||||
|
https://github.com/solana-foundation/transaction-v1-examples
|
||||||
|
https://github.com/rpcpool/yellowstone-grpc/blob/master/yellowstone-grpc-proto/proto/solana-storage.proto
|
||||||
|
```
|
||||||
|
|
||||||
|
KSP implémente la capacité de lecture/canonicalisation V1 sans déclarer son activation sur Mainnet ou un autre cluster.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/tests/yellowstone_raw_parity.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/wire.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/wire.rs
|
||||||
|
deltas/0.3.10/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/hardening.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/lib.rs
|
||||||
|
crates/ksp-raw-transaction-lib/src/signature.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/public_api.rs
|
||||||
|
crates/ksp-raw-transaction-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-raw-transaction-lib/unit_tests/signature.rs
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges dev
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Les arbres Cargo sont requis car `pre.006` ajoute deux dev-dependencies test-only.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
common sérialise Legacy/V0/V1 sans dépendre de Transport
|
||||||
|
V1 est reconnu par 0x81 et possède des signatures terminales
|
||||||
|
Yellowstone Transaction/Block conservent un transaction wire V1 exact sur canari
|
||||||
|
Yellowstone RAW complet reste hydraté par HTTP tant que TR-C4 meta/block_time complet n'est pas prouvé
|
||||||
|
aucun RAW v2 automatique
|
||||||
|
aucun Worker concret avant pre.007
|
||||||
|
TR-C2 reste Worker-owned
|
||||||
|
```
|
||||||
121
deltas/0.3.10/pre.007.md
Normal file
121
deltas/0.3.10/pre.007.md
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.007.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.007` — gate technique final RAW commune / cross-source
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.006-fix.002 appliquée
|
||||||
|
workspace.package.version = 0.3.10-pre.6.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rejeu opérateur communiqué le 8 septembre 2026 ferme entièrement `pre.006-fix.002` : audits Rust/Markdown, `cargo check --workspace`, Clippy workspace `--all-targets --all-features -- -D warnings`, puis les suites `ksp-raw-transaction-lib`, `ksp-onchain-transport-lib` et `ksp-job-backfill-lib` sont PASS.
|
||||||
|
|
||||||
|
Les smokes réseau Transport opt-in restent `ignored` par politique ; aucun succès live n'est revendiqué à partir de ces tests non exécutés.
|
||||||
|
|
||||||
|
## 2. Objet
|
||||||
|
|
||||||
|
Ouvrir exclusivement le gate technique final de `0.3.10` sans nouveau développement fonctionnel et sans effectuer la réconciliation documentaire réservée à `pre.008`.
|
||||||
|
|
||||||
|
Cette tranche fige l'état à tester sous l'identifiant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate vert de `pre.006-fix.002` est une précondition. Il ne remplace jamais la preuve finale sur cette version exacte.
|
||||||
|
|
||||||
|
## 3. Version
|
||||||
|
|
||||||
|
Cette livraison est une prerelease non-fix. Conformément à la politique de version KSP :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.7
|
||||||
|
commit attendu = v0.3.10-pre.007
|
||||||
|
aucun tag prerelease
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune version npm/Tauri n'est modifiée.
|
||||||
|
|
||||||
|
## 4. Périmètre
|
||||||
|
|
||||||
|
Aucun changement de code métier, API, golden RAW, dépendance, feature, Config, Store, Transport, Backfill ou frontend.
|
||||||
|
|
||||||
|
Le seul ajustement de validation corrige une référence devenue obsolète après le redécoupage de `pre.006` : `ksp-worker-raw-transaction-ingest-lib` n'existe pas dans `0.3.10` et appartient désormais à `0.3.11`. Le gate final utilise donc les graphes réellement pertinents de la common RAW et du Backfill.
|
||||||
|
|
||||||
|
## 5. Gate déterministe final
|
||||||
|
|
||||||
|
À exécuter séquentiellement sur `0.3.10-pre.7` :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges dev
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate est séquentiel : tout échec bloque `pre.007`, même si des commandes suivantes réussissent. Un défaut découvert ouvre `pre.007-fix.NNN` avant toute poursuite vers `pre.008`.
|
||||||
|
|
||||||
|
## 6. Non-claims live
|
||||||
|
|
||||||
|
Cette tranche n'ajoute aucun nouveau smoke réseau. Les smokes déjà présents restent opt-in selon leurs politiques respectives. En particulier, un test `ignored` n'est pas déclaré PASS live.
|
||||||
|
|
||||||
|
La qualification déterministe acquise reste inchangée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HTTP / WS standard / Helius / Yellowstone : preuves fixture/parité existantes
|
||||||
|
Yellowstone RAW complet : hydration HTTP conservée selon qualification actuelle
|
||||||
|
Helius transactionSubscribe RAW complet : hydration HTTP conservée selon qualification actuelle
|
||||||
|
Worker concret : hors 0.3.10, transféré à 0.3.11+
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Validation de l'environnement d'assemblage
|
||||||
|
|
||||||
|
À enregistrer avant livraison :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
contrôle statique version/scope/header
|
||||||
|
contrôle archive delta et reconstruction overlay
|
||||||
|
```
|
||||||
|
|
||||||
|
Cargo/rustc/rustfmt ne sont pas disponibles dans l'environnement d'assemblage ; aucun gate Cargo de `pre.007` n'est déclaré PASS ici.
|
||||||
|
|
||||||
|
## 10. Suite
|
||||||
|
|
||||||
|
Après retour opérateur entièrement vert sur `0.3.10-pre.7` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.008 = réconciliation documentaire finale 0.3.10
|
||||||
|
pre.009 = préparation publication + prompt 0.3.11
|
||||||
|
rel.001 = publication stable mécanique
|
||||||
|
```
|
||||||
109
deltas/0.3.10/pre.008-fix.001.md
Normal file
109
deltas/0.3.10/pre.008-fix.001.md
Normal file
@@ -0,0 +1,109 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.008-fix.001` — fermeture de la nomenclature D1–D4
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.008
|
||||||
|
workspace.package.version = 0.3.10-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.008` est vert : format check, audits Rust/Markdown, `cargo check --workspace` et Clippy workspace/all-targets/all-features avec `-D warnings` passent sur cette baseline.
|
||||||
|
|
||||||
|
## 2. Objet
|
||||||
|
|
||||||
|
Corriger les dernières survivances actives de l'ancienne notation des couches de données découvertes pendant la préparation de `pre.009`, sans les absorber dans le couloir de publication.
|
||||||
|
|
||||||
|
`pre.008` a réservé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
N1–N4 = niveaux architecturaux de composants
|
||||||
|
D1–D4 = couches durables Store/data
|
||||||
|
|
||||||
|
D1 RAW
|
||||||
|
D2 STRUCTURAL
|
||||||
|
D3 DECODED
|
||||||
|
D4 DOMAIN
|
||||||
|
```
|
||||||
|
|
||||||
|
La correction reste donc strictement dans la responsabilité de réconciliation documentaire de `pre.008`.
|
||||||
|
|
||||||
|
## 3. Version Cargo
|
||||||
|
|
||||||
|
Correction strictement documentaire :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version reste 0.3.10-pre.8
|
||||||
|
Cargo.toml inchangé
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun identifiant `fix` n'est reflété dans Cargo pour ce correctif documentaire.
|
||||||
|
|
||||||
|
## 4. Corrections
|
||||||
|
|
||||||
|
### `ROADMAP.md`
|
||||||
|
|
||||||
|
Les usages actifs qui désignaient encore les couches de données avec `N1`, `N2` ou `N3` deviennent :
|
||||||
|
|
||||||
|
```text
|
||||||
|
N1 data/event-only -> D1
|
||||||
|
« taxonomie N1 » -> « taxonomie D1 »
|
||||||
|
« nouvelle famille N1 » -> « nouvelle famille D1 »
|
||||||
|
« ouverture N2/N3 » -> « ouverture D2/D3 »
|
||||||
|
« autres familles N1 » -> « autres familles D1 »
|
||||||
|
```
|
||||||
|
|
||||||
|
Les occurrences où `N1–N4` désignent réellement les niveaux architecturaux restent inchangées.
|
||||||
|
|
||||||
|
### `docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md`
|
||||||
|
|
||||||
|
La mention active « migration de données N1 » devient « migration de données D1 ».
|
||||||
|
|
||||||
|
Les anciennes traces historiques de releases, prompts, plans ou validations clôturés ne sont pas réécrites.
|
||||||
|
|
||||||
|
## 5. Hors périmètre
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
prompt 0.3.11
|
||||||
|
code / tests / Config / schemas
|
||||||
|
README / USAGE
|
||||||
|
plans / validations
|
||||||
|
autres corrections de publication
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.009` reste la prochaine tranche et conservera uniquement CHANGELOG, ROADMAP, prompt suivant et fichiers mécaniques.
|
||||||
|
|
||||||
|
## 6. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
ROADMAP.md
|
||||||
|
docs/architecture/011-RAW_TRANSACTION_ACQUISITION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Fichier ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/pre.008-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Validation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
```
|
||||||
|
|
||||||
|
Comme le correctif est exclusivement documentaire et ne touche pas Cargo, Rust, build, runtime ou Config, aucun replay du workspace complet n'est requis si ces audits restent verts.
|
||||||
|
|
||||||
|
## 9. Suite
|
||||||
|
|
||||||
|
Après application et audits documentaires verts :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.009 — préparation de publication + prompt 0.3.11
|
||||||
|
0.3.10-rel.001 — publication stable mécanique
|
||||||
|
```
|
||||||
201
deltas/0.3.10/pre.008.md
Normal file
201
deltas/0.3.10/pre.008.md
Normal file
@@ -0,0 +1,201 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.008.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.008` — réconciliation documentaire finale RAW commune / cross-source
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.007 appliquée
|
||||||
|
workspace.package.version = 0.3.10-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rejeu opérateur communiqué le 8 septembre 2026 ferme entièrement le gate technique final de `pre.007` : audits Rust/Markdown, `cargo check --workspace`, Clippy workspace strict, `cargo test --workspace --all-targets --all-features`, suites ciblées common/Transport/Backfill et graphes Cargo demandés sont exécutés sans échec.
|
||||||
|
|
||||||
|
Les smokes live opt-in restent `ignored` par politique et ne sont pas revendiqués comme preuves réseau exécutées.
|
||||||
|
|
||||||
|
## 2. Objet
|
||||||
|
|
||||||
|
Effectuer exclusivement la réconciliation documentaire finale de `0.3.10` après fermeture technique de `pre.007`, sans nouveau développement fonctionnel.
|
||||||
|
|
||||||
|
Cette tranche :
|
||||||
|
|
||||||
|
```text
|
||||||
|
réconcilie la nomenclature durable des couches de données Store ;
|
||||||
|
synchronise la séquence fonctionnelle active avec le redécoupage 0.3.10 -> 0.3.16 ;
|
||||||
|
réconcilie les règles et documents d'architecture actifs ;
|
||||||
|
complète la documentation durable de ksp-raw-transaction-lib ;
|
||||||
|
ne modifie aucun comportement Rust, API, dépendance, feature ou runtime.
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Version
|
||||||
|
|
||||||
|
Cette livraison est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.8
|
||||||
|
commit attendu = v0.3.10-pre.008
|
||||||
|
aucun tag prerelease
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune version npm/Tauri n'est modifiée.
|
||||||
|
|
||||||
|
## 4. Nomenclature canonique des couches de données
|
||||||
|
|
||||||
|
La progression durable du Store devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
D1 RAW -> D2 STRUCTURAL -> D3 DECODED -> D4 DOMAIN
|
||||||
|
```
|
||||||
|
|
||||||
|
Sémantique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
D1 RAW = matériau acquis et conservé avant décomposition métier ;
|
||||||
|
D2 STRUCTURAL = décomposition Solana générique en structures unitaires exploitables ;
|
||||||
|
D3 DECODED = décodage générique des structures selon les contrats de programmes ;
|
||||||
|
D4 DOMAIN = matérialisation métier/spécialisée issue des données décodées.
|
||||||
|
```
|
||||||
|
|
||||||
|
L'ancien nom de couche D2 `CORE` est abandonné. Il créait une ambiguïté avec `ksp-core-lib` et le domaine Core fondamental alors que cette couche Store représente une décomposition structurelle du RAW.
|
||||||
|
|
||||||
|
Cette normalisation ne renomme jamais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib ;
|
||||||
|
le domaine Core architectural fondamental ;
|
||||||
|
les noms propres comme « Solana Core Programs » ;
|
||||||
|
les traces historiques clôturées qui documentent l'ancienne terminologie.
|
||||||
|
```
|
||||||
|
|
||||||
|
Les niveaux architecturaux `N1–N4` de `docs/architecture/002-LAYERS_AND_DEPENDENCIES.md` restent réservés aux niveaux de composants. Les couches de données utilisent désormais `D1–D4` afin d'éviter toute collision de vocabulaire. Les plans Store historiques qui parlaient de N1–N4 data restent interprétables selon la correspondance ci-dessus mais ne sont pas réécrits en masse.
|
||||||
|
|
||||||
|
## 5. Conséquences STRUCTURAL
|
||||||
|
|
||||||
|
Les rôles actifs deviennent explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
STRUCTURAL persistence
|
||||||
|
RAW -> STRUCTURAL replay/backfill
|
||||||
|
STRUCTURAL job
|
||||||
|
STRUCTURAL worker
|
||||||
|
STRUCTURAL inspection/control app lorsque pertinent
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `STRUCTURAL job` est borné et rejouable. Le `STRUCTURAL worker` traite continuellement le backlog RAW. Aucun des deux ne réalise le décodage Program : la frontière `STRUCTURAL -> DECODED` reste en aval.
|
||||||
|
|
||||||
|
## 6. Séquence fonctionnelle active
|
||||||
|
|
||||||
|
`docs/plans/002-FUNCTIONAL_RELEASE_SEQUENCE.md` est remis en cohérence avec la trajectoire réellement matérialisée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.1 ksp-store-api RAW
|
||||||
|
0.3.2 Store runtime + backend PostgreSQL foundation
|
||||||
|
0.3.3 persistence RawTransaction
|
||||||
|
0.3.4 persistence RawAccountState
|
||||||
|
0.3.5 Interface acquisition events
|
||||||
|
0.3.6 ksp-job-api + RawTransaction Backfill
|
||||||
|
0.3.7 ksp-app-backfill-desk
|
||||||
|
0.3.8 ksp-app-store-desk RAW
|
||||||
|
0.3.9 ksp-worker-api + audit acquisition RawTransaction
|
||||||
|
0.3.10 common ksp-raw-transaction-lib + preuves cross-source
|
||||||
|
0.3.11-0.3.14 Worker RAW ingest découpé par responsabilité
|
||||||
|
0.3.15 ksp-app-raw-transaction-ingest-desk
|
||||||
|
0.3.16 Backfill multi-source / multi-stratégie
|
||||||
|
```
|
||||||
|
|
||||||
|
Le redécoupage ne change pas les frontières Job/Worker arrêtées précédemment et ne réintroduit aucun Worker concret dans `0.3.10`.
|
||||||
|
|
||||||
|
## 7. Documentation de `ksp-raw-transaction-lib`
|
||||||
|
|
||||||
|
La crate common RAW étant désormais une bibliothèque complétée de `0.3.10`, elle reçoit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-raw-transaction-lib/README.md
|
||||||
|
crates/ksp-raw-transaction-lib/USAGE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `README.md` fixe son rôle, ses dépendances autorisées et ses frontières. Le `USAGE.md` reste volontairement version-neutre et montre uniquement l'utilisation durable de la surface publique de crate root : signatures, wire Solana, canonicalisation RAW et assemblage acquisition/observation.
|
||||||
|
|
||||||
|
La documentation rappelle explicitement que le résultat de cette crate reste D1 RAW et qu'elle n'implémente pas la transformation RAW -> STRUCTURAL.
|
||||||
|
|
||||||
|
## 8. ROADMAP et publication
|
||||||
|
|
||||||
|
`ROADMAP.md` est modifié uniquement pour éliminer l'ancien libellé actif `RAW/CORE/DECODE/SPECIALIZED` au profit de `RAW/STRUCTURAL/DECODED/DOMAIN`.
|
||||||
|
|
||||||
|
Les changements de statut de release, le CHANGELOG et la préparation du prompt `0.3.11` restent réservés à `pre.009`.
|
||||||
|
|
||||||
|
## 9. Traces historiques
|
||||||
|
|
||||||
|
Les prompts, deltas, plans et validations clôturés ne sont pas réécrits en masse. Une ancienne occurrence de `CORE` dans une trace historique peut donc rester légitime lorsqu'elle décrit l'état documentaire de sa livraison.
|
||||||
|
|
||||||
|
Cette tranche corrige les documents durables actifs, pas l'histoire du repository.
|
||||||
|
|
||||||
|
## 10. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
ROADMAP.md
|
||||||
|
docs/IDEAS.md
|
||||||
|
docs/architecture/002-LAYERS_AND_DEPENDENCIES.md
|
||||||
|
docs/architecture/003-COMPONENT_CONTRACTS.md
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/006-WIRE_AND_PROGRAM.md
|
||||||
|
docs/architecture/008-DATA_MATERIALIZATION_AND_STORE.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/architecture/010-APPS_SERVICES_SCENARIOS_AND_CONTROL.md
|
||||||
|
docs/plans/002-FUNCTIONAL_RELEASE_SEQUENCE.md
|
||||||
|
docs/plans/031-V0_3_10_RAW_TRANSACTION_INGEST_PLAN.md
|
||||||
|
docs/rules/RULES_DEPENDENCIES.md
|
||||||
|
docs/rules/RULES_KSP.md
|
||||||
|
docs/validation/027-V0_3_10_RAW_TRANSACTION_INGEST.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 11. Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-raw-transaction-lib/README.md
|
||||||
|
crates/ksp-raw-transaction-lib/USAGE.md
|
||||||
|
deltas/0.3.10/pre.008.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun fichier Rust source, manifest de crate, Config, frontend ou dépendance n'est modifié.
|
||||||
|
|
||||||
|
## 12. Validation d'assemblage
|
||||||
|
|
||||||
|
À enregistrer avant livraison :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
contrôle statique nomenclature D1-D4 / STRUCTURAL
|
||||||
|
contrôle statique version/scope/headers
|
||||||
|
contrôle exact de l'inventaire delta
|
||||||
|
contrôle archive delta et reconstruction overlay
|
||||||
|
```
|
||||||
|
|
||||||
|
Cargo/rustc/rustfmt ne sont pas disponibles dans l'environnement d'assemblage ; aucun gate Cargo de `pre.008` n'est déclaré PASS localement. Le gate technique complet de `pre.007` reste la baseline opérateur fermée ; `pre.008` ne modifie aucun code ni graphe de dépendances.
|
||||||
|
|
||||||
|
## 13. Rejeu opérateur `pre.008`
|
||||||
|
|
||||||
|
Comme cette tranche est documentaire hors incrément mécanique de version workspace, le rejeu minimal attendu est :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
```
|
||||||
|
|
||||||
|
Un échec ouvre `pre.008-fix.NNN`. Aucun nouveau test fonctionnel ou smoke live n'est requis par cette réconciliation documentaire seule.
|
||||||
|
|
||||||
|
## 14. Suite
|
||||||
|
|
||||||
|
Après retour opérateur vert sur `0.3.10-pre.8` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.009 = préparation publication, CHANGELOG, statuts ROADMAP et prompt 0.3.11
|
||||||
|
rel.001 = publication stable mécanique de 0.3.10
|
||||||
|
```
|
||||||
188
deltas/0.3.10/pre.009.md
Normal file
188
deltas/0.3.10/pre.009.md
Normal file
@@ -0,0 +1,188 @@
|
|||||||
|
<!-- file: deltas/0.3.10/pre.009.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-pre.009` — préparation de publication
|
||||||
|
|
||||||
|
## 1. Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.008 appliquée
|
||||||
|
0.3.10-pre.008-fix.001 appliqué
|
||||||
|
workspace.package.version = 0.3.10-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.008` est vert sur format, audits Rust/Markdown, `cargo check --workspace` et Clippy workspace/all-targets/all-features avec `-D warnings`. Le correctif documentaire `pre.008-fix.001` est ensuite validé par les audits Rust/Markdown, tous deux verts.
|
||||||
|
|
||||||
|
## 2. Objet
|
||||||
|
|
||||||
|
Préparer exclusivement la publication stable de `0.3.10` après fermeture technique `pre.007` et réconciliation documentaire `pre.008` + `pre.008-fix.001`.
|
||||||
|
|
||||||
|
Conformément à `PROMPT_STRUCTURE.md` et `VER-LIFECYCLE-003`, cette tranche ne modifie fonctionnellement que :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompt de démarrage 0.3.11
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
```
|
||||||
|
|
||||||
|
avec les fichiers mécaniques obligatoires :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
deltas/0.3.10/pre.009.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.10-pre.9
|
||||||
|
delivery = 0.3.10-pre.009
|
||||||
|
commit attendu = v0.3.10-pre.009
|
||||||
|
aucun tag prerelease
|
||||||
|
```
|
||||||
|
|
||||||
|
Le header du `Cargo.toml` racine passe de `503` à `504`.
|
||||||
|
|
||||||
|
## 4. CHANGELOG
|
||||||
|
|
||||||
|
`CHANGELOG.md` reçoit la synthèse stable de `0.3.10` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib comme common D1 RAW source-neutral
|
||||||
|
migration Backfill sans changement des golden bytes/hash
|
||||||
|
wire Solana Legacy/V0/V1
|
||||||
|
get_block_observed
|
||||||
|
qualification HTTP / WS standard / Helius / Yellowstone
|
||||||
|
non-claims explicites lorsqu'une hydration reste nécessaire
|
||||||
|
gate technique final pre.007
|
||||||
|
nomenclature D1 RAW -> D2 STRUCTURAL -> D3 DECODED -> D4 DOMAIN
|
||||||
|
redécoupage Worker 0.3.11 -> 0.3.14
|
||||||
|
handoff 0.3.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune nouvelle preuve live n'est inventée par la synthèse.
|
||||||
|
|
||||||
|
## 5. ROADMAP
|
||||||
|
|
||||||
|
`ROADMAP.md` marque `0.3.10` comme réalisé et remplace son texte prospectif par le résultat effectivement livré. Les releases `0.3.11` à `0.3.16` restent prévues et conservent leurs frontières déjà réconciliées.
|
||||||
|
|
||||||
|
Aucune autre réconciliation documentaire n'appartient à cette tranche.
|
||||||
|
|
||||||
|
## 6. Prompt `0.3.11`
|
||||||
|
|
||||||
|
Ajout :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/030-V0_3_11_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le prompt ouvre `0.3.11` uniquement depuis `v0.3.10` et impose un `pre.001` audit/brainstorming/sizing avant codage lourd.
|
||||||
|
|
||||||
|
Mission initiale à réauditer :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
crate/dependency firewall
|
||||||
|
settings source-neutral
|
||||||
|
handle/start-stop/lifecycle
|
||||||
|
supervisor privé
|
||||||
|
channels bornés
|
||||||
|
pipeline déterministe common RAW -> Store
|
||||||
|
persistence/deduplication/content conflict
|
||||||
|
snapshots latest-value sûrs
|
||||||
|
hardening runtime de fondation
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune voie Yellowstone/WS/Helius/HTTP live complexe n'est requise pour fermer `0.3.11`. Le prompt impose de rescinder la release avant implémentation si son sizing réel ne tient pas dans une session.
|
||||||
|
|
||||||
|
## 7. Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/030-V0_3_11_START_PROMPT.md
|
||||||
|
deltas/0.3.10/pre.009.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. Hors périmètre
|
||||||
|
|
||||||
|
Aucun fichier suivant ne doit être corrigé dans `pre.009` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
README / USAGE
|
||||||
|
architecture
|
||||||
|
plans / validations
|
||||||
|
code Rust / TypeScript
|
||||||
|
tests
|
||||||
|
Config / schemas
|
||||||
|
migrations
|
||||||
|
manifest de crate
|
||||||
|
dépendances/features
|
||||||
|
runtime/backend
|
||||||
|
smokes live
|
||||||
|
```
|
||||||
|
|
||||||
|
Toute anomalie appartenant à un couloir antérieur renvoie vers une tranche dédiée avant publication.
|
||||||
|
|
||||||
|
## 11. Validations exécutées à l'assemblage
|
||||||
|
|
||||||
|
Résultats réellement obtenus avant empaquetage :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 771 file(s))
|
||||||
|
contrôle du scope: exactement cinq fichiers ajoutés/modifiés
|
||||||
|
unzip -t: PASS
|
||||||
|
reconstruction overlay: 1713 fichiers, 0 différence byte
|
||||||
|
```
|
||||||
|
|
||||||
|
Cargo/rustc/rustfmt ne sont pas disponibles dans l'environnement d'assemblage ; aucun gate Rust local n'est déclaré PASS.
|
||||||
|
|
||||||
|
## 12. Validation opérateur attendue
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
Comme la tranche ne change aucun code ni dépendance, ce gate de publication reste volontairement minimal. Un échec ouvre `pre.009-fix.NNN` seulement s'il relève du CHANGELOG, ROADMAP, prompt ou de la mécanique de publication ; une anomalie de code/documentation durable antérieure retourne dans son couloir propre.
|
||||||
|
|
||||||
|
## 13. Décisions prises
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10 est fermé fonctionnellement autour de la common RAW et de ses preuves cross-source
|
||||||
|
le Worker concret commence en 0.3.11
|
||||||
|
0.3.11 n'exige aucune source live complexe pour sa clôture
|
||||||
|
pre.001 de 0.3.11 doit re-sizer la release avant codage lourd
|
||||||
|
D1/D2/D3/D4 et STRUCTURAL restent la terminologie durable des couches Store
|
||||||
|
```
|
||||||
|
|
||||||
|
## 14. Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question de `0.3.10` ne doit être résolue dans `pre.009`. Les choix exacts de runtime/settings/supervisor/harness du Worker sont transférés au gate `0.3.11-pre.001`.
|
||||||
|
|
||||||
|
## 15. Suite
|
||||||
|
|
||||||
|
Si le gate opérateur de cette tranche est vert :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-rel.001
|
||||||
|
```
|
||||||
|
|
||||||
|
doit rester une publication stable mécanique : version Cargo finale `0.3.10`, delta `rel.001`, commit de release puis tag stable `v0.3.10`, sans rattrapage fonctionnel ou documentaire.
|
||||||
184
deltas/0.3.10/rel.001.md
Normal file
184
deltas/0.3.10/rel.001.md
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
<!-- file: deltas/0.3.10/rel.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.10-rel.001` — publication stable
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
Base directe attendue :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.009
|
||||||
|
workspace.package.version = 0.3.10-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate minimal opérateur de `pre.009`, exécuté le **8 septembre 2026**, est propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check: PASS
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 771 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les couloirs de fermeture antérieurs sont déjà fermés :
|
||||||
|
|
||||||
|
- `pre.007` : gate technique final sur `0.3.10-pre.7`, avec audits Rust/Markdown, `cargo check`, Clippy workspace/all-targets/all-features `-D warnings`, workspace tests complets, suites common/Transport/Backfill et graphes Cargo propres ;
|
||||||
|
- `pre.008` puis `pre.008-fix.001` : réconciliation documentaire finale, terminologie Store `D1 RAW -> D2 STRUCTURAL -> D3 DECODED -> D4 DOMAIN`, réservation de `N1–N4` aux niveaux architecturaux et documentation durable de `ksp-raw-transaction-lib` ;
|
||||||
|
- `pre.009` : préparation minimale de publication avec `CHANGELOG.md`, `ROADMAP.md` et `prompts/030-V0_3_11_START_PROMPT.md`.
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Publier mécaniquement la version stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10
|
||||||
|
```
|
||||||
|
|
||||||
|
Conformément à `VER-LIFECYCLE-012`, cette tranche ne corrige ni code, ni test, ni documentation durable, ni architecture, ni configuration, ni dépendance, ni prompt.
|
||||||
|
|
||||||
|
Tout défaut nouveau renvoie vers une prerelease appropriée ; `rel.001` n'est pas une tranche de rattrapage.
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
### `Cargo.toml`
|
||||||
|
|
||||||
|
Le header est incrémenté parce que le fichier est réellement modifié :
|
||||||
|
|
||||||
|
```text
|
||||||
|
504 -> 505
|
||||||
|
```
|
||||||
|
|
||||||
|
La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.10-pre.9 -> 0.3.10
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune autre ligne du `Cargo.toml` racine n'est modifiée.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.10/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Surfaces explicitement inchangées
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
README.md
|
||||||
|
RULES.md
|
||||||
|
prompts/030-V0_3_11_START_PROMPT.md
|
||||||
|
crates/**
|
||||||
|
docs/**
|
||||||
|
config/**
|
||||||
|
```
|
||||||
|
|
||||||
|
La surface préparée en `pre.009` reste byte-identique hors mécanique Cargo et ajout du présent delta.
|
||||||
|
|
||||||
|
## Surface stable publiée
|
||||||
|
|
||||||
|
`0.3.10` stabilise `ksp-raw-transaction-lib` comme lower-layer D1 RAW source-neutral commune aux producteurs `RawTransaction` :
|
||||||
|
|
||||||
|
- canonicalisation RAW v1 commune extraite du Backfill sans changement des golden bytes/hash ;
|
||||||
|
- construction de `RawTransaction` et `RawTransactionObservation` via les modèles backend-neutral de `ksp-store-api` ;
|
||||||
|
- identité canonique `(network, signature)` et observations producteur séparées ;
|
||||||
|
- wire Solana source-neutral Legacy/V0/V1, sérialisation Base64 et extraction de signature depuis le matériau transactionnel complet ;
|
||||||
|
- dépendances volontairement limitées, sans Transport, Config, runtime async, Job, Worker, `ksp-store-lib` ni backend Store physique ;
|
||||||
|
- migration de `ksp-job-backfill-lib` vers la common crate sans edge Job vers Worker.
|
||||||
|
|
||||||
|
La release stabilise aussi la qualification cross-source préparatoire au futur Worker :
|
||||||
|
|
||||||
|
- `ksp-onchain-transport-lib::get_block_observed` conserve l'identité sûre de la route HTTP réellement gagnante ;
|
||||||
|
- `blockSubscribe` standard `full/base64` est qualifié comme RAW-direct sur le sous-ensemble déterministe prouvé ;
|
||||||
|
- Helius `transactionSubscribe` reste un signal nécessitant hydration HTTP lorsque les métadonnées indispensables manquent ;
|
||||||
|
- Yellowstone Transaction V1 qualifie le wire source-neutral ; Transaction nécessite hydration HTTP pour le `block_time` ;
|
||||||
|
- Yellowstone Block conserve le `block_time`, mais le RAW complet reste conservateur tant que la meta protobuf n'est pas prouvée byte-identical à la meta JSON HTTP ;
|
||||||
|
- aucun RAW-direct Yellowstone complet ni feed EARLY n'est revendiqué sans preuve.
|
||||||
|
|
||||||
|
La nomenclature durable des couches Store est désormais :
|
||||||
|
|
||||||
|
```text
|
||||||
|
D1 RAW
|
||||||
|
D2 STRUCTURAL
|
||||||
|
D3 DECODED
|
||||||
|
D4 DOMAIN
|
||||||
|
```
|
||||||
|
|
||||||
|
`N1–N4` restent réservés aux niveaux architecturaux. Le terme `CORE` n'est plus le nom de la couche Store D2 ; `ksp-core-lib` et les usages réellement liés au Core fondamental restent inchangés.
|
||||||
|
|
||||||
|
Le runtime Worker concret est volontairement hors de `0.3.10`. La trajectoire stabilisée devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11 fondation runtime/persistence déterministe du Worker RawTransaction ingest
|
||||||
|
0.3.12 Yellowstone + hydration HTTP + continuité
|
||||||
|
0.3.13 WS standard / Helius / HTTP live + convergence multi-source
|
||||||
|
0.3.14 gap repair / hardening / smokes provider accessibles
|
||||||
|
0.3.15 ksp-app-raw-transaction-ingest-desk
|
||||||
|
0.3.16 Backfill multi-source / multi-stratégie
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Job Backfill historique paramétré et le Worker Ingest continu restent deux producteurs indépendants du même Store.
|
||||||
|
|
||||||
|
## Suite préparée
|
||||||
|
|
||||||
|
La session suivante part exclusivement de la base stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.10
|
||||||
|
```
|
||||||
|
|
||||||
|
et exécute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/030-V0_3_11_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
pour ouvrir :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11 — fondation runtime du Worker RawTransaction ingest
|
||||||
|
```
|
||||||
|
|
||||||
|
`0.3.11-pre.001` doit commencer par lecture, audit, brainstorming, sizing et planification avant toute implémentation lourde. Si le périmètre n'est pas clôturable dans une seule session ou si une tranche dépasse le budget prévu, la release doit être rescindée avant le développement lourd.
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
La tranche est purement mécanique. Aucune dépendance, feature, source runtime, configuration ou documentation durable n'est modifiée par `rel.001`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.10
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-raw-transaction-lib
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-raw-transaction-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
```
|
||||||
|
|
||||||
|
Après gate propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
commit : v0.3.10-rel.001
|
||||||
|
tag : v0.3.10
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag de prerelease ou `rel.001` n'est requis.
|
||||||
393
deltas/0.3.11/pre.001.md
Normal file
393
deltas/0.3.11/pre.001.md
Normal file
@@ -0,0 +1,393 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.001.md -->
|
||||||
|
<!-- version: 2 -->
|
||||||
|
|
||||||
|
# Delta 0.3.11-pre.001 — audit/sizing fondation Worker RawTransaction ingest
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
archive : khadhroony-solana-project-v0.3.10.zip
|
||||||
|
SHA-256 : befafea61304aaea15c94db7c8b3525c58b7c1622bf880a9110373bcc0ae6ac0
|
||||||
|
ZIP bytes : 7993096
|
||||||
|
ZIP entries : 1901
|
||||||
|
unzip -t : PASS
|
||||||
|
workspace members : 20
|
||||||
|
workspace.package.version base : 0.3.10
|
||||||
|
stable delta : deltas/0.3.10/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le ZIP a une racine unique et a été contrôlé sans entrée absolue, path traversal, symlink, `.git/`, `target/`, `node_modules/`, `Cargo.lock` ou `.env`.
|
||||||
|
|
||||||
|
Préconditions du prompt `030` confirmées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-raw-transaction-lib présent et documenté
|
||||||
|
ksp-worker-api présent
|
||||||
|
ksp-job-backfill-lib présent et consommateur de ksp-raw-transaction-lib
|
||||||
|
ksp-worker-raw-transaction-ingest-lib absent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type de livraison
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-general-0.3.11-pre.001.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
Le type `general` est requis par le bump `Cargo.toml` de toute prerelease non-fix (`VER-ID-009`). L'archive d'échange contient uniquement les fichiers ajoutés/modifiés par cette tranche et son delta.
|
||||||
|
|
||||||
|
## Baseline exécutée avant modification
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 772 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L'audit mécanique était vert mais la lecture humaine des règles a trouvé deux divergences normatives actives non détectées par les scripts.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Exécuter intégralement le gate d'ouverture demandé par `prompts/030-V0_3_11_START_PROMPT.md` : lecture/audit de la stable `v0.3.10`, contrôle renforcé des règles, inventaire Worker/Common RAW/Store/Backfill, fraîcheur des dépendances runtime, brainstorming public/private et supervisor, définition du harness déterministe, threat model, sizing et planification, sans commencer l'implémentation lourde du Worker.
|
||||||
|
|
||||||
|
## Réconciliation normative
|
||||||
|
|
||||||
|
### D1–D4
|
||||||
|
|
||||||
|
Défauts hérités trouvés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
KSP-DURABLE-002 : D2 encore nommé Core canonique
|
||||||
|
KSP-DURABLE-006 : instructions encore qualifiées de faits Core
|
||||||
|
DEP-STORE-004 : D2 encore nommé Core canonique
|
||||||
|
```
|
||||||
|
|
||||||
|
Les règles plus récentes et l'architecture `008` imposent déjà :
|
||||||
|
|
||||||
|
```text
|
||||||
|
D1 RAW
|
||||||
|
D2 STRUCTURAL
|
||||||
|
D3 DECODED
|
||||||
|
D4 DOMAIN
|
||||||
|
```
|
||||||
|
|
||||||
|
Correction : normalisation des trois règles actives. Les archives/deltas/plans historiques ne sont pas réécrits.
|
||||||
|
|
||||||
|
### Nom du Worker RAW
|
||||||
|
|
||||||
|
Les règles `KSP-WORKER-005/006/007/010` utilisaient encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-retriever
|
||||||
|
```
|
||||||
|
|
||||||
|
Le nom durable retenu par les architectures, ROADMAP, CHANGELOG, handoff `0.3.10` et prompt `030` est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Correction : normalisation du nom dans ces quatre règles, sans changement de leur responsabilité fonctionnelle.
|
||||||
|
|
||||||
|
Un scan des règles actives après correction ne trouve plus `D2 Core canonique`, `faits Core distincts` ni `ksp-worker-raw-retriever`.
|
||||||
|
|
||||||
|
## Inventaire et décisions de fondation
|
||||||
|
|
||||||
|
### Frontières préservées
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-api reste runtime-neutral et générique
|
||||||
|
ksp-raw-transaction-lib reste source-neutral sans runtime/Transport/Config/Worker/Job/backend
|
||||||
|
Worker persistera via ksp-store-lib default-features=false
|
||||||
|
Store reste vérité de convergence (network, signature)
|
||||||
|
Backfill et Worker restent producteurs indépendants
|
||||||
|
aucun paramètre historique métier au start Worker
|
||||||
|
aucun edge Worker -> Config/Job/backend physique
|
||||||
|
```
|
||||||
|
|
||||||
|
### Surface publique minimale retenue
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE
|
||||||
|
RawTransactionIngestSettings
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
RawTransactionIngestSnapshot
|
||||||
|
RawTransactionIngestSnapshotSource / future associée
|
||||||
|
ErrorCodes statiques Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
Settings de fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
worker_id
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
shutdown_drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission queue : 1..=65_536, default 256
|
||||||
|
persistence concurrency : 1..=64, default 8
|
||||||
|
shutdown drain : 100 ms..=30 s, default 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
### Types reportés
|
||||||
|
|
||||||
|
Ne sont pas figés publiquement avant une vraie source :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionSourceId
|
||||||
|
SourceSettings
|
||||||
|
SourceCapability
|
||||||
|
SourceRole
|
||||||
|
ContinuityState
|
||||||
|
run/source frontiers
|
||||||
|
provider-specific settings
|
||||||
|
retry/replay/gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
### Runtime
|
||||||
|
|
||||||
|
```text
|
||||||
|
caller-owned active Tokio runtime
|
||||||
|
Worker-owned private supervisor/tasks
|
||||||
|
JoinHandle/JoinSet privés
|
||||||
|
start(settings, Arc<Store>) -> Handle
|
||||||
|
request_stop idempotent
|
||||||
|
wait_terminal async sans Tokio nominal public
|
||||||
|
mpsc admission borné
|
||||||
|
watch stop privé
|
||||||
|
watch snapshot latest-value
|
||||||
|
aucun unbounded channel
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline
|
||||||
|
|
||||||
|
```text
|
||||||
|
private deterministic ingress
|
||||||
|
-> ksp-raw-transaction-lib canonicalization
|
||||||
|
-> Worker-owned deterministic observation key/provenance
|
||||||
|
-> common RawTransactionAcquisition
|
||||||
|
-> ksp-store-lib atomic Normal persistence
|
||||||
|
-> explicit outcomes/counters
|
||||||
|
-> concrete/common latest-value snapshot
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun cache mémoire n'est utilisé comme vérité de correction ; Store décide new/idempotent/conflict.
|
||||||
|
|
||||||
|
### Fault/shutdown
|
||||||
|
|
||||||
|
```text
|
||||||
|
content conflict -> terminal fault, no overwrite, no winner
|
||||||
|
Store error -> terminal safe fault
|
||||||
|
source harness failure -> terminal safe fault
|
||||||
|
stop -> close new admissions + bounded drain admitted work
|
||||||
|
expiry drain -> abort private remaining tasks + join + terminal drain_timeout
|
||||||
|
terminal publié seulement après absence de tâche privée survivante
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fraîcheur externe
|
||||||
|
|
||||||
|
Vérifié le 8 septembre 2026 :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio latest stable = 1.53.1 ; workspace = ^1.53
|
||||||
|
futures-util latest stable = 0.3.34 ; workspace = ^0.3
|
||||||
|
sha2 latest stable = 0.11.0 ; workspace = ^0.11
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun bump workspace n'est requis.
|
||||||
|
|
||||||
|
Dépendances prévues pour la crate foundation lorsqu'elle est créée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-worker-api
|
||||||
|
ksp-raw-transaction-lib
|
||||||
|
ksp-store-lib default-features=false
|
||||||
|
ksp-logging-lib
|
||||||
|
sha2
|
||||||
|
tokio features macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Non retenus en fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
futures-util : pas d'usage nécessaire après choix JoinSet/mpsc/watch/select!
|
||||||
|
ksp-onchain-transport-lib : aucune source live en 0.3.11
|
||||||
|
ksp-config-lib : edge interdit
|
||||||
|
SDK provider : interdit/inutile
|
||||||
|
```
|
||||||
|
|
||||||
|
## Harness et threat model
|
||||||
|
|
||||||
|
Le plan `032` fixe un harness privé sans réseau couvrant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
start/stop
|
||||||
|
source failure
|
||||||
|
queue saturation/backpressure
|
||||||
|
bounded persistence concurrency
|
||||||
|
new/idempotent/new observation
|
||||||
|
content conflict
|
||||||
|
Store failure
|
||||||
|
stop pendant admission/persistence
|
||||||
|
drain timeout
|
||||||
|
no orphan tasks
|
||||||
|
slow/no snapshot listener
|
||||||
|
latest-value concrete/common
|
||||||
|
redaction et absence de surface Job/Config/provider
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sizing recalibré
|
||||||
|
|
||||||
|
Le forecast initial est scindé pour respecter le budget de 15–20 minutes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 crate + dependency firewall
|
||||||
|
pre.003 identity/settings
|
||||||
|
pre.004 lifecycle/start/handle/terminal
|
||||||
|
pre.005 supervisor/task ownership + harness
|
||||||
|
pre.006 bounded admission + common RAW
|
||||||
|
pre.007 Store persistence/idempotence/conflict
|
||||||
|
pre.008 snapshots + Worker API projection
|
||||||
|
pre.009 shutdown/backpressure/fault hardening
|
||||||
|
pre.010 public/release/security hardening
|
||||||
|
pre.011 gate technique final
|
||||||
|
pre.012 réconciliation documentaire
|
||||||
|
pre.013 préparation publication
|
||||||
|
rel.001
|
||||||
|
```
|
||||||
|
|
||||||
|
Décision : `0.3.11` reste clôturable dans une seule session **à condition** de ne pas introduire source live, Transport productif, Config, migration Store ou continuity/gap repair. Toute tranche réellement trop grosse est scindée avant exécution.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
deltas/0.3.11/pre.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/rules/RULES_KSP.md
|
||||||
|
docs/rules/RULES_DEPENDENCIES.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Versions :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml : 505 -> 506
|
||||||
|
workspace.package.version : 0.3.10 -> 0.3.11-pre.1
|
||||||
|
RULES_KSP.md : 40 -> 41
|
||||||
|
RULES_DEPENDENCIES.md : 18 -> 19
|
||||||
|
```
|
||||||
|
|
||||||
|
Le bump Cargo est requis par `VER-ID-009` pour une prerelease non-fix, même sans modification Rust.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validation après modification
|
||||||
|
|
||||||
|
Audits statiques réellement exécutés sur le delta complet :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 775 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
Un scan sémantique complémentaire des règles actives confirme l'absence des trois formulations obsolètes corrigées. La comparaison byte-à-byte contre la stable confirme exactement 3 fichiers ajoutés, 3 fichiers modifiés et aucun fichier supprimé.
|
||||||
|
|
||||||
|
## Preuve opérateur stable distincte
|
||||||
|
|
||||||
|
Le journal opérateur fourni pour `0.3.10` montre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check : sans erreur
|
||||||
|
cargo check --workspace : sans erreur
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings : sans erreur
|
||||||
|
ksp-raw-transaction-lib : 26 tests verts
|
||||||
|
ksp-job-backfill-lib : 77 tests verts
|
||||||
|
cargo tree normal des deux crates : fourni
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette preuve concerne la stable d'entrée ; elle ne devient pas une exécution locale de `0.3.11-pre.001`.
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --workspace --all-targets --all-features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Motif : `cargo` n'est pas installé dans l'environnement d'assemblage. `pre.001` ne contient aucun Rust Worker, mais l'opérateur doit rejouer les gates applicables après application du delta.
|
||||||
|
|
||||||
|
## Questions fermées
|
||||||
|
|
||||||
|
```text
|
||||||
|
settings techniques et bornes V1
|
||||||
|
public/private surface de fondation
|
||||||
|
absence de SourceCapability/Role publics avant 0.3.12
|
||||||
|
start + Arc<Store> + handle
|
||||||
|
terminal future runtime-neutral
|
||||||
|
supervisor/task ownership privé
|
||||||
|
mpsc/watch exacts
|
||||||
|
harness déterministe sans réseau
|
||||||
|
absence de frontier/timestamp public en fondation
|
||||||
|
Store conflict/failure policy
|
||||||
|
Tokio direct + features minimales retenues
|
||||||
|
absence de futures-util/Transport en foundation
|
||||||
|
sizing et maintien de la release
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions reportées
|
||||||
|
|
||||||
|
```text
|
||||||
|
premier contrat public SourceId/capability/role
|
||||||
|
Yellowstone source settings/adapters
|
||||||
|
source required/optional policy
|
||||||
|
retry/reconnect
|
||||||
|
run frontier/continuity
|
||||||
|
gap repair
|
||||||
|
hydration productive
|
||||||
|
WS/Helius/HTTP live
|
||||||
|
multi-provider coalescence
|
||||||
|
hot reconfiguration desired/effective
|
||||||
|
provider smokes
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune question reportée ne bloque `pre.002`.
|
||||||
|
|
||||||
|
## Non inclus
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune nouvelle crate Worker créée
|
||||||
|
aucun Rust modifié
|
||||||
|
aucun Transport/Config/Store code modifié
|
||||||
|
aucune migration
|
||||||
|
aucune source live
|
||||||
|
aucun CHANGELOG/ROADMAP modifié
|
||||||
|
aucun prompt 0.3.12 produit
|
||||||
|
```
|
||||||
164
deltas/0.3.11/pre.002.md
Normal file
164
deltas/0.3.11/pre.002.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.002` — squelette Worker RAW + dependency firewall
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.001
|
||||||
|
workspace.package.version = 0.3.11-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.001` est vert sur `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et Clippy workspace strict `--all-targets --all-features -- -D warnings`. Aucun `cargo test` supplémentaire n'a été communiqué pour ce gate et aucun n'est inventé.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Créer uniquement le squelette de `ksp-worker-raw-transaction-ingest-lib`, l'inscrire au workspace et verrouiller son graphe de dépendances. Aucun settings, identity, `start`, task, channel, Store write, canonicalisation ou snapshot runtime n'est matérialisé dans cette tranche.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.11-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `Cargo.toml` racine passe du header `506` au header `507` et le workspace passe de 20 à 21 membres.
|
||||||
|
|
||||||
|
## Graphe de production exact
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-worker-raw-transaction-ingest-lib
|
||||||
|
-> ksp-core-lib
|
||||||
|
-> ksp-logging-lib
|
||||||
|
-> ksp-raw-transaction-lib
|
||||||
|
-> ksp-store-lib default-features=false
|
||||||
|
-> ksp-worker-api
|
||||||
|
-> sha2
|
||||||
|
-> tokio features=macros,rt,sync,time
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces edges matérialisent uniquement la frontière décidée en `pre.001`. Le code runtime qui les consommera reste réservé à ses tranches dédiées.
|
||||||
|
|
||||||
|
## Firewall explicite
|
||||||
|
|
||||||
|
Absents du manifest de production :
|
||||||
|
|
||||||
|
```text
|
||||||
|
futures-util
|
||||||
|
ksp-config-lib
|
||||||
|
ksp-interface-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-program-api
|
||||||
|
ksp-store-api direct
|
||||||
|
ksp-store-postgres-lib
|
||||||
|
ksp-wallet-lib
|
||||||
|
reqwest
|
||||||
|
Solana protocol SDK direct
|
||||||
|
tokio-tungstenite
|
||||||
|
tonic
|
||||||
|
yellowstone-grpc-proto
|
||||||
|
```
|
||||||
|
|
||||||
|
Il n'existe ni `[dev-dependencies]`, ni `[build-dependencies]` dans cette tranche.
|
||||||
|
|
||||||
|
## Crate-root volontairement vide de comportement
|
||||||
|
|
||||||
|
`src/lib.rs` contient uniquement les lints workspace et la rustdoc de responsabilité. Il n'expose encore :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun pub mod
|
||||||
|
aucun type public
|
||||||
|
aucune fonction publique
|
||||||
|
aucun start/spawn
|
||||||
|
aucun appel Tokio
|
||||||
|
aucun appel Store/common RAW/Worker API
|
||||||
|
```
|
||||||
|
|
||||||
|
Le test externe `dependency_boundary.rs` verrouille le manifest exact, les edges interdits et l'absence de comportement runtime prématuré au crate-root.
|
||||||
|
|
||||||
|
## Documentation volontairement différée
|
||||||
|
|
||||||
|
`README.md` et `USAGE.md` ne sont pas créés à ce stade car la crate ne possède encore aucune API consommable. Leur réconciliation version-neutral reste réservée à `pre.012`, après freeze de la fondation runtime.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
deltas/0.3.11/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 776 file(s))
|
||||||
|
|
||||||
|
contrôle TOML/statique : workspace 0.3.11-pre.2, 21 membres, manifest Worker exact, Store sans default features, Tokio macros/rt/sync/time
|
||||||
|
comparaison byte-à-byte avec pre.001 : 4 fichiers ajoutés, 3 modifiés, 0 supprimé
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées localement
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions conservées
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucune source live dans 0.3.11
|
||||||
|
aucun Transport productif avant 0.3.12
|
||||||
|
Store uniquement via ksp-store-lib default-features=false
|
||||||
|
aucun backend Store direct
|
||||||
|
aucun Job/Backfill/Config dans le Worker
|
||||||
|
futures-util absent tant qu'un usage réel n'est pas démontré
|
||||||
|
Tokio direct parce que le Worker concret possédera ses tâches runtime
|
||||||
|
aucune API publique avant la tranche qui en possède le contrat
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.003` introduit uniquement l'identité Worker concrète et `RawTransactionIngestSettings` avec defaults, bornes et validation. Aucun spawn n'y est autorisé.
|
||||||
100
deltas/0.3.11/pre.003-fix.001.md
Normal file
100
deltas/0.3.11/pre.003-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.003-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.003-fix.001` — propagation d'erreur explicite des settings
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.003
|
||||||
|
workspace.package.version = 0.3.11-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais `cargo clippy --workspace --all-targets --all-features -- -D warnings` rejette deux usages de l'opérateur `?` dans `RawTransactionIngestSettings::new`.
|
||||||
|
|
||||||
|
Diagnostics :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs:42:9
|
||||||
|
clippy::question_mark_used
|
||||||
|
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs:48:9
|
||||||
|
clippy::question_mark_used
|
||||||
|
```
|
||||||
|
|
||||||
|
La règle normative `RUST-ERR-002` interdit également l'opérateur `?` en code de production. Les tests de la crate communiqués dans le même journal restent verts : 4 tests unitaires, 3 tests dependency-boundary et 3 tests public API, sans échec.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement `pre.003` sans modifier son contrat fonctionnel : remplacer les deux propagations `?` par un contrôle de flux explicite retournant exactement la même erreur. Aucun type, borne, default, code d'erreur, dépendance, test, lifecycle ou comportement runtime n'est ajouté.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de production. Conformément à `VER-ID-007` et `VER-ID-010` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.003-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `RawTransactionIngestSettings::new`, les validations de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
utilisent désormais `if let Result::Err(error)` puis `return Err(error)`. La validation de `shutdown_drain_timeout` était déjà explicite et reste inchangée.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.003-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : aucun opérateur `?` restant dans settings.rs
|
||||||
|
scan statique : aucune modification de surface publique/settings hors propagation d'erreur
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.004` reste bloquée jusqu'à validation opérateur verte de ce fix. Aucun périmètre de `pre.003` n'est redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
185
deltas/0.3.11/pre.003.md
Normal file
185
deltas/0.3.11/pre.003.md
Normal file
@@ -0,0 +1,185 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.003.md -->
|
||||||
|
<!-- version: 3 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.003` — identité Worker + settings bornés
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.002
|
||||||
|
workspace.package.version = 0.3.11-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour `pre.002` est entièrement vert : `cargo fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, tests de la crate et arbres Cargo normal/features. Les 3 tests `pre.002` passent et le firewall de dépendances reste conforme.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement l'identité de la verticale Worker RAW transaction ingest et ses settings runtime techniques bornés. Aucun lifecycle, `start`, spawn, task, channel, supervisor, persistence Store, canonicalisation ou source live n'est introduit.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une prerelease non-fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.11-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Identité Worker
|
||||||
|
|
||||||
|
La verticale possède le code stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RAW_TRANSACTION_INGEST_WORKER_KIND_CODE = "raw_transaction_ingest"
|
||||||
|
```
|
||||||
|
|
||||||
|
La transformation de ce texte en `WorkerKindCode` reste sous validation de `ksp-worker-api`; aucun second validateur d'identité n'est créé dans la crate concrète.
|
||||||
|
|
||||||
|
## `RawTransactionIngestSettings`
|
||||||
|
|
||||||
|
Surface exacte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network: ksp_store_lib::RawNetworkId
|
||||||
|
worker_id: ksp_worker_api::WorkerId
|
||||||
|
admission_queue_capacity: usize
|
||||||
|
persistence_concurrency: usize
|
||||||
|
shutdown_drain_timeout: Duration
|
||||||
|
```
|
||||||
|
|
||||||
|
Les champs sont privés et exposés par getters. Deux constructions existent : paramètres explicites bornés et defaults V1 sur des identités déjà validées.
|
||||||
|
|
||||||
|
Bornes exactes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission queue : 1..=65_536, default 256
|
||||||
|
persistence concurrency : 1..=64, default 8
|
||||||
|
shutdown drain : 100 ms..=30 s, default 5 s
|
||||||
|
```
|
||||||
|
|
||||||
|
Les bornes sont inclusives. Aucun champ historique, provider-specific, retry, replay, gap repair, source capability ou frontier n'est ajouté.
|
||||||
|
|
||||||
|
## Erreur settings
|
||||||
|
|
||||||
|
La crate ouvre un seul code propre à cette tranche :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.settings_invalid
|
||||||
|
```
|
||||||
|
|
||||||
|
Il couvre uniquement les limites techniques possédées par le Worker concret. Le contexte d'erreur contient seulement le nom statique du champ invalide :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission_queue_capacity
|
||||||
|
persistence_concurrency
|
||||||
|
shutdown_drain_timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune valeur caller-supplied n'est copiée dans le diagnostic.
|
||||||
|
|
||||||
|
Les erreurs de `RawNetworkId`, `WorkerId` et `WorkerKindCode` restent celles de leurs crates propriétaires.
|
||||||
|
|
||||||
|
## Redaction
|
||||||
|
|
||||||
|
`Debug` du settings rend le network logique mais conserve le `WorkerId(..)` redacted fourni par `ksp-worker-api`; le texte réel du Worker ID n'est pas rendu.
|
||||||
|
|
||||||
|
## Frontière runtime maintenue
|
||||||
|
|
||||||
|
Toujours absents :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
start/spawn
|
||||||
|
JoinHandle/JoinSet
|
||||||
|
mpsc/watch
|
||||||
|
supervisor
|
||||||
|
Store write
|
||||||
|
common RAW conversion
|
||||||
|
Transport
|
||||||
|
source live/provider SDK
|
||||||
|
```
|
||||||
|
|
||||||
|
Le manifest de production est inchangé depuis `pre.002`.
|
||||||
|
|
||||||
|
## Tests ajoutés/étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/settings.rs
|
||||||
|
defaults exacts
|
||||||
|
bornes min/max inclusives
|
||||||
|
six cas hors bornes
|
||||||
|
code/contexte d'erreur stable
|
||||||
|
redaction Debug du WorkerId
|
||||||
|
|
||||||
|
tests/public_api.rs
|
||||||
|
kind + settings consommables depuis crate root
|
||||||
|
validation network/Worker identity reste lower-layer-owned
|
||||||
|
erreur de borne publique stable et sans echo d'identité
|
||||||
|
|
||||||
|
tests/dependency_boundary.rs
|
||||||
|
firewall pre.002 conservé
|
||||||
|
crate-root pre.003 limité à identity/settings
|
||||||
|
absence de comportement runtime prématuré
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/identity.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/settings.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/settings.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
deltas/0.3.11/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Premier passage : quatre écarts de forme détectés par l'audit Rust (rustdoc `pub(crate)`, ordre de constantes, ligne vide de fonction), puis un espacement Markdown documentaire détecté après ajout de la validation. Ils ont tous été corrigés avant livraison.
|
||||||
|
|
||||||
|
Gate final local :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
Markdown table audit: clean (340 table(s), 777 file(s))
|
||||||
|
|
||||||
|
contrôle statique runtime-boundary : clean
|
||||||
|
comparaison byte-à-byte avec pre.002 : 6 fichiers ajoutés, 5 modifiés, 0 supprimé
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
Après gate opérateur vert, `pre.004` introduit le `start` caller-runtime-owned, le handle concret, le lifecycle/terminal et le stop idempotent. Aucun supervisor complet ni persistence réelle ne doit être anticipé avant leurs tranches dédiées.
|
||||||
135
deltas/0.3.11/pre.004.md
Normal file
135
deltas/0.3.11/pre.004.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.004` — start, handle, lifecycle et terminal Worker
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.003-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert après le fix `pre.003-fix.001` sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict et les dix tests de la crate Worker.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.004` du plan `032` : start synchrone sur runtime Tokio caller-owned, handle clonable, stop idempotent, lifecycle générique `ksp-worker-api` et future terminal boxed, sans ouvrir le supervisor de tâches enfants, l'admission, la canonicalisation, la persistence, les snapshots concrets ou une source live.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.004
|
||||||
|
workspace.package.version : 0.3.11-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
## Contrat public ajouté
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_RAW_TRANSACTION_INGEST_RUNTIME_INVALID
|
||||||
|
RawTransactionIngestWorker
|
||||||
|
RawTransactionIngestHandle
|
||||||
|
RawTransactionIngestTerminalFuture
|
||||||
|
RawTransactionIngestWorker::start(settings, Arc<Store>)
|
||||||
|
RawTransactionIngestHandle::request_stop()
|
||||||
|
RawTransactionIngestHandle::wait_terminal()
|
||||||
|
```
|
||||||
|
|
||||||
|
`start` exige un runtime Tokio courant, vérifie le réseau du Store avant spawn et conserve le Store par `Arc` sans le fermer ni l'utiliser encore pour une écriture.
|
||||||
|
|
||||||
|
`request_stop()` partage `WorkerStopToken` et retourne `true` uniquement lors de la première demande acceptée. Le task racine est réveillé par un `watch<bool>` privé ; la disparition de tous les handles ferme ce channel et provoque également une terminaison coopérative.
|
||||||
|
|
||||||
|
`wait_terminal()` retourne une future boxed `Send`. Elle attend un `WorkerState` terminal puis la fermeture du sender d'état par le task racine ; aucun `JoinHandle` public n'est nécessaire.
|
||||||
|
|
||||||
|
## Lifecycle de fondation
|
||||||
|
|
||||||
|
```text
|
||||||
|
Starting -> Running -> Stopping -> Stopped
|
||||||
|
Starting -> Stopping -> Stopped si stop immédiat
|
||||||
|
```
|
||||||
|
|
||||||
|
Les transitions utilisent `WorkerLifecycle`. Un invariant impossible devient `Faulted(worker_raw_transaction_ingest.runtime_invalid)` sans payload, worker id, réseau ou erreur externe dans le diagnostic.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
JoinSet / tasks enfants
|
||||||
|
mpsc / admission
|
||||||
|
canonicalisation common RAW
|
||||||
|
observation key
|
||||||
|
persistence Store
|
||||||
|
snapshots concrets / WorkerSnapshotSource
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
```
|
||||||
|
|
||||||
|
Les méthodes de snapshot prévues par la surface finale restent différées à `pre.008`. `pre.005` introduit le supervisor et l'ownership des tâches enfants autour du wake-up de stop déjà minimalement nécessaire ici.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
Les tests couvrent statiquement ou via harness runtime privé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
runtime Tokio courant obligatoire
|
||||||
|
network Store mismatch sûr
|
||||||
|
stop immédiat idempotent
|
||||||
|
Running -> stop via clone de handle
|
||||||
|
drop du dernier handle -> sortie du task racine
|
||||||
|
surface publique start/handle/terminal future
|
||||||
|
absence de JoinHandle public et du scope pre.005+
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
deltas/0.3.11/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des interdits/scope runtime
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.004` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.005` reste bloquée jusqu'à validation opérateur verte de cette tranche.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Le supervisor/tasks enfants, l'admission, la persistence et les snapshots restent dans leurs tranches planifiées.
|
||||||
125
deltas/0.3.11/pre.005.md
Normal file
125
deltas/0.3.11/pre.005.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.005` — supervisor privé et ownership des tâches
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.004
|
||||||
|
workspace.package.version = 0.3.11-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 16 tests de la crate Worker, doc-tests et les arbres Cargo normal/features.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.005` du plan `032` : supervisor privé, `JoinSet` privé, ownership et join des tâches enfants, intégration du stop au seam source déterministe et preuve qu'aucune tâche enfant coopérative ne survit au terminal. Aucun admission `mpsc`, aucune canonicalisation RAW et aucune persistence Store réelle.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.005
|
||||||
|
workspace.package.version : 0.3.11-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Runtime privé ajouté
|
||||||
|
|
||||||
|
Le task racine devient le supervisor du run. Il possède un `tokio::task::JoinSet<()>` privé et un seam privé de spawn de tâches source.
|
||||||
|
|
||||||
|
La production appelle ce seam sans source. Les tests injectent des tâches déterministes qui reçoivent un clone du `watch<bool>` de stop.
|
||||||
|
|
||||||
|
Le supervisor :
|
||||||
|
|
||||||
|
```text
|
||||||
|
reap des enfants terminés via join_next()
|
||||||
|
stop prioritaire via select! biased
|
||||||
|
join de tous les enfants restants après stop/fermeture du contrôle
|
||||||
|
Stopped publié uniquement après join complet
|
||||||
|
JoinError -> supervision non clean ; Faulted à la fermeture de supervision
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun `JoinHandle` ou `JoinSet` n'est ajouté à la surface publique.
|
||||||
|
|
||||||
|
## Arrêt coopératif
|
||||||
|
|
||||||
|
Le timeout de drain configuré existe déjà dans les settings mais n'est pas encore appliqué ici. `pre.005` attend coopérativement les enfants ; le timeout, `abort_all` et les races stop/fault sont réservés au hardening `pre.009`.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
mpsc / admission
|
||||||
|
PrivateRawTransactionIngress
|
||||||
|
canonicalisation common RAW
|
||||||
|
observation key
|
||||||
|
persistence Store
|
||||||
|
snapshots concrets / WorkerSnapshotSource
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
abort/timeout de drain
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
supervisor joint trois tâches source coopératives avant Stopped
|
||||||
|
compteur d'enfants actifs == 0 au terminal
|
||||||
|
reap d'une source terminée pendant Running
|
||||||
|
join d'une source encore active lors du stop
|
||||||
|
canary statique : JoinSet présent mais privé
|
||||||
|
canary statique : aucun scope pre.006+ ouvert
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des interdits/scope runtime
|
||||||
|
Markdown table audit: clean (340 table(s), 780 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.005` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.006` reste bloquée jusqu'à validation opérateur verte de cette tranche.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. L'admission bornée et la canonicalisation common RAW restent dans `pre.006`, la persistence dans `pre.007`, les snapshots dans `pre.008` et le drain forcé/races dans `pre.009`.
|
||||||
97
deltas/0.3.11/pre.006-fix.001.md
Normal file
97
deltas/0.3.11/pre.006-fix.001.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.006-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.006-fix.001` — correction syntaxique du test `tokio::select!`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.006
|
||||||
|
workspace.package.version = 0.3.11-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler le test `unit_tests/admission.rs`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/../unit_tests/admission.rs:152:13
|
||||||
|
error: no rules expected `result`
|
||||||
|
```
|
||||||
|
|
||||||
|
La cause est syntaxique : deux branches adjacentes du `tokio::select!` du test de préemption du sender bloqué n'étaient pas séparées par des virgules.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement le test de `pre.006` sans modifier le contrat fonctionnel, le runtime de production, l'admission, la canonicalisation Common RAW, l'observation key, le golden ou les dépendances.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.006-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le `tokio::select!` du test `pre_006_bounded_sender_is_preempted_by_stop_while_queue_is_full` sépare désormais explicitement ses branches :
|
||||||
|
|
||||||
|
```text
|
||||||
|
_changed = blocked_stop.changed() => false,
|
||||||
|
result = blocked_sender.send(blocked_ingress) => result.is_ok(),
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun comportement de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.006-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le `tokio::select!` fautif contient deux branches séparées par des virgules
|
||||||
|
scan statique : aucun fichier de production de la crate Worker n'est modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.007` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre de `pre.006` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
180
deltas/0.3.11/pre.006.md
Normal file
180
deltas/0.3.11/pre.006.md
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.006.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.006` — admission bornée et canonicalisation Common RAW
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.005
|
||||||
|
workspace.package.version = 0.3.11-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 18 tests de la crate Worker, doc-tests et les arbres Cargo normal/features.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.006` du plan `032` : admission `mpsc` centrale bornée, backpressure async, ingress privé, canonicalisation via `ksp-raw-transaction-lib`, observation key Worker déterministe et assembly du `RawTransactionAcquisition`. Aucune persistence Store et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.006
|
||||||
|
workspace.package.version : 0.3.11-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Admission privée bornée
|
||||||
|
|
||||||
|
Le supervisor crée un seul channel central dimensionné par `RawTransactionIngestSettings::admission_queue_capacity()` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio::sync::mpsc::channel(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le seam source reçoit le sender `mpsc` borné et le signal de stop. Le contrat prouvé est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
send().await pour le backpressure
|
||||||
|
select! biased : stop avant send dans le harness source
|
||||||
|
aucun unbounded_channel
|
||||||
|
receiver close au shutdown
|
||||||
|
```
|
||||||
|
|
||||||
|
Au shutdown, le receiver ferme les nouvelles admissions et consume les entrées déjà en queue avant le join final des tâches enfants.
|
||||||
|
|
||||||
|
## Ingress et Common RAW
|
||||||
|
|
||||||
|
L'ingress crate-private, absent de l'API publique, transporte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionMaterial
|
||||||
|
RawNetworkId attendu
|
||||||
|
RawAcquisitionProvenance
|
||||||
|
source key opaque [u8; 32]
|
||||||
|
```
|
||||||
|
|
||||||
|
Le pipeline est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network guard
|
||||||
|
ksp_raw_transaction_lib::canonicalize_raw_transaction(...)
|
||||||
|
material network guard
|
||||||
|
observation key Worker
|
||||||
|
ksp_raw_transaction_lib::assemble_raw_transaction_acquisition(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune canonicalisation RAW v1 n'est réimplémentée dans le Worker.
|
||||||
|
|
||||||
|
## Observation key V1
|
||||||
|
|
||||||
|
Domaine exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp.raw_transaction_ingest.observation.v1\0
|
||||||
|
```
|
||||||
|
|
||||||
|
Le SHA-256 encode avec longueur `u64` big-endian :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network
|
||||||
|
signature
|
||||||
|
source key
|
||||||
|
```
|
||||||
|
|
||||||
|
Golden harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
network : mainnet
|
||||||
|
signature : 64 x 0x07
|
||||||
|
source key : 32 x 0x09
|
||||||
|
SHA-256 : f8d161dca8a8a72a0da110fc87882ee7136848461c6aa32b09e30a6188b95dcc
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun timestamp local, run id ou ordre d'admission ne participe à la clé.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
capacity=1 : second send bloqué jusqu'à consommation d'un slot
|
||||||
|
stop : sender bloqué rejeté sans admission post-stop
|
||||||
|
payload common RAW canonique exact
|
||||||
|
format id/version exacts
|
||||||
|
observation-key golden exact
|
||||||
|
assembly transaction/provenance/reference exact
|
||||||
|
mismatch network ingress/material rejetés sans fuite de valeurs
|
||||||
|
dependency boundary : mpsc/common RAW requis, persistence/Transport/snapshot toujours interdits
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
deltas/0.3.11/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionWrite / persistence Store
|
||||||
|
RawTransactionAcquisitionMode::Normal
|
||||||
|
persistence concurrency
|
||||||
|
Store outcome mapping / idempotence / conflict
|
||||||
|
snapshots concrets
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (340 table(s), 781 file(s))
|
||||||
|
|
||||||
|
scan production : aucun ?, unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucune persistence Store, aucun Transport, aucun snapshot concret
|
||||||
|
pipeline : exactement 1 mpsc::channel, 1 canonicalize_raw_transaction, 1 assemble_raw_transaction_acquisition
|
||||||
|
observation golden indépendant : f8d161dca8a8a72a0da110fc87882ee7136848461c6aa32b09e30a6188b95dcc
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.006` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib --edges normal
|
||||||
|
cargo tree -p ksp-worker-raw-transaction-ingest-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.007` reste bloquée jusqu'à validation opérateur verte de l'admission bornée, du backpressure, du golden observation et de l'assembly Common RAW.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. La persistence réelle et les outcomes Store restent dans `pre.007`, les snapshots dans `pre.008`, et le hardening fault/drain dans `pre.009`.
|
||||||
161
deltas/0.3.11/pre.007.md
Normal file
161
deltas/0.3.11/pre.007.md
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.007.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.007` — persistence Store bornée et outcomes déterministes
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.006-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, les 22 tests de la crate Worker et les doc-tests. Aucun `cargo tree` n'a été rejoué après le fix, celui-ci n'ayant modifié aucune dépendance ni feature.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.007` du plan `032` : persistence atomique via la façade Store, mode `Normal`, concurrence bornée, classification entity/observation, idempotence/purge et terminaux dédiés content conflict/Store failure. Aucun snapshot concret et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.007
|
||||||
|
workspace.package.version : 0.3.11-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
## Port Store privé
|
||||||
|
|
||||||
|
La crate ajoute un port `RawTransactionIngestPersistencePort` crate-private, implémenté par `ksp_store_lib::Store` et utilisé exclusivement avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionWrite::persist_raw_transaction_acquisition(..., RawTransactionAcquisitionMode::Normal)
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun backend Store concret n'entre dans le graphe ou dans le code Worker.
|
||||||
|
|
||||||
|
## Outcomes
|
||||||
|
|
||||||
|
Classification exacte :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Inserted + Inserted -> Inserted / Inserted
|
||||||
|
AlreadyPresent + Inserted -> AlreadyPresent / Inserted
|
||||||
|
AlreadyPresent + AlreadyPresent -> AlreadyPresent / AlreadyPresent
|
||||||
|
SkippedPurged + NotRecorded -> SkippedPurged / NotRecorded
|
||||||
|
```
|
||||||
|
|
||||||
|
`Rehydrated` et les couples impossibles deviennent `runtime_invalid`.
|
||||||
|
|
||||||
|
## Faults Store
|
||||||
|
|
||||||
|
Deux codes publics deviennent actifs :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.content_conflict
|
||||||
|
worker_raw_transaction_ingest.store_failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le conflit Store canonique est projeté sur `content_conflict`. Toute autre erreur Store est réduite à `store_failed` en conservant uniquement son domain/code sûr, jamais son texte arbitraire.
|
||||||
|
|
||||||
|
## Concurrence bornée
|
||||||
|
|
||||||
|
Le supervisor possède un `JoinSet` privé de persistences et n'admet une nouvelle acquisition que lorsque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
in_flight_persistence < persistence_concurrency
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune queue secondaire ou persistence non bornée n'est introduite. Le harness bloque quatre writes avec une limite de deux et exige `max_active == 2`.
|
||||||
|
|
||||||
|
## Drain
|
||||||
|
|
||||||
|
Sur stop ou fault : nouvelles admissions fermées, acquisitions déjà admises drainées/persistées, persistences rejointes, sources rejointes, puis terminal. Le timeout/abort et la priorité complète des races terminales restent dans `pre.009`.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
mode Normal exact
|
||||||
|
new/idempotent/purged + observation distincte
|
||||||
|
Rehydrated/couples impossibles rejetés
|
||||||
|
network mismatch avant write
|
||||||
|
content conflict redacted
|
||||||
|
Store failure redacted
|
||||||
|
persistence concurrency réellement bornée
|
||||||
|
terminaux runtime content_conflict/store_failed
|
||||||
|
ErrorCode publics stables
|
||||||
|
dependency boundary persistence privée / backend direct interdit
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/persistence.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/persistence.rs
|
||||||
|
deltas/0.3.11/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
snapshots concrets / compteurs publics
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
backend Store direct
|
||||||
|
ForceRehydrate
|
||||||
|
source_failed / counter_exhausted
|
||||||
|
drain timeout / abort forcé
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean
|
||||||
|
|
||||||
|
scan production : aucun unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucun backend Store direct, aucun Transport, aucun snapshot concret
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.007` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'a changé dans cette tranche ; aucun `cargo tree` intermédiaire n'est donc requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.008` reste bloquée jusqu'à validation opérateur verte de la persistence `Normal`, des outcomes/idempotences, de la concurrence bornée et des terminaux Store/conflict.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les snapshots/counters restent dans `pre.008`; le hardening fault/shutdown reste dans `pre.009`.
|
||||||
100
deltas/0.3.11/pre.008-fix.001.md
Normal file
100
deltas/0.3.11/pre.008-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.008-fix.001` — correction Clippy du traitement de publication snapshot
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.008
|
||||||
|
workspace.package.version = 0.3.11-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et tous les tests de `ksp-worker-raw-transaction-ingest-lib` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` rejette un `if` imbriqué dans `src/runtime.rs`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs:176:17
|
||||||
|
clippy::collapsible_if
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code traitait une erreur de publication snapshot puis testait séparément `fault.is_none()`. Clippy exige la forme conditionnelle fusionnée.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement `pre.008` sans modifier son comportement : même erreur, même priorité du premier fault, mêmes snapshots, mêmes compteurs, même lifecycle et même runtime. Aucune dépendance, feature ou surface publique n'est modifiée.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de production. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.008-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le traitement fautif devient la forme équivalente acceptée par Clippy :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
if let Result::Err(error) = published
|
||||||
|
&& fault.is_none()
|
||||||
|
{
|
||||||
|
fault = Option::Some(error.code());
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La sémantique reste identique : une erreur de publication snapshot ne remplace jamais un fault déjà fixé.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.008-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le nested-if signalé n'existe plus
|
||||||
|
scan statique : aucune dépendance/feature ni surface publique modifiée
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.009` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre de `pre.008` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
167
deltas/0.3.11/pre.008.md
Normal file
167
deltas/0.3.11/pre.008.md
Normal file
@@ -0,0 +1,167 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.008.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.008` — snapshots latest-value et projection Worker API
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.007
|
||||||
|
workspace.package.version = 0.3.11-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 31 tests de la crate Worker et les doc-tests. Aucun `cargo tree` intermédiaire n'était requis : `pre.007` n'avait modifié aucune dépendance ni feature.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la responsabilité `pre.008` du plan `032` : snapshot concret latest-value, compteurs checked, projection directe vers `ksp-worker-api::WorkerSnapshotSource`, coalescence des listeners et terminal retenu. Aucun hardening timeout/abort et aucune source réseau.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.008
|
||||||
|
workspace.package.version : 0.3.11-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Snapshot concret
|
||||||
|
|
||||||
|
La crate expose :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionIngestSnapshot
|
||||||
|
RawTransactionIngestSnapshotFuture
|
||||||
|
RawTransactionIngestSnapshotSource
|
||||||
|
```
|
||||||
|
|
||||||
|
Le snapshot contient uniquement la projection `WorkerSnapshot`, les capacités/profondeurs techniques bornées et les compteurs sûrs prévus par le plan.
|
||||||
|
|
||||||
|
## Un seul flux latest-value
|
||||||
|
|
||||||
|
Un unique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokio::sync::watch::channel<RawTransactionIngestSnapshot>
|
||||||
|
```
|
||||||
|
|
||||||
|
porte l'état observable du run. Le watch terminal spécialisé des tranches précédentes disparaît : `wait_terminal()`, `snapshot_source()` et `worker_snapshot_source()` lisent tous ce même flux.
|
||||||
|
|
||||||
|
`RawTransactionIngestSnapshotSource` implémente directement `ksp_worker_api::WorkerSnapshotSource`. La sequence `WorkerSnapshotSequence` est donc identique pour la vue concrète et la projection commune.
|
||||||
|
|
||||||
|
## Compteurs checked
|
||||||
|
|
||||||
|
Les compteurs `u64` effectivement incrémentés dans cette tranche utilisent `checked_add(1)`. Aucun wrap silencieux n'est admis. Le nouveau code public est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.counter_exhausted
|
||||||
|
```
|
||||||
|
|
||||||
|
Les compteurs sont mis à jour au point d'ownership du supervisor : dequeue/canonicalisation et récolte de persistence. `admission_queue_depth` provient directement du receiver `mpsc` borné ; `in_flight_persistence` provient du `JoinSet` privé existant.
|
||||||
|
|
||||||
|
`source_failure_total` et `backpressure_wait_total` font partie du snapshot sûr mais restent à zéro en `pre.008`; leur comportement effectif appartient à `pre.009`.
|
||||||
|
|
||||||
|
## Projection common Worker
|
||||||
|
|
||||||
|
Mapping de fondation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Starting -> health Unknown
|
||||||
|
Running -> health Healthy
|
||||||
|
Stopping / Stopped -> conserve le dernier health non terminal
|
||||||
|
Faulted -> health Unhealthy
|
||||||
|
queue > 0 ou persistence > 0 -> activity Active
|
||||||
|
sinon Running/Stopping/terminal -> activity Idle
|
||||||
|
```
|
||||||
|
|
||||||
|
La state machine reste celle de `ksp-worker-api::WorkerLifecycle`.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
snapshot initial exact
|
||||||
|
projection common/concrete même sequence
|
||||||
|
counter exhaustion stable sans wrap
|
||||||
|
listener lent coalescé vers la dernière valeur
|
||||||
|
terminal latest-value retenu après fermeture publisher
|
||||||
|
surface publique snapshot_source/worker_snapshot_source
|
||||||
|
WorkerSnapshotSource Send + Sync
|
||||||
|
pipeline runtime réussi -> counters exacts
|
||||||
|
Store failure -> health Unhealthy + store_failure_total
|
||||||
|
absence de watch terminal séparé
|
||||||
|
absence de nouveau dependency edge
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
deltas/0.3.11/pre.008.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Hors scope conservé
|
||||||
|
|
||||||
|
```text
|
||||||
|
source live / Transport
|
||||||
|
Config
|
||||||
|
backend Store direct
|
||||||
|
source_failed public
|
||||||
|
source failure behavior
|
||||||
|
backpressure wait instrumentation productive
|
||||||
|
drain timeout / abort forcé
|
||||||
|
fault/stop precedence finalisée
|
||||||
|
retry / reconnect / gap repair
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean
|
||||||
|
|
||||||
|
scan production : aucun unwrap, expect, panic, unbounded_channel
|
||||||
|
scan scope : aucun backend Store direct, aucun Transport, aucune nouvelle dépendance
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.008` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'a changé dans cette tranche ; aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.009` reste bloquée jusqu'à validation opérateur verte des snapshots common/concrete, des compteurs checked et du terminal latest-value retenu.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Le hardening shutdown/backpressure/fault races reste dans `pre.009`.
|
||||||
100
deltas/0.3.11/pre.009-fix.001.md
Normal file
100
deltas/0.3.11/pre.009-fix.001.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.001` — correction de compilation des tests de hardening
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009
|
||||||
|
workspace.package.version = 0.3.11-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler deux tests ajoutés par `pre.009`.
|
||||||
|
|
||||||
|
Diagnostics exacts :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/admission.rs : appel à RawTransactionAdmission::receive sans expected_network et traitement Option au lieu de Result<Option<_>>
|
||||||
|
unit_tests/runtime.rs : type du paramètre children non inférable dans le closure source-failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production compile ; les défauts sont strictement localisés dans les tests de `pre.009`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement les tests de hardening de `pre.009`, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche des fichiers Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modifications
|
||||||
|
|
||||||
|
Le test `pre_009_full_queue_dequeue_marks_source_neutral_backpressure_observation` conserve désormais une copie du `RawNetworkId`, appelle explicitement `receive(&network)` et vérifie la forme `Result<Option<_>>` avec `matches!`, sans `unwrap`, `expect` ni `?`.
|
||||||
|
|
||||||
|
Le test `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault` annote explicitement le paramètre `children` du closure avec le type privé attendu par le harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
&mut tokio::task::JoinSet<ksp_core_lib::Result<()>>
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : les deux appels receive utilisent expected_network et traitent Result<Option<_>>
|
||||||
|
scan statique : le closure source-failure annote explicitement JoinSet<ksp_core_lib::Result<()>>
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
97
deltas/0.3.11/pre.009-fix.002.md
Normal file
97
deltas/0.3.11/pre.009-fix.002.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.002` — complément de compilation du test source-failure
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent, mais que `cargo clippy --workspace --all-targets --all-features -- -D warnings` et `cargo test -p ksp-worker-raw-transaction-ingest-lib` ne peuvent compiler un test de hardening `pre.009`.
|
||||||
|
|
||||||
|
Diagnostic exact :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/runtime.rs : appel de start_foundation_with_source_spawner avec 3 arguments alors que le harness en exige 4 ; store_guard manque
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production compile ; le défaut est strictement localisé dans le test source-failure.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement l'appel du harness dans ce test, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.002
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Le test `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault` fournit désormais explicitement le troisième argument du harness :
|
||||||
|
|
||||||
|
```text
|
||||||
|
std::option::Option::None
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce test ne possède pas de Store réel et ne teste aucune persistence ; l'absence de `store_guard` est donc explicite et conforme au rôle du harness.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le test source-failure fournit explicitement store_guard = None
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
96
deltas/0.3.11/pre.009-fix.003.md
Normal file
96
deltas/0.3.11/pre.009-fix.003.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009-fix.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009-fix.003` — conformité Clippy du test source-failure
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.002
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown, `cargo check --workspace` et les 45 tests de `ksp-worker-raw-transaction-ingest-lib` passent. Le seul échec restant est le gate Clippy strict :
|
||||||
|
|
||||||
|
```text
|
||||||
|
clippy::implicit_return
|
||||||
|
unit_tests/runtime.rs : le bloc async de la source faultée retourne implicitement Result::Err(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code de production et le comportement du Worker ne sont pas en cause.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Rendre le test source-failure conforme à la règle d'explicitation des retours, sans modifier le runtime de production, l'admission, la persistence, les snapshots, les terminaux, les compteurs, les dépendances ou les features.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009-fix.003
|
||||||
|
workspace.package.version : 0.3.11-pre.9.fix.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `pre_009_source_failure_is_counted_and_late_stop_cannot_replace_terminal_fault`, le future injecté dans le `JoinSet` utilise désormais un retour explicite :
|
||||||
|
|
||||||
|
```text
|
||||||
|
return Result::Err(...);
|
||||||
|
```
|
||||||
|
|
||||||
|
La valeur retournée, le code d'erreur injecté et toute la sémantique du test restent inchangés.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009-fix.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique : le bloc async source-failure contient un return explicite
|
||||||
|
scan statique : aucun fichier src/ de production ni manifest de crate Worker modifié
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo du fix n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'étant modifiée, aucun `cargo tree` n'est requis pour ce fix.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.009` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
135
deltas/0.3.11/pre.009.md
Normal file
135
deltas/0.3.11/pre.009.md
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.009.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.009` — hardening shutdown, backpressure et races fault/stop
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.008-fix.001
|
||||||
|
workspace.package.version = 0.3.11-pre.8.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 38 tests de la crate Worker et les doc-tests. Aucune dépendance/feature n'ayant changé, aucun `cargo tree` intermédiaire n'était requis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer uniquement le hardening `pre.009` du plan `032` : faults source sûrs, saturation observable, ordering stop/fault déterministe, deadline de drain, abort+join au timeout et absence de tâches orphelines. Aucune source live n'est introduite.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.009
|
||||||
|
workspace.package.version : 0.3.11-pre.9
|
||||||
|
```
|
||||||
|
|
||||||
|
## Faults source
|
||||||
|
|
||||||
|
Les tâches source privées retournent `ksp_core_lib::Result<()>`. Une erreur source ou un `JoinError` devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
worker_raw_transaction_ingest.source_failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Le détail distant/provider n'est pas propagé et `source_failure_total` est incrémenté via checked-add.
|
||||||
|
|
||||||
|
## Shutdown borné
|
||||||
|
|
||||||
|
Le drain complet est enveloppé par `shutdown_drain_timeout`. En cas de timeout :
|
||||||
|
|
||||||
|
```text
|
||||||
|
admission.close()
|
||||||
|
persistence.abort_all()
|
||||||
|
sources.abort_all()
|
||||||
|
join complet des persistences
|
||||||
|
join complet des sources
|
||||||
|
Faulted(worker_raw_transaction_ingest.drain_timeout)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le terminal n'est donc jamais publié avant la récolte de toutes les tâches possédées.
|
||||||
|
|
||||||
|
## Ordering stop/fault
|
||||||
|
|
||||||
|
Le stop reste prioritaire dans le `select!`, mais un Store fault découvert pendant le drain n'est pas masqué. Un fault terminal déjà décidé n'est pas remplacé par un stop tardif. Le timeout de drain prévaut lorsqu'il empêche la clôture coopérative.
|
||||||
|
|
||||||
|
## Backpressure source-neutral
|
||||||
|
|
||||||
|
L'admission conserve une unique queue `mpsc` bornée. Le dequeue constate si cette queue était pleine à sa capacité configurée et incrémente `backpressure_wait_total`. Aucun drop, aucune queue secondaire et aucun `unbounded_channel` ne sont ajoutés.
|
||||||
|
|
||||||
|
## Tests ajoutés ou étendus
|
||||||
|
|
||||||
|
```text
|
||||||
|
source failure terminal + compteur
|
||||||
|
late stop ne remplace pas source_failed
|
||||||
|
stop puis Store fault pendant drain
|
||||||
|
saturation capacity=1 sans drop
|
||||||
|
backpressure_wait_total observable
|
||||||
|
drain timeout + abort/join des sources et persistences
|
||||||
|
aucune tâche active après terminal timeout
|
||||||
|
projection snapshot source/backpressure
|
||||||
|
error codes source_failed/drain_timeout publics
|
||||||
|
dependency firewall inchangé
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.009.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/error.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/lib.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/src/snapshot.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/public_api.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/admission.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/runtime.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/unit_tests/snapshot.rs
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à la base.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scan statique des frontières runtime/dependencies
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.009` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'ayant changé, aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.010` reste bloquée jusqu'à validation opérateur verte du hardening shutdown/backpressure/fault races.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les sources live restent explicitement hors scope de `0.3.11`.
|
||||||
78
deltas/0.3.11/pre.010-fix.001.md
Normal file
78
deltas/0.3.11/pre.010-fix.001.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.010-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.010-fix.001` — correction du shadowing dans la canarie hardening
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.010
|
||||||
|
workspace.package.version = 0.3.11-pre.10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Défaut opérateur constaté
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 confirme que `cargo fmt --all`, les audits Rust/Markdown et `cargo check --workspace` passent. Le gate Clippy/compilation des tests externes échoue sur `tests/hardening.rs` avec `E0618` : la variable locale `worker_id` masque la fonction helper `worker_id(...)`, puis le second appel tente d'appeler la valeur `WorkerId` comme une fonction.
|
||||||
|
|
||||||
|
Le code de production, les dépendances et le comportement du Worker ne sont pas en cause.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger uniquement cette collision de nom dans la canarie `pre.010`, sans modifier la surface publique, le runtime, les dépendances, les features ou les règles de hardening.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le correctif touche un fichier Rust de test. Conformément aux règles de version applicables aux corrections code/build :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.010-fix.001
|
||||||
|
workspace.package.version : 0.3.11-pre.10.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
Dans `pre_010_debug_and_settings_errors_redact_worker_identity_and_invalid_values`, la variable locale issue du premier parse est renommée de `worker_id` vers `parsed_worker_id`. La fonction helper `worker_id(...)` reste ainsi appelable pour le second identifiant de test.
|
||||||
|
|
||||||
|
Aucun fichier Rust de production n'est modifié.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.11/pre.010-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à `pre.010`.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucun `cargo tree` n'est requis pour ce fix :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.011` reste bloquée jusqu'à validation opérateur verte de ce fix. Le périmètre fonctionnel de `pre.010` n'est pas redéfini.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune.
|
||||||
126
deltas/0.3.11/pre.010.md
Normal file
126
deltas/0.3.11/pre.010.md
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
<!-- file: deltas/0.3.11/pre.010.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.11-pre.010` — hardening public, release et sécurité
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.11-pre.009-fix.003
|
||||||
|
workspace.package.version = 0.3.11-pre.9.fix.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur du 8 septembre 2026 est vert sur `fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy strict, 45 tests de la crate Worker et les doc-tests. Aucune dépendance/feature n'ayant changé, aucun `cargo tree` intermédiaire n'était requis.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Fermer uniquement le hardening externe `pre.010` du plan `032` : surface publique crate-root exacte, dépendances exactes, absence de surface historique, redaction, codes d'erreur, inventaire des modules et scans Config/secret/backend/Transport. Aucun comportement runtime n'est ajouté.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
identifiant de livraison : 0.3.11-pre.010
|
||||||
|
workspace.package.version : 0.3.11-pre.10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Hardening externe
|
||||||
|
|
||||||
|
Nouvelle suite `tests/hardening.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
7 ErrorCode publics uniques et domain-scoped
|
||||||
|
WorkerId redacted dans Debug
|
||||||
|
settings invalides sans echo de valeur ni WorkerId
|
||||||
|
manifest productif exact
|
||||||
|
aucun dev/build dependency
|
||||||
|
crate::Item pour les impl partagées
|
||||||
|
aucune surface Backfill/historical/checkpoint/discovery/hydration/retriever
|
||||||
|
aucun Config productif
|
||||||
|
aucun secret-like material productif
|
||||||
|
aucun backend Store direct
|
||||||
|
aucun Transport productif
|
||||||
|
aucune dépendance retour depuis les couches basses
|
||||||
|
aucun type runtime/backend/live-source exposé par la crate-root
|
||||||
|
```
|
||||||
|
|
||||||
|
## Release completeness
|
||||||
|
|
||||||
|
Nouvelle suite `tests/release_completeness.rs` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
8 modules de production exacts
|
||||||
|
24 exports publics crate-root exacts
|
||||||
|
aucun pub mod
|
||||||
|
présence des canaries dependency/public/hardening requises
|
||||||
|
```
|
||||||
|
|
||||||
|
Le corpus attendu après cette tranche est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
35 unit tests
|
||||||
|
3 dependency-boundary tests
|
||||||
|
7 public API tests
|
||||||
|
8 hardening tests
|
||||||
|
3 release-completeness tests
|
||||||
|
56 tests Worker au total
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/hardening.rs
|
||||||
|
crates/ksp-worker-raw-transaction-ingest-lib/tests/release_completeness.rs
|
||||||
|
deltas/0.3.11/pre.010.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/032-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION_PLAN.md
|
||||||
|
docs/validation/028-V0_3_11_RAW_TRANSACTION_INGEST_WORKER_FOUNDATION.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Code de production
|
||||||
|
|
||||||
|
Aucun fichier `crates/ksp-worker-raw-transaction-ingest-lib/src/*.rs` n'est modifié.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'est ajoutée, supprimée ou modifiée. Le `Cargo.toml` de la crate Worker reste identique à la base.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
scans statiques API/dependency/historical/Config/secret/backend/Transport
|
||||||
|
comparaison byte-for-byte des sources productives et du manifest Worker
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Aucun gate Cargo de `pre.010` n'est déclaré PASS localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
Aucune dépendance ni feature n'ayant changé, aucun `cargo tree` intermédiaire n'est requis :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets --all-features -- -D warnings
|
||||||
|
cargo test -p ksp-worker-raw-transaction-ingest-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décision
|
||||||
|
|
||||||
|
`pre.011` reste bloquée jusqu'à validation opérateur verte des canaries public/release/security.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune nouvelle question architecturale. Les sources live restent hors scope de `0.3.11`.
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user